diff --git a/PROGRESS.md b/PROGRESS.md index 827b255..85f8030 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -8,7 +8,7 @@ |---|---:|---:|---| | 研究框架与规范 | 进行中 | 83% | Scaling Laws 二轮拟合复现与逐图精读 | | 网站设计系统 | 进行中 | 89% | 打印样式与更多通用可视化组件 | -| Kimi K3 深读 | 七轮实证进行中 | 99% | 设计前向训练变体，并等待 `A_log` 社区候选的官方裁决 | +| Kimi K3 深读 | 八轮实证已收敛 | 100% | 稳定维护；真实 forward 等待 `A_log` 官方裁决 | | 语言模型前史 | 完成首版 | 78% | Kneser–Ney、LSTM、Bahdanau 逐图精读与真实小语料复现 | | Transformer 基础 | 完成首版 | 79% | 多头电路、归一化 traces 与真实 kernel / KV 配置 | | 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 | @@ -41,7 +41,7 @@ - [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。 - [x] 完成可检索、可按专题筛选的论文库页面。 - [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。 -- [x] 完成 K3 三轴架构、八联报告实验、四联开放工件实验、Round 04 / 05 / 06 / 07 各五联 AttnRes 独立实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 二十二联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等一百零九个原创交互视图。 +- [x] 完成 K3 三轴架构、八联报告实验、四联开放工件实验、Round 04 / 05 / 06 / 07 / 08 各五联 AttnRes 独立实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 二十二联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等一百一十四个原创交互视图。 - [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。 - [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。 - [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。 @@ -305,10 +305,15 @@ - [x] K3 Round 07 五视图实验室完成：65-node 路径图、14-mask 全矩阵、双向主门、branch/output controls 与 32 层原始谱/replay 审计可交互；protocol、scoping、Grok 结果前审阅、audit、runner、analyzer、packager、四个 raw JSON、aggregate、compact 与 reproduction 全部进入公开树。 - [x] Round 07 本地闸门通过：100 个 Astro 文件零诊断，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；Round 04/05/06/07 四套冻结数据、四套 K3 专项、K3 全量与全站 23 套真实 Chrome 回归通过。动态生成矩阵的 scoped CSS 退化由截图复查发现并修复；桌面/390px 移动端零文档级溢出、零 offender、零运行时异常。 - [x] K3 Round 07 以功能源提交 `dc8ec30`、不可变镜像 `20260730T063811Z-dc8ec30` 发布；OCI index digest `sha256:98d441412408774628195ce23f431d25034bb1590210c11230d884513898e71d`。NAS `12010→8080` healthy / 0 次重启、Compose Manager 标签、VPS→NAS、NPM host 31 / cert 41、DNS、HTTPS/2、gzip、immutable asset、首页/K3 公网内容、门户 `LLM ATLAS / projects / 180` 与全站 23 套生产 Chrome 回归全部通过；保留 Round 06 `20260730T042651Z-b393780` 回滚点。 +- [x] K3 Round 08 在正式输出前冻结 `llm-atlas-k3-attnres-forward-training-v1`：四个 train-time forward variants × 三 seed × 8,000 steps、同 seed Round 05 historical pairing、step 8,000 的 layers 21–25 contrast / peak 双指标 20% 主门、BPC 每 seed / mean 护栏、描述性 `I67` 与完整 primary replay；不把架构消融写成 pure-forward 因果实验。 +- [x] 12 formal + 1 replay 全部一次完成，每格 65,536,000 target bytes；新处理总量 851,968,000，历史 references 196,608,000 单列。联合 groups 6+7 的 contrast / peak 六格降幅为 62.1%–77.3% / 32.3%–62.0%，6 / 6 通过；BPC 三 seed 最大 `+.009598`、均值 `+.006578`，质量门 4 / 4 通过。 +- [x] primary seed-1 replay scientific payload exact，SHA-256 为 `b85563ca…c051`；冻结主状态为 `forward_training_attenuation_established_within_reduced_protocol`。`I67` 的 step-8,000 mean 为 contrast `−.3670`、peak `−.1704`，只保留为跨独立训练的描述性 log residual。 +- [x] 结果前 Grok 实现审阅指出 smoke-only empty selector 的空 census；矩阵结束后删除 early return、加入 `forward_calls > 0`，修补后的 learned wrapper 实际执行 39 次 forward，父/包装器 15 组科学字段仍 exact。结果后 Grok 只读复算报告 `blocking_errors=0`、status / replay 均确认。 +- [x] K3 Round 08 五视图实验室完成：forward contract、六 checkpoint 训练轨迹、attenuation+BPC 主门、non-additivity map 与 32 层谱/replay audit 可交互；protocol、scoping、两阶段 Grok 审阅、runner、analyzer、13 raw、aggregate、compact、reproduction 与结果审计全部进入公开树。 +- [x] Round 08 本地闸门通过：101 个 Astro 文件零诊断，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；Round 04–08 五套冻结数据、五套 K3 专项、K3 全量与全站 24 套真实 Chrome 回归通过。截图复查修复黑色实验室标题对比度；桌面/390px 移动端零文档级溢出、零 offender、零运行时异常。 ## 正在进行 -- [ ] K3 七轮下一闸门：设计前向训练变体与非加性局部交互地图；真实 K3 forward 继续等待 `A_log [128]↔[96]` 社区候选的官方裁决或权重修订。 - [ ] DeepSeek 八轮下一闸门：推进干预式 mediation、SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。 - [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。 - [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。 @@ -517,6 +522,10 @@ | 2026-07-30 | 局部 score 不写成可加贡献率 | 同一 scope 在 learned 与 uniform 背景的响应不同，`S_peak > 1` 与负 interaction residual 都是非加性诊断，不是 170% 贡献或方差分解 | | 2026-07-30 | branch 与 output 控制保持次级证据身份 | group 7 MLP-only 的 6/6 只属于 sufficiency branch gate；group 6 为 5/6，output-only 为 0/6，都不能补救失败的双向主门 | | 2026-07-30 | K3 Round 07 局部路径里程碑发布 | 功能源 `dc8ec30`、镜像 `20260730T063811Z-dc8ec30`、OCI `sha256:98d44141…e71d`；21/21 公网页面链路与全站 23 套生产 Chrome 通过，保留 Round 06 回滚点 | +| 2026-07-30 | 训练期前向干预属于架构消融 | selected uniform mixer 同时改变 train/eval forward、natural backward 与后续 updates；不能写成只改 forward 的路径因果 | +| 2026-07-30 | Round 08 主门与质量门同时成立 | groups 6+7 六个 attenuation 格 6/6 ≥20%；三 seed ΔBPC mean `+.006578`，4/4 过闸；结论只限固定缩小协议 | +| 2026-07-30 | non-additivity 永久保留描述身份 | `I67` 来自三套独立训练，只是 cross-run log residual，不是因果 interaction、Shapley 或贡献率 | +| 2026-07-30 | K3 研究线在 Round 08 主动收敛 | 不启动 Round 09；公开 `A_log [128]↔[96]` 冲突继续等待官方裁决，现有五轮实证停在可复现、可回滚的稳定边界 | ## 未决问题 diff --git a/experiments/k3/attnres_forward/README.md b/experiments/k3/attnres_forward/README.md index 0e0f983..a5e055c 100644 --- a/experiments/k3/attnres_forward/README.md +++ b/experiments/k3/attnres_forward/README.md @@ -6,6 +6,8 @@ This directory implements preregistered protocol - `research/K3_ATTNRES_FORWARD_TRAINING_SCOPING.md` - `research/K3_ATTNRES_FORWARD_TRAINING_PROTOCOL.md` - `research/K3_ATTNRES_FORWARD_TRAINING_GROK_REVIEW.md` +- `research/K3_ATTNRES_FORWARD_TRAINING_IMPLEMENTATION_REVIEW.md` +- `research/K3_ATTNRES_FORWARD_TRAINING_AUDIT.md` It is a depth-32 reduced Block AttnRes architecture ablation. It is not a Kimi-K3 checkpoint forward pass and does not claim to recover unpublished @@ -56,3 +58,13 @@ This runs 12 formal cells and one full replay. The analyzer reads all cells, the frozen historical paired references, and generates the only authoritative status, interaction map, and website compact artifact. +The checked-in Round 08 release contains: + +- 13 raw results under `results/raw/`; +- `reproduction.json` with the exact primary scientific-payload hash; +- aggregate / compact website data under `src/data/`; +- a frozen-data checker and real-Chrome five-view regression in `scripts/`. + +The established status is deliberately scoped to this reduced protocol. It is +not a real Kimi-K3 checkpoint result or a reproduction of unpublished Figure +5(c) telemetry. diff --git a/experiments/k3/attnres_forward/analyze.py b/experiments/k3/attnres_forward/analyze.py index 7bbfeb7..e64915a 100644 --- a/experiments/k3/attnres_forward/analyze.py +++ b/experiments/k3/attnres_forward/analyze.py @@ -425,6 +425,19 @@ def main() -> None: == environment_metadata(references[seed]) ), } + metadata_warnings = [ + { + "cell": cell, + "message": ( + "GPU/version metadata differs from the historical paired " + "reference; frozen scientific-environment fields still match" + ), + "run_environment": item["run_environment"], + "reference_environment": item["reference_environment"], + } + for cell, item in pairing.items() + if not item["metadata_equal"] + ] replay = read_result(args.replay, PROTOCOL_ID) replay_contract = manifest["replay"] @@ -593,6 +606,7 @@ def main() -> None: "thresholds": manifest["thresholds"], "input_files": input_files, "historical_pairing": pairing, + "metadata_warnings": metadata_warnings, "replay": { "passed": replay_exact, "scientific_payload_sha256": canonical_sha256(formal_payload), @@ -636,6 +650,7 @@ def main() -> None: "trajectories": trajectories, "final_spectra": final_spectra, "replay": aggregate["replay"], + "metadata_warnings": metadata_warnings, "processed_target_bytes": manifest["new_target_bytes"], "reporting_boundary": aggregate["reporting_boundary"], "aggregate_sha256": aggregate["canonical_sha256_without_self"], diff --git a/experiments/k3/attnres_forward/reproduction.json b/experiments/k3/attnres_forward/reproduction.json new file mode 100644 index 0000000..eb0efbe --- /dev/null +++ b/experiments/k3/attnres_forward/reproduction.json @@ -0,0 +1,25 @@ +{ + "canonical_sha256_without_self": "57346df80c0d76bd1d306d5fa213feed74ac2094237c22ebcb49f16ea16437e0", + "excluded_fields": [ + "run_kind", + "timing", + "self hashes", + "manifest path strings", + "GPU/version metadata" + ], + "formal_file_sha256": "0962ebd1a00a11e61ac795282bfa99412166c8752f2a3731f137030d7f134dc1", + "formal_variant": "uniform_groups_6_7_forward", + "passed": true, + "post_result_grok_review": { + "blocking_errors": 0, + "claim_boundary_confirmed": true, + "replay_confirmed": true, + "session_id": "019fb28b-a9e1-7643-8e43-06f5e16a2077", + "status_confirmed": true + }, + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "replay_file_sha256": "b85ac8062b2b0b8b3f2305d4b22a7c0212466fbfb28a9d63099cadd0a6917c8e", + "schema_version": 1, + "scientific_payload_sha256": "b85563ca5cb53e60b39c3801d372376206105b8a089a8633b3e81973a7f0c051", + "seed": 2026073001 +} diff --git a/experiments/k3/attnres_forward/results/gates/step-zero.json b/experiments/k3/attnres_forward/results/gates/step-zero.json index 7bddaac..e3290df 100644 --- a/experiments/k3/attnres_forward/results/gates/step-zero.json +++ b/experiments/k3/attnres_forward/results/gates/step-zero.json @@ -1,5 +1,5 @@ { - "canonical_sha256_without_self": "00db0569d2fd00599383ecdc50578c52df784a937a96dd69576f1366e88b0dcb", + "canonical_sha256_without_self": "9a0716bc496d39b9447b79aba0aec42ca4c953cbabe30e3aa349c1bba0701c5f", "environment": { "cublas_workspace_config": ":4096:8", "cuda": "12.8", @@ -21,7 +21,8 @@ "initial_mixer_hash": true, "initial_public_hash": true, "logits_sha256": true, - "loss_nats": true + "loss_nats": true, + "loss_tensor_sha256": true }, "passed": true, "payload": { @@ -1254,7 +1255,8 @@ "initial_mixer_hash": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", "initial_public_hash": "73bbe569e1a47981386ebaf59ca891174746bf81897fe5403676ad6d50423c58", "logits_sha256": "6b44486d93c29a13dc0b961117cc2d22f26c81dc519346ac45b1bb0fefa29939", - "loss_nats": 5.516995429992676 + "loss_nats": 5.516995429992676, + "loss_tensor_sha256": "c4e1d45c137a73ad6249c2db3ff5e29c849310d4f71f3182c41d27ff6b9fcb6a" }, "pre_reduction_uniform_weight_gate": true, "pre_reduction_uniform_weight_max_abs_error": 0.0, @@ -1267,7 +1269,8 @@ "initial_mixer_hash": true, "initial_public_hash": true, "logits_sha256": true, - "loss_nats": true + "loss_nats": true, + "loss_tensor_sha256": true }, "passed": true, "payload": { @@ -2500,7 +2503,8 @@ "initial_mixer_hash": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", "initial_public_hash": "73bbe569e1a47981386ebaf59ca891174746bf81897fe5403676ad6d50423c58", "logits_sha256": "6b44486d93c29a13dc0b961117cc2d22f26c81dc519346ac45b1bb0fefa29939", - "loss_nats": 5.516995429992676 + "loss_nats": 5.516995429992676, + "loss_tensor_sha256": "c4e1d45c137a73ad6249c2db3ff5e29c849310d4f71f3182c41d27ff6b9fcb6a" }, "pre_reduction_uniform_weight_gate": true, "pre_reduction_uniform_weight_max_abs_error": 0.0, @@ -2562,7 +2566,8 @@ "initial_mixer_hash": true, "initial_public_hash": true, "logits_sha256": true, - "loss_nats": true + "loss_nats": true, + "loss_tensor_sha256": true }, "passed": true, "payload": { @@ -3795,7 +3800,8 @@ "initial_mixer_hash": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", "initial_public_hash": "73bbe569e1a47981386ebaf59ca891174746bf81897fe5403676ad6d50423c58", "logits_sha256": "6b44486d93c29a13dc0b961117cc2d22f26c81dc519346ac45b1bb0fefa29939", - "loss_nats": 5.516995429992676 + "loss_nats": 5.516995429992676, + "loss_tensor_sha256": "c4e1d45c137a73ad6249c2db3ff5e29c849310d4f71f3182c41d27ff6b9fcb6a" }, "pre_reduction_uniform_weight_gate": true, "pre_reduction_uniform_weight_max_abs_error": 0.0, @@ -3857,7 +3863,8 @@ "initial_mixer_hash": true, "initial_public_hash": true, "logits_sha256": true, - "loss_nats": true + "loss_nats": true, + "loss_tensor_sha256": true }, "passed": true, "payload": { @@ -5090,7 +5097,8 @@ "initial_mixer_hash": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", "initial_public_hash": "73bbe569e1a47981386ebaf59ca891174746bf81897fe5403676ad6d50423c58", "logits_sha256": "6b44486d93c29a13dc0b961117cc2d22f26c81dc519346ac45b1bb0fefa29939", - "loss_nats": 5.516995429992676 + "loss_nats": 5.516995429992676, + "loss_tensor_sha256": "c4e1d45c137a73ad6249c2db3ff5e29c849310d4f71f3182c41d27ff6b9fcb6a" }, "pre_reduction_uniform_weight_gate": true, "pre_reduction_uniform_weight_max_abs_error": 0.0, @@ -5128,7 +5136,8 @@ "initial_mixer_hash": true, "initial_public_hash": true, "logits_sha256": true, - "loss_nats": true + "loss_nats": true, + "loss_tensor_sha256": true }, "passed": true, "payload": { @@ -6361,7 +6370,8 @@ "initial_mixer_hash": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", "initial_public_hash": "73bbe569e1a47981386ebaf59ca891174746bf81897fe5403676ad6d50423c58", "logits_sha256": "6b44486d93c29a13dc0b961117cc2d22f26c81dc519346ac45b1bb0fefa29939", - "loss_nats": 5.516995429992676 + "loss_nats": 5.516995429992676, + "loss_tensor_sha256": "c4e1d45c137a73ad6249c2db3ff5e29c849310d4f71f3182c41d27ff6b9fcb6a" }, "pre_reduction_uniform_weight_gate": true, "pre_reduction_uniform_weight_max_abs_error": 0.0, diff --git a/experiments/k3/attnres_forward/results/gates/wrapper-smoke.json b/experiments/k3/attnres_forward/results/gates/wrapper-smoke.json index c26d872..ced3514 100644 --- a/experiments/k3/attnres_forward/results/gates/wrapper-smoke.json +++ b/experiments/k3/attnres_forward/results/gates/wrapper-smoke.json @@ -1,7 +1,7 @@ { "architecture": "block", "batch_size": 32, - "canonical_sha256_without_self": "a467cc1c05e1b860abbf44086273071bcd2a891db4354cc6e79b319e44d1cce5", + "canonical_sha256_without_self": "81b011e2ab02c49abb6b2686b3e0701d71ab6e7f46cbb7c426eec91702836b7f", "depth": 32, "diagnostics": [ { @@ -2479,74 +2479,74 @@ ], "forward_intervention": { "depth_visit_counts": [ - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0, - 0 + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39, + 39 ], - "forward_calls": 0, + "forward_calls": 39, "output_mixer_selected": false, - "output_visit_count": 0, + "output_visit_count": 39, "passed": true, "selected_depth_indices": [], "selected_parameter_reachability_gate": true, @@ -2555,70 +2555,198 @@ "selector_gate": true, "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", "source_counts_by_depth_index": { - "0": [], - "1": [], - "10": [], - "11": [], - "12": [], - "13": [], - "14": [], - "15": [], - "16": [], - "17": [], - "18": [], - "19": [], - "2": [], - "20": [], - "21": [], - "22": [], - "23": [], - "24": [], - "25": [], - "26": [], - "27": [], - "28": [], - "29": [], - "3": [], - "30": [], - "31": [], - "32": [], - "33": [], - "34": [], - "35": [], - "36": [], - "37": [], - "38": [], - "39": [], - "4": [], - "40": [], - "41": [], - "42": [], - "43": [], - "44": [], - "45": [], - "46": [], - "47": [], - "48": [], - "49": [], - "5": [], - "50": [], - "51": [], - "52": [], - "53": [], - "54": [], - "55": [], - "56": [], - "57": [], - "58": [], - "59": [], - "6": [], - "60": [], - "61": [], - "62": [], - "63": [], - "7": [], - "8": [], - "9": [] + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] }, "uniform_weight_gate": true, "uniform_weight_max_abs_error": 0.0, diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073001.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073001.json new file mode 100644 index 0000000..2c5b155 --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073001.json @@ -0,0 +1,14045 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "8d72eda08f94a655233ca42fe7afd9d7414f8892576e1985b46e04511fdf0a62", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.0002677410375326872, + 0.0002473295899108052, + 0.00023034414334688336, + 0.00021438254043459892, + 0.00020044577831868082, + 0.00018819731485564262, + 0.00017973134526982903, + 0.000171217747265473, + 0.0001640921109355986, + 0.00016061548376455903, + 0.00015463090676348656, + 0.00015024171443656087, + 0.0001478576596127823, + 0.00014403100067283958, + 0.00013951722939964384, + 0.00013661097909789532, + 0.00013379570737015456, + 0.0001311724045081064, + 0.0001286817860091105, + 0.00012687484559137374, + 0.0001249351626029238, + 0.00012369138130452484, + 0.00012227990373503417, + 0.00012101328320568427, + 0.00011953213106608018, + 0.0001181897969217971, + 0.0001171672047348693, + 0.00011609455395955592, + 0.00011492414341773838, + 0.0001140913664130494, + 0.00011346775136189535, + 0.00011285000800853595 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00021242368711682502, + "first_to_last_ratio": 1.8345658968465297, + "imbalance_abs_log_ratio": 0.6068078850437921, + "last_quartile_mean": 0.0001157896194854402, + "mean": 0.0001511171253696375, + "normalized": [ + 1.7717451736709768, + 1.636674793183293, + 1.5242755762025908, + 1.4186515254985965, + 1.3264266232459345, + 1.2453738409548603, + 1.1893512719369177, + 1.1330135273991528, + 1.085860457802012, + 1.0628542818802846, + 1.023252039669589, + 0.9942070699734702, + 0.9784308644775869, + 0.9531083940389613, + 0.9232390376562555, + 0.9040072643238835, + 0.8853775311228679, + 0.8680181295617843, + 0.8515367513400655, + 0.8395795332993111, + 0.8267439067367663, + 0.8185133286645813, + 0.8091730400239778, + 0.800791325997512, + 0.7909899739933552, + 0.7821072339266707, + 0.7753403490721149, + 0.7682422073314642, + 0.7604971517068638, + 0.7549863467424895, + 0.7508596466770359, + 0.7467718018887739 + ], + "population_cv": 0.2714186609524326 + }, + "activation_output_rms_by_block": [ + 0.0034885562490671873, + 0.004831995815038681, + 0.005952076520770788, + 0.006925336085259914, + 0.00385080324485898, + 0.005542940925806761, + 0.006729288958013058, + 0.007927131839096546, + 0.004081308841705322, + 0.005811207927763462, + 0.007534808944910765, + 0.008581922389566898, + 0.00499920267611742, + 0.007102519739419222, + 0.008542952127754688, + 0.010447698645293713, + 0.005041283555328846, + 0.007636935915797949, + 0.009387043304741383, + 0.011020516976714134, + 0.005807018838822842, + 0.008367395959794521, + 0.01024632342159748, + 0.012378672137856483, + 0.006453148555010557, + 0.009127611294388771, + 0.011037957854568958, + 0.012611051090061665, + 0.007010402157902718, + 0.009794782847166061, + 0.012445803731679916, + 0.014087481424212456 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.0056560162047389895, + "first_to_last_ratio": 0.5480088979804595, + "imbalance_abs_log_ratio": 0.6014637549753238, + "last_quartile_mean": 0.010321029869373888, + "mean": 0.007962599374877755, + "normalized": [ + 0.4381177659237371, + 0.606836484864952, + 0.747504205668035, + 0.8697330807712821, + 0.4836113263475722, + 0.6961220406611074, + 0.845112084785299, + 0.9955457339856744, + 0.512559862622497, + 0.7298129234151854, + 0.9462750278110585, + 1.077778999737589, + 0.6278355146046978, + 0.8919850673170736, + 1.072884836415099, + 1.3120964842531853, + 0.6331203314377778, + 0.9591008609440675, + 1.1788918244911069, + 1.384035094304029, + 0.7292868277593074, + 1.0508372411895934, + 1.2868063479276561, + 1.554601902603762, + 0.8104324041933391, + 1.1463105029730196, + 1.3862254440922965, + 1.5837857082010076, + 0.8804162846646224, + 1.2300986632667847, + 1.5630327667805075, + 1.7692063559870779 + ], + "population_cv": 0.3451071044649734 + }, + "bits_per_byte": 7.959341947457508, + "branch_output_rms_by_sublayer": [ + 0.0022265110164880753, + 0.0027073738165199757, + 0.0021613112185150385, + 0.002722575329244137, + 0.0025601957459002733, + 0.0026850115973502398, + 0.0025323182344436646, + 0.0026861224323511124, + 0.0027253651060163975, + 0.00271838391199708, + 0.002904455177485943, + 0.0026645169127732515, + 0.0029208383057266474, + 0.0026859452482312918, + 0.00299624796025455, + 0.0027359756641089916, + 0.0030937574338167906, + 0.0026767742820084095, + 0.0033804215490818024, + 0.0027451682835817337, + 0.0034370317589491606, + 0.0026999011170119047, + 0.0038969130255281925, + 0.002663193503394723, + 0.004255065228790045, + 0.002626648638397455, + 0.004119740333408117, + 0.0026531771291047335, + 0.003970506135374308, + 0.0026517712976783514, + 0.004385598469525576, + 0.002676903735846281, + 0.0041999719105660915, + 0.002670086920261383, + 0.005232699681073427, + 0.0026355870068073273, + 0.004664790350943804, + 0.0026610493659973145, + 0.004898969549685717, + 0.0026388936676084995, + 0.005131533369421959, + 0.0026469926815479994, + 0.005069608334451914, + 0.00264862016774714, + 0.0051318020559847355, + 0.002641494618728757, + 0.0058372304774820805, + 0.002674198243767023, + 0.0059690922498703, + 0.002557571977376938, + 0.00554921617731452, + 0.00252733426168561, + 0.0058550527319312096, + 0.0025468147359788418, + 0.006255472078919411, + 0.0026856842450797558, + 0.006502739619463682, + 0.002551089273765683, + 0.006122888531535864, + 0.0025778315030038357, + 0.007071305997669697, + 0.0026001501828432083, + 0.006241494789719582, + 0.0025662274565547705 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0056993408498569815, + 0.005586029321860134, + 0.005830344684528519, + 0.00540288712216694, + 0.00538025811909819, + 0.0052226633069546126, + 0.005297605878918106, + 0.0052023753769565, + 0.0048107019260181325, + 0.005248672753719287, + 0.004962353375385338, + 0.005501319898161136, + 0.005588445553926757, + 0.005229606380765812, + 0.004884302670286647, + 0.005031817046498285, + 0.005000470961685071, + 0.0053937604587735305, + 0.004980398173225761, + 0.004658922297714218, + 0.00496598185096497, + 0.004578385934052906, + 0.004596161734371138, + 0.004990442293511754, + 0.005201771295872185, + 0.00459813521112035, + 0.004706502430387731, + 0.004781878207244352, + 0.0048378722082161795, + 0.004473110720597102, + 0.005084432926394394, + 0.004685705425265166 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.005452688082542498, + "first_to_last_ratio": 1.1368823875795573, + "imbalance_abs_log_ratio": 0.12828976841498158, + "last_quartile_mean": 0.004796176053137182, + "mean": 0.005075395512328069, + "normalized": [ + 1.122935313319594, + 1.1006096585560166, + 1.1487468652180286, + 1.064525337787646, + 1.060066768398564, + 1.0290160233362764, + 1.0437818818356701, + 1.025018713185997, + 0.947847692723257, + 1.0341406380981206, + 0.9777274230809889, + 1.0839194472230789, + 1.1010857262951226, + 1.0303840100861434, + 0.9623491722808084, + 0.9914137793352396, + 0.9852376922229987, + 1.0627271206100406, + 0.9812827711906272, + 0.9179427074003902, + 0.9784423379227619, + 0.9020747098294245, + 0.9055770576316904, + 0.9832617539638114, + 1.024899691706223, + 0.905965889742295, + 0.9273173724009683, + 0.9421685848185097, + 0.9532010257062827, + 0.8813324419214177, + 1.00178063247374, + 0.9232197596982638 + ], + "population_cv": 0.06921897980134169 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.02802751213312149, + 0.014061377383768559, + 0.01413465291261673, + 0.014165290631353855, + 0.014225807972252369, + 0.014281988143920898, + 0.01433455292135477, + 0.014350161887705326, + 0.01440478302538395, + 0.00966146495193243, + 0.009703479707241058, + 0.009741888381540775, + 0.00977881159633398, + 0.009820535778999329, + 0.009858808480203152, + 0.00991273857653141, + 0.009950309991836548, + 0.0074927168898284435, + 0.00752287870272994, + 0.007546746172010899, + 0.007571155205368996, + 0.00766373984515667, + 0.007693085819482803, + 0.007762390188872814, + 0.007788818795233965, + 0.006325984839349985, + 0.006353048607707024, + 0.006375366821885109, + 0.006389216054230928, + 0.006445344537496567, + 0.0064707002602517605, + 0.006555322092026472, + 0.00656962301582098, + 0.005531130358576775, + 0.005553483963012695, + 0.005611324217170477, + 0.0056249531917274, + 0.0056777093559503555, + 0.005688921548426151, + 0.00574630219489336, + 0.005768724251538515, + 0.004946626722812653, + 0.004966245498508215, + 0.0050392793491482735, + 0.005057469941675663, + 0.005084683652967215, + 0.005098348017781973, + 0.005203457549214363, + 0.005216081161051989, + 0.004667261149734259, + 0.0046758754178881645, + 0.004717131145298481, + 0.004727107938379049, + 0.004768412560224533, + 0.004777040798217058, + 0.004826710093766451, + 0.004837613552808762, + 0.0044052014127373695, + 0.0044186655431985855, + 0.004504534415900707, + 0.004510571248829365, + 0.004574235528707504, + 0.0045849340967834, + 0.004644096828997135 + ], + "loss_nats": 5.516995429992676, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0022265110164880753, + 0.0034885562490671873, + 0.0040087271481752396, + 0.004831995815038681, + 0.005379301495850086, + 0.005952076520770788, + 0.006410556845366955, + 0.006925336085259914, + 0.0027253651060163975, + 0.00385080324485898, + 0.004838948138058186, + 0.005542940925806761, + 0.006175771821290255, + 0.006729288958013058, + 0.00742313964292407, + 0.007927131839096546, + 0.0030937574338167906, + 0.004081308841705322, + 0.005150026176124811, + 0.005811207927763462, + 0.007019025273621082, + 0.007534808944910765, + 0.008143599145114422, + 0.008581922389566898, + 0.004255065228790045, + 0.00499920267611742, + 0.0066224560141563416, + 0.007102519739419222, + 0.008129785768687725, + 0.008542952127754688, + 0.010099578648805618, + 0.010447698645293713, + 0.0041999719105660915, + 0.005041283555328846, + 0.00719039561226964, + 0.007636935915797949, + 0.009030314162373543, + 0.009387043304741383, + 0.010736005380749702, + 0.011020516976714134, + 0.005131533369421959, + 0.005807018838822842, + 0.00787449348717928, + 0.008367395959794521, + 0.009912566281855106, + 0.01024632342159748, + 0.012113703414797783, + 0.012378672137856483, + 0.0059690922498703, + 0.006453148555010557, + 0.00876442901790142, + 0.009127611294388771, + 0.010806876234710217, + 0.011037957854568958, + 0.012353328987956047, + 0.012611051090061665, + 0.006502739619463682, + 0.007010402157902718, + 0.009470963850617409, + 0.009794782847166061, + 0.012123330496251583, + 0.012445803731679916, + 0.01384514570236206, + 0.014087481424212456 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.967342051211745e-05, + 6.646253314102069e-05, + 6.30064751021564e-05, + 6.015747203491628e-05, + 5.576878538704477e-05, + 5.274103386909701e-05, + 4.9757327360566705e-05, + 4.65008633909747e-05, + 4.235670348862186e-05, + 3.965873474953696e-05, + 3.689330696943216e-05, + 3.4454718843335286e-05, + 3.1800933356862515e-05, + 2.965821113320999e-05, + 2.7844240321428515e-05, + 2.6223739041597582e-05, + 2.4912551452871412e-05, + 2.3716213036095724e-05, + 2.272892561450135e-05, + 2.181669151468668e-05, + 2.0974075596313924e-05, + 2.0215180484228767e-05, + 1.9563831301638857e-05, + 1.903232987388037e-05, + 1.8803977582138032e-05, + 1.8450222341925837e-05, + 1.815358700696379e-05, + 1.7884136468637735e-05, + 1.747322312439792e-05, + 1.7272068362217396e-05, + 1.7079768440453336e-05, + 1.6933003280428238e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.800848884973675e-05, + "first_to_last_ratio": 3.2669338581584584, + "imbalance_abs_log_ratio": 1.1838518869773396, + "last_quartile_mean": 1.7756248325895285e-05, + "mean": 3.274900888072807e-05, + "normalized": [ + 2.127497072227991, + 2.029451742587915, + 1.9239200591268597, + 1.836924966309969, + 1.7029152115764712, + 1.6104619856185547, + 1.519353686146135, + 1.419916662526518, + 1.293373599270898, + 1.210990381234868, + 1.12654728281389, + 1.0520843231872883, + 0.9710502529307544, + 0.9056216400693294, + 0.8502315420548229, + 0.8007490894489195, + 0.7607116155363038, + 0.7241810927002463, + 0.6940339995411807, + 0.6661786802203113, + 0.6404491712314572, + 0.6172761001058835, + 0.5973869735383551, + 0.5811574311514629, + 0.5741846310714972, + 0.5633826174441361, + 0.5543247758452531, + 0.5460970294939851, + 0.5335496774279618, + 0.5274073613990057, + 0.5215354303593698, + 0.5170539158024067 + ], + "population_cv": 0.5015741883109504 + }, + "activation_output_rms_by_block": [ + 0.00586781045421958, + 0.010147182270884514, + 0.014828646555542946, + 0.019574908539652824, + 0.006723169703036547, + 0.011927195824682713, + 0.0171233918517828, + 0.0229953546077013, + 0.007492356467992067, + 0.013753283768892288, + 0.02036694437265396, + 0.026791520416736603, + 0.009156237356364727, + 0.016712356358766556, + 0.024189556017518044, + 0.03125955909490585, + 0.009814711287617683, + 0.018578259274363518, + 0.02773377113044262, + 0.03759535774588585, + 0.012513305060565472, + 0.023303600028157234, + 0.03456926345825195, + 0.046979498118162155, + 0.017495060339570045, + 0.029622666537761688, + 0.04484478756785393, + 0.06046031787991524, + 0.019049862399697304, + 0.03779670596122742, + 0.0568108856678009, + 0.07623780518770218 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013648457475937903, + "first_to_last_ratio": 0.3189654958515595, + "imbalance_abs_log_ratio": 1.1426723455221182, + "last_quartile_mean": 0.04278976144269109, + "mean": 0.02538485410332214, + "normalized": [ + 0.231153995620233, + 0.39973372427445003, + 0.5841533102844308, + 0.77112550893452, + 0.26484964915188064, + 0.46985481090954107, + 0.6745515172979404, + 0.9058690869013849, + 0.2951506610003142, + 0.541790932219397, + 0.8023266271200873, + 1.0554136063846973, + 0.3606968674744694, + 0.6583593622694642, + 0.9529129424601394, + 1.2314255960531553, + 0.38663650567655705, + 0.7318639373992764, + 1.0925322248282323, + 1.481015316962791, + 0.4929437454961714, + 0.9180119741207207, + 1.361806663041953, + 1.850690097604851, + 0.6891928654922012, + 1.1669425562656648, + 1.766596230387042, + 2.3817477001769625, + 0.7504420676266258, + 1.4889471417635969, + 2.237983540758897, + 3.003279234042352 + ], + "population_cv": 0.6547435260478078 + }, + "bits_per_byte": 5.151101219064826, + "branch_output_rms_by_sublayer": [ + 0.0029161525890231133, + 0.0050826589576900005, + 0.002689435612410307, + 0.005224332213401794, + 0.0028427443467080593, + 0.005694052670150995, + 0.0029091176111251116, + 0.005875763949006796, + 0.003070527920499444, + 0.005908864550292492, + 0.0032786319497972727, + 0.005917144473642111, + 0.0038492560852319, + 0.005888325162231922, + 0.0033992670942097902, + 0.006673647556453943, + 0.003735452424734831, + 0.006282240152359009, + 0.004292168654501438, + 0.006417271681129932, + 0.004331387113779783, + 0.006861214991658926, + 0.004823344759643078, + 0.006506127770990133, + 0.00549117149785161, + 0.0067209163680672646, + 0.005449254531413317, + 0.0070295799523591995, + 0.005356284789741039, + 0.006968545261770487, + 0.005691505502909422, + 0.006837713532149792, + 0.005596098955720663, + 0.007023271638900042, + 0.007134208921343088, + 0.007149730809032917, + 0.006842488422989845, + 0.00750368507578969, + 0.007228665053844452, + 0.007692660670727491, + 0.007786077912896872, + 0.007865832187235355, + 0.008201404474675655, + 0.008177963085472584, + 0.008896155282855034, + 0.008069024421274662, + 0.009352477267384529, + 0.008248327299952507, + 0.012863222509622574, + 0.00803769938647747, + 0.009082870557904243, + 0.008133924566209316, + 0.012288292869925499, + 0.008494680747389793, + 0.012317892163991928, + 0.009096086025238037, + 0.013481991365551949, + 0.008436120115220547, + 0.014904185198247433, + 0.008798415772616863, + 0.015451964922249317, + 0.008850903250277042, + 0.014702510088682175, + 0.009197148494422436 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0003954594232341481, + 0.0003333032387489503, + 0.00031571017410000077, + 0.0003097616568970538, + 0.00028396677604728267, + 0.0002824472847093103, + 0.0002941768312455418, + 0.00026211853705911163, + 0.0002403917243366811, + 0.00025978209753505416, + 0.00023690847847974793, + 0.0002427592127380721, + 0.00024932863564548205, + 0.00023502181393964924, + 0.00020997685322941362, + 0.00020987503920984784, + 0.0001983266067999414, + 0.00020500259277806096, + 0.00019764042501953466, + 0.00018739122840003757, + 0.00018739378321478182, + 0.00017701547668010079, + 0.000174244442578282, + 0.00016985738702361125, + 0.00018511033607010563, + 0.00014915405042497035, + 0.0001725148886973076, + 0.00015873712285016152, + 0.00016129288507161482, + 0.00016917854188300805, + 0.00016987429730684002, + 0.00016216168690526595 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00030961799025517494, + "first_to_last_ratio": 1.86513517669251, + "imbalance_abs_log_ratio": 0.623333531269446, + "last_quartile_mean": 0.00016600297615115925, + "mean": 0.00022455886027684287, + "normalized": [ + 1.7610501885635415, + 1.4842577947627813, + 1.4059127914649363, + 1.3794230007899506, + 1.2645538710750488, + 1.2577873095771008, + 1.3100210380604524, + 1.1672598299437575, + 1.0705065212760654, + 1.156855254852946, + 1.0549950163965032, + 1.081049362464662, + 1.1103041551693942, + 1.0465933682149406, + 0.9350637644426405, + 0.934610368751906, + 0.8831831732465975, + 0.9129125099999512, + 0.8801274853990515, + 0.8344860147981517, + 0.8344973918364084, + 0.7882809721329669, + 0.7759410711448581, + 0.7564047431226093, + 0.8243288011076297, + 0.6642091531863351, + 0.7682390642908772, + 0.706884255889372, + 0.718265513428275, + 0.7533817266192022, + 0.7564800475804602, + 0.7221344404106264 + ], + "population_cv": 0.2651452660716966 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6930909156799316, + "mean_weights": [ + 0.5023669600486755, + 0.4976330101490021 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929057836532593, + "mean_weights": [ + 0.5102996230125427, + 0.48970043659210205 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928634643554688, + "mean_weights": [ + 0.5053070187568665, + 0.49469298124313354 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927847862243652, + "mean_weights": [ + 0.5126395225524902, + 0.4873604476451874 + ], + "sources": 2 + }, + { + "entropy_mean": 0.692810595035553, + "mean_weights": [ + 0.5030027031898499, + 0.49699729681015015 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927527189254761, + "mean_weights": [ + 0.5128813982009888, + 0.48711860179901123 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6925663948059082, + "mean_weights": [ + 0.5112606287002563, + 0.48873937129974365 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929250955581665, + "mean_weights": [ + 0.5095058679580688, + 0.4904940724372864 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0983479022979736, + "mean_weights": [ + 0.3398919701576233, + 0.3283834755420685, + 0.3317245543003082 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984477996826172, + "mean_weights": [ + 0.3407944142818451, + 0.3304300308227539, + 0.3287755250930786 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0980424880981445, + "mean_weights": [ + 0.3472509980201721, + 0.3313714563846588, + 0.32137754559516907 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985236167907715, + "mean_weights": [ + 0.3373773694038391, + 0.3341967463493347, + 0.32842591404914856 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982673168182373, + "mean_weights": [ + 0.34456542134284973, + 0.330670028924942, + 0.32476457953453064 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985300540924072, + "mean_weights": [ + 0.33718734979629517, + 0.33326292037963867, + 0.32954972982406616 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0983145236968994, + "mean_weights": [ + 0.34230899810791016, + 0.33276212215423584, + 0.3249289095401764 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984891653060913, + "mean_weights": [ + 0.3379484713077545, + 0.3344283998012543, + 0.3276231288909912 + ], + "sources": 3 + }, + { + "entropy_mean": 1.386104941368103, + "mean_weights": [ + 0.2562043368816376, + 0.2513892650604248, + 0.24579350650310516, + 0.24661290645599365 + ], + "sources": 4 + }, + { + "entropy_mean": 1.386183500289917, + "mean_weights": [ + 0.25370198488235474, + 0.25212907791137695, + 0.2472040355205536, + 0.24696491658687592 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3860962390899658, + "mean_weights": [ + 0.25710558891296387, + 0.2499506175518036, + 0.24624311923980713, + 0.2467007040977478 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862228393554688, + "mean_weights": [ + 0.25145405530929565, + 0.25281307101249695, + 0.24949496984481812, + 0.2462378889322281 + ], + "sources": 4 + }, + { + "entropy_mean": 1.386008858680725, + "mean_weights": [ + 0.2548893094062805, + 0.253750205039978, + 0.24855026602745056, + 0.2428102344274521 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862276077270508, + "mean_weights": [ + 0.2517310380935669, + 0.2530181407928467, + 0.24867194890975952, + 0.2465788871049881 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859810829162598, + "mean_weights": [ + 0.25464460253715515, + 0.25474923849105835, + 0.24874070286750793, + 0.24186545610427856 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862539529800415, + "mean_weights": [ + 0.25022467970848083, + 0.2521536350250244, + 0.24994289875030518, + 0.24767878651618958 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6090645790100098, + "mean_weights": [ + 0.20367035269737244, + 0.20636315643787384, + 0.19982853531837463, + 0.19553229212760925, + 0.19460564851760864 + ], + "sources": 5 + }, + { + "entropy_mean": 1.60930597782135, + "mean_weights": [ + 0.20176184177398682, + 0.2036270946264267, + 0.20011499524116516, + 0.19684121012687683, + 0.1976548433303833 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6086409091949463, + "mean_weights": [ + 0.2043638527393341, + 0.20835235714912415, + 0.204175665974617, + 0.19546180963516235, + 0.18764632940292358 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093453168869019, + "mean_weights": [ + 0.20028206706047058, + 0.20383989810943604, + 0.20016074180603027, + 0.19764459133148193, + 0.19807270169258118 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6086113452911377, + "mean_weights": [ + 0.20522160828113556, + 0.2102712094783783, + 0.20175999402999878, + 0.1938629150390625, + 0.18888425827026367 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093699932098389, + "mean_weights": [ + 0.20030194520950317, + 0.2033929079771042, + 0.2000882923603058, + 0.198019877076149, + 0.19819696247577667 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6083927154541016, + "mean_weights": [ + 0.20369869470596313, + 0.2109438180923462, + 0.20386341214179993, + 0.19415616989135742, + 0.18733789026737213 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093387603759766, + "mean_weights": [ + 0.20086133480072021, + 0.2033138871192932, + 0.20117565989494324, + 0.1990128755569458, + 0.19563624262809753 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7908570766448975, + "mean_weights": [ + 0.17161643505096436, + 0.17722433805465698, + 0.16894498467445374, + 0.16102741658687592, + 0.15854904055595398, + 0.16263779997825623 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7916772365570068, + "mean_weights": [ + 0.16707415878772736, + 0.16980010271072388, + 0.1685885488986969, + 0.16527867317199707, + 0.16461925208568573, + 0.16463926434516907 + ], + "sources": 6 + }, + { + "entropy_mean": 1.78986656665802, + "mean_weights": [ + 0.17284223437309265, + 0.18084144592285156, + 0.17343337833881378, + 0.16238240897655487, + 0.1569904088973999, + 0.15351013839244843 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917201519012451, + "mean_weights": [ + 0.1661807894706726, + 0.16888806223869324, + 0.16765862703323364, + 0.16659800708293915, + 0.1660696268081665, + 0.16460490226745605 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7891721725463867, + "mean_weights": [ + 0.17240555584430695, + 0.1838424652814865, + 0.1746402382850647, + 0.16218900680541992, + 0.15571486949920654, + 0.15120787918567657 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917195558547974, + "mean_weights": [ + 0.16665861010551453, + 0.16899651288986206, + 0.166007399559021, + 0.165146142244339, + 0.16656044125556946, + 0.16663089394569397 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7891744375228882, + "mean_weights": [ + 0.1722867786884308, + 0.1833745241165161, + 0.1744941771030426, + 0.16411003470420837, + 0.1555696725845337, + 0.15016481280326843 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458708763122559, + "mean_weights": [ + 0.1419069766998291, + 0.14182090759277344, + 0.14162558317184448, + 0.1428799033164978, + 0.1439807265996933, + 0.14333884418010712, + 0.14444705843925476 + ], + "sources": 7 + }, + { + "entropy_mean": 2.073298931121826, + "mean_weights": [ + 0.13280296325683594, + 0.14796042442321777, + 0.13775500655174255, + 0.1277381032705307, + 0.12050490081310272, + 0.11343234777450562, + 0.10916654765605927, + 0.11063969135284424 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0794126987457275, + "mean_weights": [ + 0.12473905086517334, + 0.12456962466239929, + 0.12423069775104523, + 0.12420755624771118, + 0.12484472990036011, + 0.12469400465488434, + 0.12634283304214478, + 0.12637150287628174 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0729880332946777, + "mean_weights": [ + 0.1322808861732483, + 0.14815035462379456, + 0.138286754488945, + 0.1285875141620636, + 0.12196782231330872, + 0.1147228479385376, + 0.11018145084381104, + 0.10582235455513 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0793216228485107, + "mean_weights": [ + 0.12419053167104721, + 0.1227085068821907, + 0.12359072268009186, + 0.1238015666604042, + 0.12408130615949631, + 0.12606750428676605, + 0.12765896320343018, + 0.1279008984565735 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0734729766845703, + "mean_weights": [ + 0.1313418447971344, + 0.14625218510627747, + 0.13754472136497498, + 0.1298069953918457, + 0.12240307033061981, + 0.11683905124664307, + 0.1108182817697525, + 0.10499383509159088 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0794219970703125, + "mean_weights": [ + 0.12476449459791183, + 0.12490230798721313, + 0.1242230087518692, + 0.12469767779111862, + 0.12560100853443146, + 0.12459909170866013, + 0.12595301866531372, + 0.1252593845129013 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0726284980773926, + "mean_weights": [ + 0.1318548023700714, + 0.1480155736207962, + 0.138479083776474, + 0.13002890348434448, + 0.12209554016590118, + 0.11552984267473221, + 0.10989700257778168, + 0.10409924387931824 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0793356895446777, + "mean_weights": [ + 0.12416703253984451, + 0.12249808013439178, + 0.12276653945446014, + 0.12436937540769577, + 0.1256466805934906, + 0.1261252462863922, + 0.12751197814941406, + 0.1269150674343109 + ], + "sources": 8 + }, + { + "entropy_mean": 2.190640687942505, + "mean_weights": [ + 0.11855027079582214, + 0.13270732760429382, + 0.12364236265420914, + 0.11760053038597107, + 0.11167599260807037, + 0.10436243563890457, + 0.09829548001289368, + 0.09473450481891632, + 0.09843108057975769 + ], + "sources": 9 + }, + { + "entropy_mean": 2.19718337059021, + "mean_weights": [ + 0.1108308807015419, + 0.11017795652151108, + 0.110158771276474, + 0.11043499410152435, + 0.11130029708147049, + 0.11165377497673035, + 0.1125313937664032, + 0.1118912398815155, + 0.11102068424224854 + ], + "sources": 9 + }, + { + "entropy_mean": 2.189920425415039, + "mean_weights": [ + 0.11906023323535919, + 0.13330945372581482, + 0.1238052248954773, + 0.11793403327465057, + 0.11261235177516937, + 0.10540484637022018, + 0.09936976432800293, + 0.09451640397310257, + 0.09398768842220306 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197188377380371, + "mean_weights": [ + 0.1106625646352768, + 0.11019707471132278, + 0.11037462949752808, + 0.111014723777771, + 0.11012890934944153, + 0.11142671853303909, + 0.11245337873697281, + 0.11143884807825089, + 0.11230315268039703 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1896934509277344, + "mean_weights": [ + 0.11968642473220825, + 0.13407054543495178, + 0.12359777092933655, + 0.11806301027536392, + 0.1115856021642685, + 0.10530231893062592, + 0.09987497329711914, + 0.0946875587105751, + 0.09313178062438965 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197195291519165, + "mean_weights": [ + 0.11115474253892899, + 0.11171463131904602, + 0.11140602827072144, + 0.1111491322517395, + 0.11216610670089722, + 0.11134834587574005, + 0.11161454021930695, + 0.10974055528640747, + 0.10970589518547058 + ], + "sources": 9 + }, + { + "entropy_mean": 2.18862247467041, + "mean_weights": [ + 0.11986692249774933, + 0.13584880530834198, + 0.12448455393314362, + 0.11772501468658447, + 0.11266382783651352, + 0.10465173423290253, + 0.09914892911911011, + 0.0941760241985321, + 0.09143419563770294 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.02857154980301857, + 0.014362954534590244, + 0.015028384514153004, + 0.014918262138962746, + 0.015755506232380867, + 0.015599310398101807, + 0.016723202541470528, + 0.01673182286322117, + 0.017851572483778, + 0.012046296149492264, + 0.012996996752917767, + 0.013207979500293732, + 0.014043750241398811, + 0.01420951820909977, + 0.015203217975795269, + 0.015378003008663654, + 0.016682690009474754, + 0.012674083933234215, + 0.013626845553517342, + 0.013717686757445335, + 0.014743366278707981, + 0.014969867654144764, + 0.01602025330066681, + 0.016199681907892227, + 0.017333699390292168, + 0.014109236188232899, + 0.014979767613112926, + 0.015222674235701561, + 0.016162078827619553, + 0.016379114240407944, + 0.01737755723297596, + 0.01751699671149254, + 0.018478207290172577, + 0.015605764463543892, + 0.01646902784705162, + 0.016765577718615532, + 0.017649497836828232, + 0.017812248319387436, + 0.018887655809521675, + 0.019008813425898552, + 0.020286723971366882, + 0.01771627552807331, + 0.01848747208714485, + 0.018903981894254684, + 0.019707493484020233, + 0.02024862915277481, + 0.02107124961912632, + 0.02168232575058937, + 0.022651705890893936, + 0.01989040896296501, + 0.02128136157989502, + 0.02106988988816738, + 0.022691315039992332, + 0.02241489477455616, + 0.024193689227104187, + 0.023614881560206413, + 0.02602931298315525, + 0.02272861637175083, + 0.024663671851158142, + 0.02417246252298355, + 0.026352496817708015, + 0.02557525597512722, + 0.027971701696515083, + 0.026890767738223076 + ], + "loss_nats": 3.5704712867736816, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.193127155303955, + "mean_weights": [ + 0.10074253380298615, + 0.09434226155281067, + 0.1019720584154129, + 0.10976769030094147, + 0.11436429619789124, + 0.11838443577289581, + 0.11974147707223892, + 0.12107785046100616, + 0.11960738897323608 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.0029161525890231133, + 0.00586781045421958, + 0.006538637448102236, + 0.010147182270884514, + 0.01062131579965353, + 0.014828646555542946, + 0.015123440884053707, + 0.019574908539652824, + 0.003070527920499444, + 0.006723169703036547, + 0.007610566448420286, + 0.011927195824682713, + 0.012688657268881798, + 0.0171233918517828, + 0.017755819484591484, + 0.0229953546077013, + 0.003735452424734831, + 0.007492356467992067, + 0.009091529995203018, + 0.013753283768892288, + 0.015125916339457035, + 0.02036694437265396, + 0.02174314856529236, + 0.026791520416736603, + 0.00549117149785161, + 0.009156237356364727, + 0.011835021898150444, + 0.016712356358766556, + 0.01893516816198826, + 0.024189556017518044, + 0.026110194623470306, + 0.03125955909490585, + 0.005596098955720663, + 0.009814711287617683, + 0.013655281625688076, + 0.018578259274363518, + 0.021974150091409683, + 0.02773377113044262, + 0.0315692275762558, + 0.03759535774588585, + 0.007786077912896872, + 0.012513305060565472, + 0.01763547956943512, + 0.023303600028157234, + 0.028473734855651855, + 0.03456926345825195, + 0.04046574607491493, + 0.046979498118162155, + 0.012863222509622574, + 0.017495060339570045, + 0.023920785635709763, + 0.029622666537761688, + 0.03841462358832359, + 0.04484478756785393, + 0.05333936959505081, + 0.06046031787991524, + 0.013481991365551949, + 0.019049862399697304, + 0.03141481801867485, + 0.03779670596122742, + 0.0499507300555706, + 0.0568108856678009, + 0.06896942108869553, + 0.07623780518770218 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00014786201063543558, + 0.00015257071936503053, + 0.0001520643854746595, + 0.0001454792363801971, + 0.00014023136463947594, + 0.00012602606147993356, + 0.00011509178875712678, + 0.00010517338523641229, + 0.00010125816334038973, + 9.275165939470753e-05, + 8.588869968662038e-05, + 7.99827539594844e-05, + 8.227435318985954e-05, + 7.597146759508178e-05, + 7.000737241469324e-05, + 6.430525536416098e-05, + 6.201478390721604e-05, + 5.615992995444685e-05, + 5.051898187957704e-05, + 4.627195085049607e-05, + 4.4271579099586233e-05, + 4.099612851860002e-05, + 3.7980073102517053e-05, + 3.562766869436018e-05, + 3.343720163684338e-05, + 3.15015458909329e-05, + 3.0464896553894505e-05, + 2.9766553780063987e-05, + 2.5172859750455245e-05, + 2.496439810784068e-05, + 2.4838445824570954e-05, + 2.4838747776811942e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.0001355623689960339, + "first_to_last_ratio": 4.820324209848439, + "imbalance_abs_log_ratio": 1.5728411892544574, + "last_quartile_mean": 2.81230811651767e-05, + "mean": 7.299263819504631e-05, + "normalized": [ + 2.0257112811887694, + 2.0902206460511907, + 2.083283865810175, + 1.9930672459249479, + 1.9211713414818432, + 1.7265585214658865, + 1.5767588568258868, + 1.4408766121779928, + 1.3872380262488124, + 1.270698822350583, + 1.1766761937979833, + 1.0957646680170614, + 1.1271596043701178, + 1.040809997743548, + 0.9591018237705556, + 0.8809827532514792, + 0.8496032674076536, + 0.7693916995352303, + 0.6921106447006808, + 0.6339262697540962, + 0.6065211532879045, + 0.5616474418838343, + 0.5203274472835069, + 0.48809947928115954, + 0.4580900548832693, + 0.4315715484451523, + 0.4173694403603845, + 0.4078021361623303, + 0.3448684740396685, + 0.3420125470890968, + 0.3402869993299767, + 0.3402911360792223 + ], + "population_cv": 0.580526022714296 + }, + "activation_output_rms_by_block": [ + 0.021685069426894188, + 0.038405608385801315, + 0.04965895041823387, + 0.058261532336473465, + 0.020244605839252472, + 0.027559109032154083, + 0.034344226121902466, + 0.04113885387778282, + 0.018506815657019615, + 0.028349323198199272, + 0.0385899543762207, + 0.04041825234889984, + 0.02097114361822605, + 0.0294428039342165, + 0.0372258760035038, + 0.04483914375305176, + 0.026322925463318825, + 0.03495544195175171, + 0.04254264011979103, + 0.05138538405299187, + 0.02588142827153206, + 0.037761904299259186, + 0.050309453159570694, + 0.06338406354188919, + 0.028033528476953506, + 0.05220331251621246, + 0.07662547379732132, + 0.10279989242553711, + 0.028420623391866684, + 0.05363891273736954, + 0.0855264663696289, + 0.11839374154806137 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.036412244429811835, + "first_to_last_ratio": 0.533862828479834, + "imbalance_abs_log_ratio": 0.6276163485260349, + "last_quartile_mean": 0.06820524390786886, + "mean": 0.04461957688909024, + "normalized": [ + 0.4859989927918012, + 0.8607344816663136, + 1.1129408631926268, + 1.3057392382113504, + 0.45371577345016023, + 0.6176461309944078, + 0.7697120527895422, + 0.9219911246590401, + 0.4147689634758649, + 0.635356163700666, + 0.8648659863391978, + 0.9058412286016623, + 0.4699987108876782, + 0.6598629119097597, + 0.8342946885407547, + 1.004920863873436, + 0.5899411715343931, + 0.7834104307765978, + 0.9534523428032095, + 1.1516331537773032, + 0.5800464745747114, + 0.8463079870327548, + 1.1275197271507893, + 1.4205438052324288, + 0.6282786711903554, + 1.1699643106426787, + 1.7173061498944138, + 2.3039190326941963, + 0.6369541213380645, + 1.2021385337359525, + 1.9167924111476873, + 2.6534035013902018 + ], + "population_cv": 0.5178760114633159 + }, + "bits_per_byte": 3.0604126979773865, + "branch_output_rms_by_sublayer": [ + 0.011115886270999908, + 0.017353586852550507, + 0.01344362087547779, + 0.014506204053759575, + 0.012044745497405529, + 0.01118853222578764, + 0.012206755578517914, + 0.010463034734129906, + 0.018151557072997093, + 0.008262216113507748, + 0.015186779201030731, + 0.0077690985053777695, + 0.01374769676476717, + 0.007438337430357933, + 0.017600182443857193, + 0.008333330042660236, + 0.01612248457968235, + 0.008349443785846233, + 0.015507777221500874, + 0.008093987591564655, + 0.022292258217930794, + 0.008150536566972733, + 0.020381825044751167, + 0.008793518878519535, + 0.01833919808268547, + 0.008415265008807182, + 0.013371670618653297, + 0.009308703243732452, + 0.022054089233279228, + 0.009577617980539799, + 0.02499178610742092, + 0.00979722011834383, + 0.02342415228486061, + 0.010345900431275368, + 0.021941926330327988, + 0.010646623559296131, + 0.015714256092905998, + 0.0121941938996315, + 0.019414687529206276, + 0.013931936584413052, + 0.02173462137579918, + 0.013224986381828785, + 0.015238175168633461, + 0.014334730803966522, + 0.014166571199893951, + 0.01547280978411436, + 0.016237368807196617, + 0.018746625632047653, + 0.013608172535896301, + 0.02215754985809326, + 0.012524046935141087, + 0.02242971770465374, + 0.008981638588011265, + 0.02420484833419323, + 0.00850011594593525, + 0.025562811642885208, + 0.008029460906982422, + 0.0262075737118721, + 0.008068720810115337, + 0.024754289537668228, + 0.00842621736228466, + 0.030781565234065056, + 0.010052710771560669, + 0.029881317168474197 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.000493691296811525, + 0.00037860665568120966, + 0.00032615539278804136, + 0.0003052732999682161, + 0.00028042716522812716, + 0.0002610519864576601, + 0.0002015769759679097, + 0.0002015044090367299, + 0.00019209683136949964, + 0.0001751665905743024, + 0.00018614945436074937, + 0.00019313381129206687, + 0.0002037332089169536, + 0.0001916150635045059, + 0.00021036966149018068, + 0.00017743998036797047, + 0.00015736598315833536, + 0.0001377233363808635, + 0.00012862880656683382, + 0.00012613996173978053, + 0.0001255982800983756, + 0.00010951373774902995, + 0.00010649264888506773, + 0.00010706355151573251, + 0.00010991083755897496, + 0.00010272299622984276, + 9.344374311031332e-05, + 9.108164084614429e-05, + 8.800096678772191e-05, + 7.663319726390902e-05, + 9.790912949431833e-05, + 9.124203497204383e-05 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0003060358977424274, + "first_to_last_ratio": 3.2602769327271885, + "imbalance_abs_log_ratio": 1.1818121404600024, + "last_quartile_mean": 9.386806828290856e-05, + "mean": 0.00017898320738040423, + "normalized": [ + 2.7583107043235175, + 2.115319427014922, + 1.822268119795411, + 1.7055974380847898, + 1.566779193045288, + 1.458527815421421, + 1.1262340133367121, + 1.1258285734508038, + 1.073267342679943, + 0.9786761179332866, + 1.0400386554986372, + 1.0790610701348506, + 1.138281137648554, + 1.0705756495761884, + 1.1753597701658618, + 0.991377811164408, + 0.8792220536302915, + 0.7694763011378574, + 0.7186641051383718, + 0.704758639572747, + 0.7017322012306646, + 0.6118659920775334, + 0.5949868171639825, + 0.5981765165722147, + 0.6140846349086513, + 0.5739253292713609, + 0.5220810626759609, + 0.5088837225526011, + 0.49167163822630544, + 0.4281585875318218, + 0.5470296958430626, + 0.5097798631919777 + ], + "population_cv": 0.5235566735473837 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6502585411071777, + "mean_weights": [ + 0.37342309951782227, + 0.6265769004821777 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6666876673698425, + "mean_weights": [ + 0.4218984544277191, + 0.5781015157699585 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6878700256347656, + "mean_weights": [ + 0.5173935890197754, + 0.4826064109802246 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6705846786499023, + "mean_weights": [ + 0.44550156593322754, + 0.5544984340667725 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6742776036262512, + "mean_weights": [ + 0.43235528469085693, + 0.5676447153091431 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6733695864677429, + "mean_weights": [ + 0.4424605369567871, + 0.5575394630432129 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6806321144104004, + "mean_weights": [ + 0.4543085992336273, + 0.5456913709640503 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6758021116256714, + "mean_weights": [ + 0.44607001543045044, + 0.5539299249649048 + ], + "sources": 2 + }, + { + "entropy_mean": 1.08938729763031, + "mean_weights": [ + 0.304912805557251, + 0.3580775856971741, + 0.33700960874557495 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0809125900268555, + "mean_weights": [ + 0.29451984167099, + 0.3244999349117279, + 0.3809801936149597 + ], + "sources": 3 + }, + { + "entropy_mean": 1.091600775718689, + "mean_weights": [ + 0.3001934289932251, + 0.336348295211792, + 0.3634582757949829 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0666152238845825, + "mean_weights": [ + 0.296264111995697, + 0.2936059832572937, + 0.41012993454933167 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0877625942230225, + "mean_weights": [ + 0.2854554057121277, + 0.35340386629104614, + 0.36114075779914856 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0701894760131836, + "mean_weights": [ + 0.2979707419872284, + 0.30030208826065063, + 0.4017271399497986 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0924328565597534, + "mean_weights": [ + 0.30129799246788025, + 0.33185315132141113, + 0.3668488562107086 + ], + "sources": 3 + }, + { + "entropy_mean": 1.06105375289917, + "mean_weights": [ + 0.29679009318351746, + 0.28642240166664124, + 0.4167875051498413 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3800113201141357, + "mean_weights": [ + 0.22949735820293427, + 0.23836687207221985, + 0.2620571255683899, + 0.2700786590576172 + ], + "sources": 4 + }, + { + "entropy_mean": 1.358888864517212, + "mean_weights": [ + 0.23578974604606628, + 0.22542984783649445, + 0.2510915696620941, + 0.28768888115882874 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3815139532089233, + "mean_weights": [ + 0.23799902200698853, + 0.2338489294052124, + 0.2521052956581116, + 0.2760467529296875 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3525551557540894, + "mean_weights": [ + 0.238496333360672, + 0.21666303277015686, + 0.23646780848503113, + 0.3083727955818176 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3820223808288574, + "mean_weights": [ + 0.24487152695655823, + 0.23898011445999146, + 0.2461952120065689, + 0.2699531316757202 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3493120670318604, + "mean_weights": [ + 0.25296124815940857, + 0.21807506680488586, + 0.22561275959014893, + 0.30335092544555664 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3808975219726562, + "mean_weights": [ + 0.2564350366592407, + 0.22641341388225555, + 0.24222421646118164, + 0.2749273180961609 + ], + "sources": 4 + }, + { + "entropy_mean": 1.341019868850708, + "mean_weights": [ + 0.24514153599739075, + 0.21444647014141083, + 0.2206835299730301, + 0.31972843408584595 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6035922765731812, + "mean_weights": [ + 0.2080087959766388, + 0.18397900462150574, + 0.1826881319284439, + 0.20116961002349854, + 0.22415447235107422 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5572702884674072, + "mean_weights": [ + 0.1911628693342209, + 0.17380034923553467, + 0.16380442678928375, + 0.19770289957523346, + 0.27352944016456604 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6064553260803223, + "mean_weights": [ + 0.20122665166854858, + 0.1979619562625885, + 0.18347720801830292, + 0.19662447273731232, + 0.22070971131324768 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5644636154174805, + "mean_weights": [ + 0.19563999772071838, + 0.17608843743801117, + 0.16662916541099548, + 0.188569575548172, + 0.27307283878326416 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6009154319763184, + "mean_weights": [ + 0.20550566911697388, + 0.1936512291431427, + 0.17488721013069153, + 0.1910301297903061, + 0.23492573201656342 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5732216835021973, + "mean_weights": [ + 0.19769304990768433, + 0.18482983112335205, + 0.16873934864997864, + 0.18331405520439148, + 0.2654237449169159 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5981827974319458, + "mean_weights": [ + 0.21186822652816772, + 0.1905551701784134, + 0.17512747645378113, + 0.18633750081062317, + 0.23611164093017578 + ], + "sources": 5 + }, + { + "entropy_mean": 1.579944133758545, + "mean_weights": [ + 0.20677590370178223, + 0.18372760713100433, + 0.16864405572414398, + 0.18130826950073242, + 0.25954416394233704 + ], + "sources": 5 + }, + { + "entropy_mean": 1.783144235610962, + "mean_weights": [ + 0.160476952791214, + 0.1750788390636444, + 0.15045803785324097, + 0.15342845022678375, + 0.17806783318519592, + 0.18248987197875977 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7663768529891968, + "mean_weights": [ + 0.1643587350845337, + 0.1559007167816162, + 0.14218389987945557, + 0.14312618970870972, + 0.188603013753891, + 0.20582742989063263 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7793965339660645, + "mean_weights": [ + 0.15385980904102325, + 0.1699720323085785, + 0.1493702530860901, + 0.14805768430233002, + 0.17362293601036072, + 0.20511728525161743 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7721542119979858, + "mean_weights": [ + 0.15929150581359863, + 0.15980638563632965, + 0.1524408459663391, + 0.14724723994731903, + 0.1665390431880951, + 0.2146749645471573 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7706701755523682, + "mean_weights": [ + 0.14818884432315826, + 0.1793207824230194, + 0.1466081440448761, + 0.1432226002216339, + 0.17000678181648254, + 0.21265284717082977 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7799956798553467, + "mean_weights": [ + 0.1690046787261963, + 0.1627887636423111, + 0.15251469612121582, + 0.1492576152086258, + 0.16148751974105835, + 0.20494674146175385 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7713623046875, + "mean_weights": [ + 0.14060063660144806, + 0.18941815197467804, + 0.1547119915485382, + 0.14588099718093872, + 0.1667281985282898, + 0.20266000926494598 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9297157526016235, + "mean_weights": [ + 0.1447434425354004, + 0.1588355451822281, + 0.14611759781837463, + 0.1339455097913742, + 0.12466445565223694, + 0.13665539026260376, + 0.15503805875778198 + ], + "sources": 7 + }, + { + "entropy_mean": 2.0422916412353516, + "mean_weights": [ + 0.10117341578006744, + 0.17748653888702393, + 0.14483100175857544, + 0.12140035629272461, + 0.1212053894996643, + 0.1212303638458252, + 0.11800158768892288, + 0.0946713536977768 + ], + "sources": 8 + }, + { + "entropy_mean": 2.063138961791992, + "mean_weights": [ + 0.12125496566295624, + 0.14725780487060547, + 0.13530178368091583, + 0.12124410271644592, + 0.11280997842550278, + 0.11582435667514801, + 0.1269630342721939, + 0.11934398114681244 + ], + "sources": 8 + }, + { + "entropy_mean": 2.043181896209717, + "mean_weights": [ + 0.10160650312900543, + 0.17736735939979553, + 0.14586785435676575, + 0.12408211827278137, + 0.12062656879425049, + 0.1164516806602478, + 0.1159423366189003, + 0.09805557131767273 + ], + "sources": 8 + }, + { + "entropy_mean": 2.068783760070801, + "mean_weights": [ + 0.11873521655797958, + 0.1428414285182953, + 0.13369877636432648, + 0.12150336802005768, + 0.11530910432338715, + 0.1198006272315979, + 0.12850314378738403, + 0.1196083277463913 + ], + "sources": 8 + }, + { + "entropy_mean": 2.039012908935547, + "mean_weights": [ + 0.09841868281364441, + 0.180045485496521, + 0.14344395697116852, + 0.12553668022155762, + 0.1265493929386139, + 0.12144944071769714, + 0.11547509580850601, + 0.0890812799334526 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0636391639709473, + "mean_weights": [ + 0.11348826438188553, + 0.15112051367759705, + 0.1422683149576187, + 0.1266966015100479, + 0.1152118667960167, + 0.1184190958738327, + 0.12351632118225098, + 0.10927900671958923 + ], + "sources": 8 + }, + { + "entropy_mean": 2.022876501083374, + "mean_weights": [ + 0.09419430047273636, + 0.1906050443649292, + 0.15066608786582947, + 0.12728895246982574, + 0.12621371448040009, + 0.1195271909236908, + 0.10970081388950348, + 0.08180391788482666 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0533299446105957, + "mean_weights": [ + 0.11233378946781158, + 0.15135176479816437, + 0.1341555416584015, + 0.12586581707000732, + 0.11997333914041519, + 0.13130219280719757, + 0.12335589528083801, + 0.10166166722774506 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1375138759613037, + "mean_weights": [ + 0.08329890668392181, + 0.16995769739151, + 0.13881553709506989, + 0.11610452085733414, + 0.11625030636787415, + 0.10720990598201752, + 0.09815935790538788, + 0.07008018344640732, + 0.10012362152338028 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1873247623443604, + "mean_weights": [ + 0.10463990271091461, + 0.13219711184501648, + 0.11872637271881104, + 0.1127173900604248, + 0.10895145684480667, + 0.10880295932292938, + 0.11222322285175323, + 0.10099013149738312, + 0.10075145959854126 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1242246627807617, + "mean_weights": [ + 0.08451886475086212, + 0.18359936773777008, + 0.14383737742900848, + 0.11585594713687897, + 0.11934906989336014, + 0.10933604836463928, + 0.10416778922080994, + 0.07247121632099152, + 0.06686431169509888 + ], + "sources": 9 + }, + { + "entropy_mean": 2.181307792663574, + "mean_weights": [ + 0.10815304517745972, + 0.12958988547325134, + 0.11978799104690552, + 0.11249610781669617, + 0.10450916737318039, + 0.10727362334728241, + 0.11092618107795715, + 0.10497796535491943, + 0.10228604823350906 + ], + "sources": 9 + }, + { + "entropy_mean": 2.107302665710449, + "mean_weights": [ + 0.08096957206726074, + 0.18907985091209412, + 0.15208786725997925, + 0.1220867782831192, + 0.1218317449092865, + 0.11133752763271332, + 0.09774941951036453, + 0.0660695880651474, + 0.05878765136003494 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1502392292022705, + "mean_weights": [ + 0.09802278876304626, + 0.14902228116989136, + 0.12074492871761322, + 0.12311965227127075, + 0.12527799606323242, + 0.11980535835027695, + 0.10616443306207657, + 0.080390065908432, + 0.07745248079299927 + ], + "sources": 9 + }, + { + "entropy_mean": 2.119602680206299, + "mean_weights": [ + 0.0835924744606018, + 0.1813923716545105, + 0.14474964141845703, + 0.12106956541538239, + 0.12520721554756165, + 0.11548885703086853, + 0.10013826936483383, + 0.06881266832351685, + 0.05954894423484802 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.043919675052165985, + 0.017475632950663567, + 0.023797158151865005, + 0.02784808911383152, + 0.030734635889530182, + 0.032344117760658264, + 0.035305511206388474, + 0.036033373326063156, + 0.038435060530900955, + 0.02621471881866455, + 0.026893306523561478, + 0.027507523074746132, + 0.028129389509558678, + 0.029366331174969673, + 0.030022980645298958, + 0.03042595274746418, + 0.03193489462137222, + 0.023309199139475822, + 0.024921594187617302, + 0.024023745208978653, + 0.025641661137342453, + 0.024866819381713867, + 0.02685319446027279, + 0.025358302518725395, + 0.027794385328888893, + 0.02052314579486847, + 0.022138359025120735, + 0.020911119878292084, + 0.022523224353790283, + 0.021444378420710564, + 0.023505374789237976, + 0.021986229345202446, + 0.023524222895503044, + 0.018526768311858177, + 0.01963404379785061, + 0.01888965256512165, + 0.019959721714258194, + 0.01965916156768799, + 0.020321443676948547, + 0.020545918494462967, + 0.021000389009714127, + 0.01799594983458519, + 0.018720002844929695, + 0.018756654113531113, + 0.01965489238500595, + 0.01959010399878025, + 0.020632389932870865, + 0.02053764835000038, + 0.021935993805527687, + 0.019902747124433517, + 0.020733026787638664, + 0.020901493728160858, + 0.02278776839375496, + 0.022159084677696228, + 0.024183709174394608, + 0.023153379559516907, + 0.026084303855895996, + 0.021790621802210808, + 0.025988241657614708, + 0.02399749867618084, + 0.028371188789606094, + 0.02453809417784214, + 0.0279060248285532, + 0.026172436773777008 + ], + "loss_nats": 2.121316432952881, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.167670726776123, + "mean_weights": [ + 0.1225091814994812, + 0.0901913046836853, + 0.09324903041124344, + 0.09737822413444519, + 0.08995121717453003, + 0.09802557528018951, + 0.12589603662490845, + 0.15228068828582764, + 0.13051873445510864 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.011115886270999908, + 0.021685069426894188, + 0.0288649033755064, + 0.038405608385801315, + 0.04274377226829529, + 0.04965895041823387, + 0.052245546132326126, + 0.058261532336473465, + 0.018151557072997093, + 0.020244605839252472, + 0.02427005022764206, + 0.027559109032154083, + 0.031013716012239456, + 0.034344226121902466, + 0.03717077895998955, + 0.04113885387778282, + 0.01612248457968235, + 0.018506815657019615, + 0.02518109604716301, + 0.028349323198199272, + 0.03504186496138573, + 0.0385899543762207, + 0.03596026822924614, + 0.04041825234889984, + 0.01833919808268547, + 0.02097114361822605, + 0.025207646191120148, + 0.0294428039342165, + 0.03227776661515236, + 0.0372258760035038, + 0.039784155786037445, + 0.04483914375305176, + 0.02342415228486061, + 0.026322925463318825, + 0.03019019030034542, + 0.03495544195175171, + 0.03634876012802124, + 0.04254264011979103, + 0.04333419352769852, + 0.05138538405299187, + 0.02173462137579918, + 0.02588142827153206, + 0.03048042207956314, + 0.037761904299259186, + 0.040726594626903534, + 0.050309453159570694, + 0.04981967434287071, + 0.06338406354188919, + 0.013608172535896301, + 0.028033528476953506, + 0.03425193950533867, + 0.05220331251621246, + 0.05559886246919632, + 0.07662547379732132, + 0.08002970367670059, + 0.10279989242553711, + 0.008029460906982422, + 0.028420623391866684, + 0.0311754010617733, + 0.05363891273736954, + 0.0571460984647274, + 0.0855264663696289, + 0.09072235971689224, + 0.11839374154806137 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.213267963379622e-05, + 7.908487168606371e-05, + 7.468609692296013e-05, + 6.886423216201365e-05, + 7.683190051466227e-05, + 7.24359997548163e-05, + 6.696541822748259e-05, + 6.180260970722884e-05, + 5.952622814220376e-05, + 5.6441014748997986e-05, + 5.3769705118611455e-05, + 5.196155689191073e-05, + 5.43221685802564e-05, + 5.190579759073444e-05, + 4.935182732879184e-05, + 4.715508839581162e-05, + 5.236389915808104e-05, + 4.964864274370484e-05, + 5.0703321903711185e-05, + 4.973597970092669e-05, + 6.313752965070307e-05, + 6.067507274565287e-05, + 5.615585541818291e-05, + 5.247291846899316e-05, + 0.00011703887139447033, + 9.17082215892151e-05, + 6.402079452527687e-05, + 5.0757735152728856e-05, + 2.912170566560235e-05, + 2.5578379791113548e-05, + 2.483207754266914e-05, + 2.442610821162816e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 7.160047607612796e-05, + "first_to_last_ratio": 1.3399424324968663, + "imbalance_abs_log_ratio": 0.2926266522167433, + "last_quartile_mean": 5.3435486734088045e-05, + "mean": 5.8112947158406314e-05, + "normalized": [ + 1.2412497242167813, + 1.3608821364796968, + 1.2851885952260886, + 1.185006707271292, + 1.3221133029999523, + 1.2464692172188014, + 1.1523321652392864, + 1.0634912309431686, + 1.024319554469421, + 0.9712296056014692, + 0.9252620585915924, + 0.8941476802109536, + 0.9347687776388819, + 0.8931881814434189, + 0.8492397949508033, + 0.8114385984808965, + 0.9010711333456498, + 0.8543473558202241, + 0.8724961369710313, + 0.8558502387661499, + 1.0864623588716051, + 1.044088722264638, + 0.9663226211038877, + 0.9029471234002404, + 2.013989603305468, + 1.5781030918847327, + 1.1016614654005885, + 0.8734324730489341, + 0.5011225052176649, + 0.4401494166418905, + 0.4273071450838828, + 0.4203212778909082 + ], + "population_cv": 0.3206904000003418 + }, + "activation_output_rms_by_block": [ + 0.08136755228042603, + 0.16205471754074097, + 0.19161629676818848, + 0.20171724259853363, + 0.0805743932723999, + 0.09270916879177094, + 0.10289278626441956, + 0.1131928563117981, + 0.041872475296258926, + 0.07351861149072647, + 0.09840589761734009, + 0.12142564356327057, + 0.041982922703027725, + 0.07002262026071548, + 0.09207555651664734, + 0.11870673298835754, + 0.03839530423283577, + 0.0703679770231247, + 0.09554289281368256, + 0.13110701739788055, + 0.04829283058643341, + 0.0819089487195015, + 0.12588800489902496, + 0.17493852972984314, + 0.04844440519809723, + 0.07240089774131775, + 0.10001245886087418, + 0.13743279874324799, + 0.06662143021821976, + 0.13172371685504913, + 0.22823166847229004, + 0.3327999711036682 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.1282656267285347, + "first_to_last_ratio": 0.9180951885241948, + "imbalance_abs_log_ratio": 0.08545420253438119, + "last_quartile_mean": 0.13970841839909554, + "mean": 0.11150763521436602, + "normalized": [ + 0.729703863991038, + 1.453306020069402, + 1.7184141372904096, + 1.8089993766861403, + 0.7225908173687029, + 0.8314154327956441, + 0.9227420711291654, + 1.015113055659304, + 0.3755121809888792, + 0.6593145962551518, + 0.8825036727589217, + 1.0889446568375147, + 0.37650267286467287, + 0.627962561721461, + 0.8257332005977726, + 1.0645614783252442, + 0.34432892562938267, + 0.6310597197030224, + 0.8568282578139846, + 1.175767176353763, + 0.43308990001979375, + 0.7345591049620681, + 1.128963094383749, + 1.5688479931758523, + 0.4344492204947767, + 0.6492909440876568, + 0.8969113071818523, + 1.2324967566484804, + 0.5974607038356117, + 1.1812977344718956, + 2.046780635545986, + 2.9845487303527007 + ], + "population_cv": 0.5515965706831862 + }, + "bits_per_byte": 2.0768734987089323, + "branch_output_rms_by_sublayer": [ + 0.04231308028101921, + 0.06680140644311905, + 0.09043751657009125, + 0.03189544752240181, + 0.05159430205821991, + 0.009685513563454151, + 0.03747963160276413, + 0.017089465633034706, + 0.07905919849872589, + 0.00861464161425829, + 0.024363122880458832, + 0.00790475308895111, + 0.026023156940937042, + 0.011602269485592842, + 0.029838817194104195, + 0.019409971311688423, + 0.036441486328840256, + 0.01817169226706028, + 0.03428736329078674, + 0.020851682871580124, + 0.0240671094506979, + 0.024081697687506676, + 0.021821660920977592, + 0.02550157718360424, + 0.03355863690376282, + 0.022015687078237534, + 0.029155723750591278, + 0.026791134849190712, + 0.02972303330898285, + 0.02735939621925354, + 0.03249407559633255, + 0.029932642355561256, + 0.02821013703942299, + 0.026344390586018562, + 0.03260891139507294, + 0.029566330835223198, + 0.0326685905456543, + 0.02734629437327385, + 0.04758678004145622, + 0.0394001305103302, + 0.042858436703681946, + 0.02518589235842228, + 0.04895510524511337, + 0.02630355954170227, + 0.0660373717546463, + 0.03576064482331276, + 0.06855079531669617, + 0.05714043229818344, + 0.02350943349301815, + 0.03930946812033653, + 0.01442041527479887, + 0.040821488946676254, + 0.01240013912320137, + 0.04926081746816635, + 0.010679885745048523, + 0.060121625661849976, + 0.022145941853523254, + 0.061492349952459335, + 0.01637289859354496, + 0.07599335163831711, + 0.017017493024468422, + 0.10882309824228287, + 0.02517884038388729, + 0.11236122250556946 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00026940881003397914, + 0.00020379702035832085, + 0.0001486251166353295, + 0.00013657858379850234, + 0.0001651067947050195, + 0.00010544414134719301, + 0.00010932433829049554, + 0.00015239622318311436, + 0.00013170724976024442, + 0.0001144252345619374, + 0.00011516888535000073, + 0.00011520899413821027, + 0.00011486680798184195, + 0.00012519369092180727, + 0.00013514532695912352, + 0.00014437119586330744, + 0.00014268717731085644, + 0.000154831588527606, + 0.00015723536628588556, + 0.00021468975684196855, + 0.00020929143621623483, + 0.00024598212313055285, + 0.00030083688593857995, + 0.0003166404031390106, + 0.0005018896210809729, + 0.0003947618368989273, + 0.0003101624393107948, + 0.0002567758503413981, + 0.00014176931525439556, + 0.00013117000010030765, + 0.00013272949060834577, + 0.00013041607258254333 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00016133512854399427, + "first_to_last_ratio": 0.6454455197138996, + "imbalance_abs_log_ratio": 0.4378144723907171, + "last_quartile_mean": 0.0002499593282722107, + "mean": 0.00018839493054552522, + "normalized": [ + 1.4300215470441073, + 1.0817542689083863, + 0.788901889265086, + 0.7249589115960763, + 0.8763866109715824, + 0.5596973392111135, + 0.5802934185857912, + 0.8089189169890543, + 0.6991018780540788, + 0.6073689681065318, + 0.6113162653395836, + 0.6115291627253753, + 0.6097128391365321, + 0.6645279178122815, + 0.7173511865090549, + 0.766322084385495, + 0.7573833165132667, + 0.8218458324715446, + 0.8346050810952685, + 1.1395728973189607, + 1.1109186197855787, + 1.305672729187964, + 1.5968417253450637, + 1.6807267702062456, + 2.6640293325710926, + 2.095395219796182, + 1.6463417482236589, + 1.3629658165316119, + 0.7525113061369627, + 0.6962501576899423, + 0.7045279308949982, + 0.6922483115915297 + ], + "population_cv": 0.48722333616824764 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.48636314272880554, + "mean_weights": [ + 0.7935467958450317, + 0.20645320415496826 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6328556537628174, + "mean_weights": [ + 0.5086358785629272, + 0.49136412143707275 + ], + "sources": 2 + }, + { + "entropy_mean": 0.4702409505844116, + "mean_weights": [ + 0.8140403032302856, + 0.18595963716506958 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6865477561950684, + "mean_weights": [ + 0.5154505372047424, + 0.48454946279525757 + ], + "sources": 2 + }, + { + "entropy_mean": 0.641852617263794, + "mean_weights": [ + 0.5881834030151367, + 0.4118165969848633 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6888687014579773, + "mean_weights": [ + 0.4875519573688507, + 0.5124480128288269 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6153738498687744, + "mean_weights": [ + 0.663661003112793, + 0.33633893728256226 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6768927574157715, + "mean_weights": [ + 0.5171866416931152, + 0.48281329870224 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0488219261169434, + "mean_weights": [ + 0.3366386890411377, + 0.2424165904521942, + 0.4209447503089905 + ], + "sources": 3 + }, + { + "entropy_mean": 1.091782569885254, + "mean_weights": [ + 0.3493679463863373, + 0.296635240316391, + 0.35399681329727173 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0304211378097534, + "mean_weights": [ + 0.3192567527294159, + 0.22267529368400574, + 0.45806795358657837 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0887223482131958, + "mean_weights": [ + 0.33021461963653564, + 0.2846145033836365, + 0.3851708471775055 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0362792015075684, + "mean_weights": [ + 0.30526289343833923, + 0.2524441182613373, + 0.4422929883003235 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0789368152618408, + "mean_weights": [ + 0.38163846731185913, + 0.24751991033554077, + 0.3708415925502777 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0607939958572388, + "mean_weights": [ + 0.38305139541625977, + 0.23213154077529907, + 0.38481706380844116 + ], + "sources": 3 + }, + { + "entropy_mean": 1.086788535118103, + "mean_weights": [ + 0.37750691175460815, + 0.27428025007247925, + 0.348212867975235 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3477751016616821, + "mean_weights": [ + 0.26439589262008667, + 0.177627295255661, + 0.24650344252586365, + 0.3114733397960663 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3783923387527466, + "mean_weights": [ + 0.2721327543258667, + 0.23036113381385803, + 0.22305163741111755, + 0.2744544744491577 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3568333387374878, + "mean_weights": [ + 0.3050576448440552, + 0.1938839554786682, + 0.22442257404327393, + 0.27663588523864746 + ], + "sources": 4 + }, + { + "entropy_mean": 1.374474048614502, + "mean_weights": [ + 0.27234533429145813, + 0.21675783395767212, + 0.2160918414592743, + 0.29480499029159546 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3339743614196777, + "mean_weights": [ + 0.29681411385536194, + 0.1998686045408249, + 0.20463740825653076, + 0.2986798882484436 + ], + "sources": 4 + }, + { + "entropy_mean": 1.365078330039978, + "mean_weights": [ + 0.27501603960990906, + 0.2072608768939972, + 0.20298418402671814, + 0.314738929271698 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3462111949920654, + "mean_weights": [ + 0.3397356867790222, + 0.1902334988117218, + 0.19029755890369415, + 0.27973324060440063 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3483333587646484, + "mean_weights": [ + 0.2402786761522293, + 0.18873727321624756, + 0.21601642668247223, + 0.3549676537513733 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6012260913848877, + "mean_weights": [ + 0.23053249716758728, + 0.18287596106529236, + 0.1824776828289032, + 0.20440185070037842, + 0.19971200823783875 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5957448482513428, + "mean_weights": [ + 0.20878584682941437, + 0.16962705552577972, + 0.1613427698612213, + 0.22147464752197266, + 0.23876966536045074 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6034154891967773, + "mean_weights": [ + 0.20266541838645935, + 0.1842348277568817, + 0.18321608006954193, + 0.19841036200523376, + 0.23147332668304443 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5956087112426758, + "mean_weights": [ + 0.2023323029279709, + 0.1826343685388565, + 0.15936921536922455, + 0.2023482620716095, + 0.25331586599349976 + ], + "sources": 5 + }, + { + "entropy_mean": 1.56992506980896, + "mean_weights": [ + 0.1990685611963272, + 0.16168993711471558, + 0.1653275191783905, + 0.19290706515312195, + 0.28100690245628357 + ], + "sources": 5 + }, + { + "entropy_mean": 1.594915509223938, + "mean_weights": [ + 0.20269615948200226, + 0.17959251999855042, + 0.16759032011032104, + 0.1907426416873932, + 0.2593783736228943 + ], + "sources": 5 + }, + { + "entropy_mean": 1.57517409324646, + "mean_weights": [ + 0.20765285193920135, + 0.16873890161514282, + 0.16227200627326965, + 0.18787050247192383, + 0.27346575260162354 + ], + "sources": 5 + }, + { + "entropy_mean": 1.582337737083435, + "mean_weights": [ + 0.20267564058303833, + 0.16946160793304443, + 0.1533556580543518, + 0.2029361128807068, + 0.27157101035118103 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7704005241394043, + "mean_weights": [ + 0.15813738107681274, + 0.14095252752304077, + 0.15112003684043884, + 0.1676008701324463, + 0.22075437009334564, + 0.16143479943275452 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7744179964065552, + "mean_weights": [ + 0.1739913523197174, + 0.14824096858501434, + 0.12550108134746552, + 0.1490170955657959, + 0.19374242424964905, + 0.20950709283351898 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7644041776657104, + "mean_weights": [ + 0.1344396471977234, + 0.13894908130168915, + 0.15592080354690552, + 0.16003203392028809, + 0.18981875479221344, + 0.22083967924118042 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7707514762878418, + "mean_weights": [ + 0.15969058871269226, + 0.14724712073802948, + 0.13208217918872833, + 0.1498047560453415, + 0.17652836441993713, + 0.2346469759941101 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7259507179260254, + "mean_weights": [ + 0.12789717316627502, + 0.1277076005935669, + 0.1471347063779831, + 0.1415582299232483, + 0.17471638321876526, + 0.28098589181900024 + ], + "sources": 6 + }, + { + "entropy_mean": 1.783919095993042, + "mean_weights": [ + 0.1669968068599701, + 0.152359738945961, + 0.14563660323619843, + 0.15705418586730957, + 0.17407512664794922, + 0.2038775384426117 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7073767185211182, + "mean_weights": [ + 0.12553739547729492, + 0.1339622288942337, + 0.15042653679847717, + 0.14262740314006805, + 0.15269038081169128, + 0.29475605487823486 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.939546823501587, + "mean_weights": [ + 0.15422281622886658, + 0.1592129021883011, + 0.13749317824840546, + 0.14361301064491272, + 0.1436930000782013, + 0.1270073652267456, + 0.13475774228572845 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9389374256134033, + "mean_weights": [ + 0.08043944835662842, + 0.1022244468331337, + 0.09947343170642853, + 0.08341135829687119, + 0.08009927719831467, + 0.08885642141103745, + 0.1792944371700287, + 0.2862011790275574 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9423264265060425, + "mean_weights": [ + 0.11040160059928894, + 0.12943531572818756, + 0.09637222439050674, + 0.10034463554620743, + 0.10099241882562637, + 0.07893810421228409, + 0.08249013125896454, + 0.3010255694389343 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8252936601638794, + "mean_weights": [ + 0.07774615287780762, + 0.08811849355697632, + 0.08495154976844788, + 0.07656968384981155, + 0.07269731163978577, + 0.07230732589960098, + 0.1414995789527893, + 0.3861098885536194 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9609801769256592, + "mean_weights": [ + 0.10630328208208084, + 0.11639947444200516, + 0.10746649652719498, + 0.1070953831076622, + 0.10321417450904846, + 0.07749591767787933, + 0.0924282968044281, + 0.2895969748497009 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8125814199447632, + "mean_weights": [ + 0.08326774090528488, + 0.0901469886302948, + 0.0897221639752388, + 0.08056290447711945, + 0.07609361410140991, + 0.06766720861196518, + 0.11518652737140656, + 0.3973528742790222 + ], + "sources": 8 + }, + { + "entropy_mean": 1.971147894859314, + "mean_weights": [ + 0.10927562415599823, + 0.13284865021705627, + 0.10684794187545776, + 0.1083848625421524, + 0.09628733992576599, + 0.07171809673309326, + 0.10677085071802139, + 0.26786667108535767 + ], + "sources": 8 + }, + { + "entropy_mean": 1.936267614364624, + "mean_weights": [ + 0.08448073267936707, + 0.107402004301548, + 0.1044008657336235, + 0.10038185864686966, + 0.08913292735815048, + 0.08267508447170258, + 0.1414467990398407, + 0.2900797128677368 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0270423889160156, + "mean_weights": [ + 0.11036348342895508, + 0.13825339078903198, + 0.1235540360212326, + 0.12005487829446793, + 0.10626235604286194, + 0.09149761497974396, + 0.13074813783168793, + 0.17926611006259918 + ], + "sources": 8 + }, + { + "entropy_mean": 2.124375343322754, + "mean_weights": [ + 0.09048692882061005, + 0.10663249343633652, + 0.09934761375188828, + 0.09716637432575226, + 0.08316338807344437, + 0.0760551244020462, + 0.12852120399475098, + 0.1873396933078766, + 0.13128720223903656 + ], + "sources": 9 + }, + { + "entropy_mean": 2.137887477874756, + "mean_weights": [ + 0.09799301624298096, + 0.10568772256374359, + 0.09747691452503204, + 0.0946710854768753, + 0.09129421412944794, + 0.08044944703578949, + 0.10569875687360764, + 0.18316687643527985, + 0.1435619741678238 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0624029636383057, + "mean_weights": [ + 0.09006796777248383, + 0.11167029291391373, + 0.08902578055858612, + 0.09357736259698868, + 0.07623497396707535, + 0.0602741613984108, + 0.11503724753856659, + 0.24456089735031128, + 0.11955131590366364 + ], + "sources": 9 + }, + { + "entropy_mean": 2.116382122039795, + "mean_weights": [ + 0.10900848358869553, + 0.11120526492595673, + 0.09882031381130219, + 0.11082525551319122, + 0.09545718133449554, + 0.0729915127158165, + 0.10388582199811935, + 0.16615299880504608, + 0.13165315985679626 + ], + "sources": 9 + }, + { + "entropy_mean": 2.127840995788574, + "mean_weights": [ + 0.08206897228956223, + 0.1412779688835144, + 0.11807555705308914, + 0.11755016446113586, + 0.08534689247608185, + 0.0873321145772934, + 0.14277508854866028, + 0.14960961043834686, + 0.07596361637115479 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1493215560913086, + "mean_weights": [ + 0.1004270389676094, + 0.1329527646303177, + 0.12460179626941681, + 0.13019627332687378, + 0.11393401026725769, + 0.09179335087537766, + 0.10835116356611252, + 0.1143532246351242, + 0.08339038491249084 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1247034072875977, + "mean_weights": [ + 0.08923884481191635, + 0.12645894289016724, + 0.11293411999940872, + 0.11077424138784409, + 0.09245959669351578, + 0.07573410868644714, + 0.12163764238357544, + 0.18632149696350098, + 0.08444099873304367 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05609767511487007, + 0.046314943581819534, + 0.05613238736987114, + 0.058331482112407684, + 0.08996930718421936, + 0.0926729142665863, + 0.10781487077474594, + 0.08399695158004761, + 0.10695178806781769, + 0.07489363849163055, + 0.08251035213470459, + 0.07767805457115173, + 0.08443672955036163, + 0.08382667601108551, + 0.07941463589668274, + 0.07728297263383865, + 0.0826355591416359, + 0.0555785596370697, + 0.06328161060810089, + 0.057926081120967865, + 0.062076833099126816, + 0.05957484245300293, + 0.06236552074551582, + 0.058642707765102386, + 0.0673491582274437, + 0.053680967539548874, + 0.053235381841659546, + 0.05456315353512764, + 0.05578923970460892, + 0.05341488867998123, + 0.05765974521636963, + 0.056713689118623734, + 0.06196129694581032, + 0.05072925239801407, + 0.04887295886874199, + 0.04936721920967102, + 0.050629522651433945, + 0.05090931057929993, + 0.053066689521074295, + 0.05574755370616913, + 0.05583072453737259, + 0.048166461288928986, + 0.04866266995668411, + 0.04889552295207977, + 0.04948895424604416, + 0.0499676950275898, + 0.050868868827819824, + 0.04961797222495079, + 0.051817622035741806, + 0.0417652390897274, + 0.03910867124795914, + 0.0400182269513607, + 0.04237509146332741, + 0.044484443962574005, + 0.04767679050564766, + 0.0499703474342823, + 0.05142516270279884, + 0.04620761796832085, + 0.04854274168610573, + 0.053496040403842926, + 0.05569733679294586, + 0.05314302444458008, + 0.05639098957180977, + 0.06023091450333595 + ], + "loss_nats": 1.4395790100097656, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.9096611738204956, + "mean_weights": [ + 0.10854298621416092, + 0.08049573004245758, + 0.0781460553407669, + 0.0618569552898407, + 0.06090468540787697, + 0.05017947405576706, + 0.05412861704826355, + 0.2897106409072876, + 0.21603484451770782 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.04231308028101921, + 0.08136755228042603, + 0.14546501636505127, + 0.16205471754074097, + 0.18846724927425385, + 0.19161629676818848, + 0.1953161358833313, + 0.20171724259853363, + 0.07905919849872589, + 0.0805743932723999, + 0.0915244072675705, + 0.09270916879177094, + 0.10115914046764374, + 0.10289278626441956, + 0.10992514342069626, + 0.1131928563117981, + 0.036441486328840256, + 0.041872475296258926, + 0.06540236622095108, + 0.07351861149072647, + 0.08656997233629227, + 0.09840589761734009, + 0.10800182074308395, + 0.12142564356327057, + 0.03355863690376282, + 0.041982922703027725, + 0.05598479136824608, + 0.07002262026071548, + 0.07697499543428421, + 0.09207555651664734, + 0.10182415693998337, + 0.11870673298835754, + 0.02821013703942299, + 0.03839530423283577, + 0.0550437867641449, + 0.0703679770231247, + 0.08187434822320938, + 0.09554289281368256, + 0.11305249482393265, + 0.13110701739788055, + 0.042858436703681946, + 0.04829283058643341, + 0.07397264242172241, + 0.0819089487195015, + 0.11487434059381485, + 0.12588800489902496, + 0.15384331345558167, + 0.17493852972984314, + 0.02350943349301815, + 0.04844440519809723, + 0.0512012280523777, + 0.07240089774131775, + 0.0740746259689331, + 0.10001245886087418, + 0.1010332927107811, + 0.13743279874324799, + 0.022145941853523254, + 0.06662143021821976, + 0.06975217908620834, + 0.13172371685504913, + 0.1354401409626007, + 0.22823166847229004, + 0.23401445150375366, + 0.3327999711036682 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.083634798415005e-05, + 6.544288044096902e-05, + 6.18911799392663e-05, + 5.824340769322589e-05, + 6.840504647698253e-05, + 6.442260928452015e-05, + 6.07360816502478e-05, + 5.57215535081923e-05, + 5.394695108407177e-05, + 5.159390639164485e-05, + 4.947753404849209e-05, + 4.8362475354224443e-05, + 5.206514833844267e-05, + 4.952356539433822e-05, + 4.720787546830252e-05, + 4.488748891162686e-05, + 5.4848358558956534e-05, + 5.069619510322809e-05, + 5.109578341944143e-05, + 4.803398041985929e-05, + 5.8494570112088695e-05, + 5.637927824864164e-05, + 5.172396413399838e-05, + 4.832855120184831e-05, + 0.00012725338456220925, + 9.983821655623615e-05, + 6.855343235656619e-05, + 5.203349428484216e-05, + 3.808153633144684e-05, + 2.811460944940336e-05, + 2.6906145649263635e-05, + 2.5965828172047623e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 6.196238837219425e-05, + "first_to_last_ratio": 1.06203035368154, + "imbalance_abs_log_ratio": 0.0601825040323362, + "last_quartile_mean": 5.83433309202519e-05, + "mean": 5.559723064152422e-05, + "normalized": [ + 1.094233422819317, + 1.1770888529129602, + 1.1132061655790688, + 1.0475954831053276, + 1.2303678742209232, + 1.158737738213969, + 1.0924299816632501, + 1.0022361341605246, + 0.9703172345382992, + 0.9279941787803836, + 0.8899280319825592, + 0.8698720205337653, + 0.9364701755406586, + 0.8907559751249602, + 0.8491048011489282, + 0.8073691511911654, + 0.9865304067500015, + 0.9118474880539165, + 0.9190346862579021, + 0.8639635439680313, + 1.0521130178092095, + 1.0140663050676004, + 0.9303334633248262, + 0.8692618435162289, + 2.2888439422946147, + 1.7957408202571408, + 1.2330368179411673, + 0.9359008296715342, + 0.6849538347869555, + 0.5056836307311545, + 0.48394758765498813, + 0.4670345603986681 + ], + "population_cv": 0.333432582075798 + }, + "activation_output_rms_by_block": [ + 0.12245061248540878, + 0.2410275638103485, + 0.26551103591918945, + 0.2780594229698181, + 0.10822394490242004, + 0.11480753123760223, + 0.12659130990505219, + 0.14161154627799988, + 0.049005042761564255, + 0.07890070974826813, + 0.10965057462453842, + 0.14385224878787994, + 0.04900499805808067, + 0.08259917795658112, + 0.11474930495023727, + 0.15529422461986542, + 0.0494392067193985, + 0.09026877582073212, + 0.12038259208202362, + 0.17377765476703644, + 0.0681336373090744, + 0.11240879446268082, + 0.17355279624462128, + 0.23771671950817108, + 0.05975046381354332, + 0.09065639972686768, + 0.12379270792007446, + 0.16509544849395752, + 0.07617168873548508, + 0.15333347022533417, + 0.26607605814933777, + 0.39294421672821045 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.1747853709384799, + "first_to_last_ratio": 1.053066296360896, + "imbalance_abs_log_ratio": 0.05170619067699571, + "last_quartile_mean": 0.1659775567241013, + "mean": 0.14171374624129385, + "normalized": [ + 0.8640701112855628, + 1.7008058159718291, + 1.8735729099074683, + 1.9621203330294477, + 0.7636799377115384, + 0.8101368729757605, + 0.8932888534998368, + 0.9992788281588438, + 0.34580302942612295, + 0.5567611600213086, + 0.7737469196378314, + 1.0150902971892712, + 0.3458027139769533, + 0.5828593213246988, + 0.8097260004322755, + 1.0958303533621099, + 0.34886669804931353, + 0.6369796735669724, + 0.8494771698226791, + 1.2262582803445747, + 0.48078354511258414, + 0.7932102385557155, + 1.2246715710211737, + 1.6774429144185803, + 0.4216278617870077, + 0.6397149333171138, + 0.8735405788320245, + 1.164992478660835, + 0.5375038819860757, + 1.0819943321818308, + 1.8775599771125526, + 2.772802407320108 + ], + "population_cv": 0.55106080825459 + }, + "bits_per_byte": 1.9514305570481558, + "branch_output_rms_by_sublayer": [ + 0.06754149496555328, + 0.09906163811683655, + 0.1411740481853485, + 0.03420129418373108, + 0.05437101423740387, + 0.010953312739729881, + 0.04357735067605972, + 0.02709466591477394, + 0.10636105388402939, + 0.0056360638700425625, + 0.030990611761808395, + 0.00608070008456707, + 0.034389760345220566, + 0.013713452965021133, + 0.04804103821516037, + 0.026967480778694153, + 0.04182898625731468, + 0.0227909367531538, + 0.03000296652317047, + 0.026110181584954262, + 0.02981160767376423, + 0.03051331639289856, + 0.03314096853137016, + 0.03473140671849251, + 0.03609837219119072, + 0.029010696336627007, + 0.032996661961078644, + 0.03551503270864487, + 0.03221718594431877, + 0.03669314458966255, + 0.042832981795072556, + 0.043400876224040985, + 0.030554378405213356, + 0.03679518774151802, + 0.04347085580229759, + 0.04104701429605484, + 0.04007279872894287, + 0.04058202728629112, + 0.0625486671924591, + 0.06331271678209305, + 0.06109128147363663, + 0.03414784371852875, + 0.06545570492744446, + 0.03752531483769417, + 0.09005658328533173, + 0.052936870604753494, + 0.08504494279623032, + 0.08369867503643036, + 0.02813500165939331, + 0.04959388077259064, + 0.01941630430519581, + 0.05439982935786247, + 0.02140895090997219, + 0.06527816504240036, + 0.016235366463661194, + 0.07518240064382553, + 0.025328030809760094, + 0.0708245187997818, + 0.020400507375597954, + 0.09678608924150467, + 0.020449036732316017, + 0.1359638124704361, + 0.03092157281935215, + 0.1444951295852661 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0002476620980690333, + 0.00017955263911543766, + 0.00011775357681837175, + 0.000134767548690359, + 0.00015257927292625358, + 8.090566719979557e-05, + 0.00010365463547085527, + 0.00014795899419583052, + 0.00012056669010185042, + 0.00011160157055380356, + 0.00011638167572523615, + 0.0001290375446210304, + 0.00011817153266196847, + 0.00013748161045529589, + 0.00014628178012741046, + 0.00017238580674161478, + 0.0001710369270029862, + 0.00020563767024830051, + 0.00020050943238950297, + 0.00027226405967860747, + 0.00024608432506836777, + 0.00028500761117977356, + 0.0003476612199725255, + 0.00037412990415790524, + 0.000627398444541366, + 0.0005107512418643295, + 0.00040675667276681674, + 0.0003218870643851114, + 0.0002135148147390785, + 0.00017969283505162506, + 0.00016304758460817907, + 0.00016251011174592004 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00014560430406074208, + "first_to_last_ratio": 0.4505155505012938, + "imbalance_abs_log_ratio": 0.7973626842136302, + "last_quartile_mean": 0.0003231948462128033, + "mean": 0.00021576976758982944, + "normalized": [ + 1.1478072245034348, + 0.8321491982915825, + 0.5457371444280232, + 0.6245895808097975, + 0.7071392560254377, + 0.37496294362050914, + 0.48039461982412196, + 0.6857262527950405, + 0.5587747136616625, + 0.5172252433712313, + 0.5393789733623551, + 0.5980334782874964, + 0.5476741898643015, + 0.6371680888892807, + 0.6779530875033747, + 0.7989340150252837, + 0.792682538028781, + 0.9530420899336107, + 0.9292749147816861, + 1.2618267272557462, + 1.1404949257588544, + 1.3208876033159695, + 1.6112601123685548, + 1.733931070774997, + 2.9077217422508785, + 2.3671121657564616, + 1.885142099888834, + 1.4918079950709644, + 0.9895492641256511, + 0.8327989461119255, + 0.7556553748443882, + 0.7531644194697652 + ], + "population_cv": 0.5690063109028042 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.26166513562202454, + "mean_weights": [ + 0.9218010902404785, + 0.07819889485836029 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5999901294708252, + "mean_weights": [ + 0.5889743566513062, + 0.41102564334869385 + ], + "sources": 2 + }, + { + "entropy_mean": 0.4633311927318573, + "mean_weights": [ + 0.8180317878723145, + 0.18196819722652435 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6880280375480652, + "mean_weights": [ + 0.5055345296859741, + 0.49446550011634827 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6534379720687866, + "mean_weights": [ + 0.5568671822547913, + 0.44313281774520874 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6888366341590881, + "mean_weights": [ + 0.48569321632385254, + 0.5143067836761475 + ], + "sources": 2 + }, + { + "entropy_mean": 0.579142689704895, + "mean_weights": [ + 0.7064165472984314, + 0.2935834527015686 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6719289422035217, + "mean_weights": [ + 0.5203733444213867, + 0.47962668538093567 + ], + "sources": 2 + }, + { + "entropy_mean": 0.8278098106384277, + "mean_weights": [ + 0.18163590133190155, + 0.14273937046527863, + 0.6756247282028198 + ], + "sources": 3 + }, + { + "entropy_mean": 1.085150957107544, + "mean_weights": [ + 0.3461553156375885, + 0.2693561017513275, + 0.384488582611084 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9436830878257751, + "mean_weights": [ + 0.23911014199256897, + 0.18222489953041077, + 0.578665018081665 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0845465660095215, + "mean_weights": [ + 0.3475063145160675, + 0.2683289647102356, + 0.3841646909713745 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0255789756774902, + "mean_weights": [ + 0.2914285361766815, + 0.2513256072998047, + 0.4572458267211914 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0624152421951294, + "mean_weights": [ + 0.4255833029747009, + 0.221516951918602, + 0.3528997600078583 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0472604036331177, + "mean_weights": [ + 0.404721736907959, + 0.21302083134651184, + 0.3822574317455292 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0726948976516724, + "mean_weights": [ + 0.4173252284526825, + 0.2521638870239258, + 0.3305109143257141 + ], + "sources": 3 + }, + { + "entropy_mean": 1.335268497467041, + "mean_weights": [ + 0.2740474343299866, + 0.15443675220012665, + 0.25526970624923706, + 0.3162460923194885 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3736662864685059, + "mean_weights": [ + 0.29010218381881714, + 0.22382467985153198, + 0.21550415456295013, + 0.27056899666786194 + ], + "sources": 4 + }, + { + "entropy_mean": 1.331147313117981, + "mean_weights": [ + 0.3359344005584717, + 0.16503211855888367, + 0.24076569080352783, + 0.2582678198814392 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3663201332092285, + "mean_weights": [ + 0.30004432797431946, + 0.21556539833545685, + 0.19624200463294983, + 0.28814825415611267 + ], + "sources": 4 + }, + { + "entropy_mean": 1.311529278755188, + "mean_weights": [ + 0.3167944550514221, + 0.19517023861408234, + 0.21175554394721985, + 0.2762797474861145 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3615537881851196, + "mean_weights": [ + 0.2919568419456482, + 0.20172014832496643, + 0.2005811333656311, + 0.3057418763637543 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3230767250061035, + "mean_weights": [ + 0.3729177713394165, + 0.1727619767189026, + 0.1876872032880783, + 0.2666330635547638 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3573906421661377, + "mean_weights": [ + 0.24635542929172516, + 0.2019730806350708, + 0.210106760263443, + 0.34156471490859985 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5965590476989746, + "mean_weights": [ + 0.22502748668193817, + 0.1665719449520111, + 0.18887992203235626, + 0.20406407117843628, + 0.21545657515525818 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5906414985656738, + "mean_weights": [ + 0.20478254556655884, + 0.16537632048130035, + 0.1616051197052002, + 0.20343966782093048, + 0.26479634642601013 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5973002910614014, + "mean_weights": [ + 0.20110894739627838, + 0.17467620968818665, + 0.1889004111289978, + 0.19413438439369202, + 0.24118004739284515 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5910214185714722, + "mean_weights": [ + 0.20541837811470032, + 0.18154606223106384, + 0.16363713145256042, + 0.18218618631362915, + 0.26721227169036865 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5553951263427734, + "mean_weights": [ + 0.2047061026096344, + 0.15580019354820251, + 0.1712058037519455, + 0.18160474300384521, + 0.2866831421852112 + ], + "sources": 5 + }, + { + "entropy_mean": 1.594994068145752, + "mean_weights": [ + 0.20726267993450165, + 0.18533048033714294, + 0.1670190691947937, + 0.18339315056800842, + 0.2569946348667145 + ], + "sources": 5 + }, + { + "entropy_mean": 1.557883620262146, + "mean_weights": [ + 0.2142346203327179, + 0.1605926901102066, + 0.162296861410141, + 0.18356922268867493, + 0.2793066203594208 + ], + "sources": 5 + }, + { + "entropy_mean": 1.581823468208313, + "mean_weights": [ + 0.20680969953536987, + 0.16870613396167755, + 0.15061055123806, + 0.19787293672561646, + 0.2760006785392761 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7632503509521484, + "mean_weights": [ + 0.159615159034729, + 0.12706682085990906, + 0.15427574515342712, + 0.16119518876075745, + 0.230606347322464, + 0.1672406941652298 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7743520736694336, + "mean_weights": [ + 0.1715245395898819, + 0.14721602201461792, + 0.13383647799491882, + 0.1418425589799881, + 0.1853652149438858, + 0.22021520137786865 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7345709800720215, + "mean_weights": [ + 0.13183030486106873, + 0.12091188877820969, + 0.1493622362613678, + 0.1412108689546585, + 0.17940853536128998, + 0.2772761583328247 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7687442302703857, + "mean_weights": [ + 0.15932488441467285, + 0.14595365524291992, + 0.1298016458749771, + 0.14878174662590027, + 0.17904385924339294, + 0.2370941936969757 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6830962896347046, + "mean_weights": [ + 0.12939350306987762, + 0.11201902478933334, + 0.12812075018882751, + 0.1307620406150818, + 0.16620692610740662, + 0.3334977328777313 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7842655181884766, + "mean_weights": [ + 0.16689783334732056, + 0.15244266390800476, + 0.15117637813091278, + 0.1540689766407013, + 0.16848403215408325, + 0.20693014562129974 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6715667247772217, + "mean_weights": [ + 0.12708550691604614, + 0.12521255016326904, + 0.138344869017601, + 0.13131040334701538, + 0.14218345284461975, + 0.33586323261260986 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9397857189178467, + "mean_weights": [ + 0.15236307680606842, + 0.16006770730018616, + 0.13264358043670654, + 0.15044324100017548, + 0.14299464225769043, + 0.12786929309368134, + 0.13361847400665283 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8866612911224365, + "mean_weights": [ + 0.07807950675487518, + 0.08882124722003937, + 0.09119457751512527, + 0.07908158004283905, + 0.0763094425201416, + 0.08141098916530609, + 0.17950907349586487, + 0.32559359073638916 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9048370122909546, + "mean_weights": [ + 0.11122602224349976, + 0.123598612844944, + 0.09025430679321289, + 0.10003083944320679, + 0.09014542400836945, + 0.08103406429290771, + 0.07350647449493408, + 0.3302042782306671 + ], + "sources": 8 + }, + { + "entropy_mean": 1.7479267120361328, + "mean_weights": [ + 0.07590004801750183, + 0.07677006721496582, + 0.07577801495790482, + 0.06876982748508453, + 0.0661000907421112, + 0.07159741222858429, + 0.1325106918811798, + 0.4325738549232483 + ], + "sources": 8 + }, + { + "entropy_mean": 1.921669363975525, + "mean_weights": [ + 0.10765805840492249, + 0.11084398627281189, + 0.09232957661151886, + 0.10730402171611786, + 0.09899432957172394, + 0.08516949415206909, + 0.07546523213386536, + 0.3222352862358093 + ], + "sources": 8 + }, + { + "entropy_mean": 1.7216477394104004, + "mean_weights": [ + 0.0804784893989563, + 0.07961706072092056, + 0.0777105838060379, + 0.0728282555937767, + 0.06623688340187073, + 0.06764764338731766, + 0.10457392036914825, + 0.4509071707725525 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9632604122161865, + "mean_weights": [ + 0.11329682171344757, + 0.12058684229850769, + 0.10146578401327133, + 0.11402662098407745, + 0.09633928537368774, + 0.08086135238409042, + 0.09301877021789551, + 0.2804045081138611 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8826301097869873, + "mean_weights": [ + 0.08328057825565338, + 0.10176259279251099, + 0.09323996305465698, + 0.09280694276094437, + 0.08008009195327759, + 0.08396600931882858, + 0.12889695167541504, + 0.33596688508987427 + ], + "sources": 8 + }, + { + "entropy_mean": 2.022946357727051, + "mean_weights": [ + 0.10972768068313599, + 0.12286777794361115, + 0.11593025177717209, + 0.13055941462516785, + 0.11177866905927658, + 0.0991082489490509, + 0.1172795221209526, + 0.19274842739105225 + ], + "sources": 8 + }, + { + "entropy_mean": 1.972595453262329, + "mean_weights": [ + 0.07736541330814362, + 0.08407740294933319, + 0.07698649168014526, + 0.0736711174249649, + 0.062108930200338364, + 0.06390505284070969, + 0.0995078980922699, + 0.18028026819229126, + 0.28209739923477173 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1120684146881104, + "mean_weights": [ + 0.09473604708909988, + 0.09436517208814621, + 0.086631178855896, + 0.10109931230545044, + 0.08663254231214523, + 0.08119484037160873, + 0.09252380579710007, + 0.18678608536720276, + 0.17603102326393127 + ], + "sources": 9 + }, + { + "entropy_mean": 1.95843505859375, + "mean_weights": [ + 0.08478866517543793, + 0.09140758216381073, + 0.07506939768791199, + 0.07730541378259659, + 0.05883209779858589, + 0.05207575485110283, + 0.09340928494930267, + 0.2624351382255554, + 0.20467662811279297 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1141533851623535, + "mean_weights": [ + 0.10537028312683105, + 0.10114683955907822, + 0.09101615846157074, + 0.11537937819957733, + 0.0982808768749237, + 0.08355101943016052, + 0.0960497111082077, + 0.16330750286579132, + 0.1458982229232788 + ], + "sources": 9 + }, + { + "entropy_mean": 2.112180709838867, + "mean_weights": [ + 0.0809403657913208, + 0.1346593201160431, + 0.10325460135936737, + 0.11045745015144348, + 0.08172597736120224, + 0.08425528556108475, + 0.1463059037923813, + 0.15970830619335175, + 0.09869279712438583 + ], + "sources": 9 + }, + { + "entropy_mean": 2.146028518676758, + "mean_weights": [ + 0.09963831305503845, + 0.13222530484199524, + 0.10801737010478973, + 0.14317527413368225, + 0.11926116794347763, + 0.09364169090986252, + 0.10652102530002594, + 0.11628478765487671, + 0.08123508095741272 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0894551277160645, + "mean_weights": [ + 0.08809129893779755, + 0.11690078675746918, + 0.10256171226501465, + 0.1082148402929306, + 0.07970866560935974, + 0.07432056218385696, + 0.10642798244953156, + 0.20887649059295654, + 0.11489762365818024 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.059552766382694244, + 0.05560841038823128, + 0.0706910565495491, + 0.07019952684640884, + 0.12937790155410767, + 0.12918730080127716, + 0.14609739184379578, + 0.0993424579501152, + 0.1428242325782776, + 0.09961877763271332, + 0.10776528716087341, + 0.10328751057386398, + 0.10852212458848953, + 0.11185097694396973, + 0.09696939587593079, + 0.09790124744176865, + 0.10320515930652618, + 0.06996677070856094, + 0.08288728445768356, + 0.07320483773946762, + 0.08104071766138077, + 0.07943439483642578, + 0.080925352871418, + 0.07350212335586548, + 0.08819101750850677, + 0.06778689473867416, + 0.06714078038930893, + 0.06981998682022095, + 0.07081172615289688, + 0.0677955374121666, + 0.07428057491779327, + 0.07163840532302856, + 0.07883031666278839, + 0.06368284672498703, + 0.06154194474220276, + 0.05909047648310661, + 0.0646035224199295, + 0.06274203211069107, + 0.06695375591516495, + 0.07093504816293716, + 0.07160285860300064, + 0.0619935616850853, + 0.062431562691926956, + 0.06299576908349991, + 0.06384546309709549, + 0.06500732898712158, + 0.06651061773300171, + 0.06578325480222702, + 0.06939081847667694, + 0.05544719099998474, + 0.0502186194062233, + 0.05246228724718094, + 0.0547821968793869, + 0.059282489120960236, + 0.06085092946887016, + 0.06604087352752686, + 0.06602385640144348, + 0.050791967660188675, + 0.05958791449666023, + 0.06760811805725098, + 0.06785163283348083, + 0.06949786841869354, + 0.06942258775234222, + 0.0797538086771965 + ], + "loss_nats": 1.3526285886764526, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.8107833862304688, + "mean_weights": [ + 0.0988113060593605, + 0.07130620628595352, + 0.0699438601732254, + 0.054648127406835556, + 0.04812014102935791, + 0.04719514772295952, + 0.04214419424533844, + 0.28913629055023193, + 0.278694748878479 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.06754149496555328, + 0.12245061248540878, + 0.22475971281528473, + 0.2410275638103485, + 0.26289117336273193, + 0.26551103591918945, + 0.2686412036418915, + 0.2780594229698181, + 0.10636105388402939, + 0.10822394490242004, + 0.11399096995592117, + 0.11480753123760223, + 0.12507657706737518, + 0.12659130990505219, + 0.13841715455055237, + 0.14161154627799988, + 0.04182898625731468, + 0.049005042761564255, + 0.06860067695379257, + 0.07890070974826813, + 0.09440769255161285, + 0.10965057462453842, + 0.1255292296409607, + 0.14385224878787994, + 0.03609837219119072, + 0.04900499805808067, + 0.062476154416799545, + 0.08259917795658112, + 0.09414677321910858, + 0.11474930495023727, + 0.13078153133392334, + 0.15529422461986542, + 0.030554378405213356, + 0.0494392067193985, + 0.06955132633447647, + 0.09026877582073212, + 0.10052070021629333, + 0.12038259208202362, + 0.14310145378112793, + 0.17377765476703644, + 0.06109128147363663, + 0.0681336373090744, + 0.10117389261722565, + 0.11240879446268082, + 0.1569545716047287, + 0.17355279624462128, + 0.2061081826686859, + 0.23771671950817108, + 0.02813500165939331, + 0.05975046381354332, + 0.06454070657491684, + 0.09065639972686768, + 0.09429720044136047, + 0.12379270792007446, + 0.12553995847702026, + 0.16509544849395752, + 0.025328030809760094, + 0.07617168873548508, + 0.08001777529716492, + 0.15333347022533417, + 0.1563209891319275, + 0.26607605814933777, + 0.27044984698295593, + 0.39294421672821045 + ] + }, + { + "activation_grad_rms_by_block": [ + 9.932377724908292e-05, + 0.0001046579927788116, + 9.967476944439113e-05, + 9.426590258954093e-05, + 0.00011655616253847256, + 0.00010931217548204586, + 0.00010276963439537212, + 9.379071707371622e-05, + 9.0069166617468e-05, + 8.606118353782222e-05, + 8.265904762083665e-05, + 8.053413330344483e-05, + 9.428523480892181e-05, + 8.944002911448479e-05, + 8.375600737053901e-05, + 7.787881622789428e-05, + 9.990238322643563e-05, + 8.991869253804907e-05, + 8.684277418069541e-05, + 7.90720951044932e-05, + 9.478014544583857e-05, + 9.203208173858002e-05, + 8.36510444059968e-05, + 7.835852011339739e-05, + 0.0002311293501406908, + 0.0001754781842464581, + 0.00011778131010942161, + 8.49109492264688e-05, + 7.42486517992802e-05, + 4.886261376668699e-05, + 4.557712600217201e-05, + 4.02126315748319e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00010254389144392917, + "first_to_last_ratio": 1.0026281013671672, + "imbalance_abs_log_ratio": 0.002624653947560263, + "last_quartile_mean": 0.0001022751021082513, + "mean": 9.461854074288567e-05, + "normalized": [ + 1.049728483120271, + 1.106104490273285, + 1.0534380329881132, + 0.9962730544079834, + 1.2318533093339465, + 1.1552933983529543, + 1.0861468966704533, + 0.9912509359934122, + 0.9519187879066954, + 0.9095594041307715, + 0.8736030694602711, + 0.8511453745866416, + 0.9964773718623544, + 0.9452695889437478, + 0.8851965662642772, + 0.8230819838948948, + 1.055843626862817, + 0.9503284645066779, + 0.9178198426953132, + 0.83569345377416, + 1.001707960212492, + 0.9726643532652439, + 0.8840872386027205, + 0.8281518558432126, + 2.442749046074982, + 1.8545856115377921, + 1.2448015915768342, + 0.8974028616357965, + 0.7847156721750955, + 0.5164169028929032, + 0.48169339375062104, + 0.42499737640326557 + ], + "population_cv": 0.35544435367170285 + }, + "activation_output_rms_by_block": [ + 0.1091526672244072, + 0.2257029414176941, + 0.24690568447113037, + 0.25746673345565796, + 0.08463229238986969, + 0.09073861688375473, + 0.1009940654039383, + 0.11929578334093094, + 0.04301517829298973, + 0.06715712696313858, + 0.09583133459091187, + 0.1316859871149063, + 0.037839874625205994, + 0.07029420882463455, + 0.09966468811035156, + 0.1356954276561737, + 0.04636101424694061, + 0.08232291042804718, + 0.10929957777261734, + 0.1586902141571045, + 0.06434766203165054, + 0.10402293503284454, + 0.15958718955516815, + 0.2186121791601181, + 0.05484472215175629, + 0.08092059940099716, + 0.10955590009689331, + 0.1442185938358307, + 0.062700055539608, + 0.12616725265979767, + 0.20539627969264984, + 0.3026390075683594 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.1543610980734229, + "first_to_last_ratio": 1.1366352897732046, + "imbalance_abs_log_ratio": 0.1280723979412113, + "last_quartile_mean": 0.13580530136823654, + "mean": 0.12330495950300246, + "normalized": [ + 0.8852252793753145, + 1.8304449579921256, + 2.0023986501947495, + 2.0880484815323967, + 0.6863656801072052, + 0.73588781221363, + 0.8190592317698226, + 0.9674856860727171, + 0.3488519721053257, + 0.544642545067324, + 0.777189619762037, + 1.0679699149627455, + 0.30688039457394733, + 0.5700841970020103, + 0.8082780166512664, + 1.1004863729983994, + 0.3759866142757358, + 0.6676366527336852, + 0.8864167200829899, + 1.2869734907397654, + 0.521857857875406, + 0.8436232853254503, + 1.2942479377829261, + 1.7729390613424183, + 0.44478926373128425, + 0.65626394694227, + 0.8884954874359748, + 1.1696090280319906, + 0.5084958122716975, + 1.0232131227189245, + 1.6657584619509846, + 2.45439444437748 + ], + "population_cv": 0.5388505933213965 + }, + "bits_per_byte": 1.8696261492520065, + "branch_output_rms_by_sublayer": [ + 0.05999157205224037, + 0.0878135934472084, + 0.14136430621147156, + 0.02695692516863346, + 0.047949206084012985, + 0.00910202693194151, + 0.034929607063531876, + 0.023768210783600807, + 0.08274894952774048, + 0.00521347438916564, + 0.023115474730730057, + 0.0055172923021018505, + 0.027306556701660156, + 0.012636999599635601, + 0.04704625532031059, + 0.025861741974949837, + 0.03607209026813507, + 0.021478334441781044, + 0.026361703872680664, + 0.02359236590564251, + 0.02868443913757801, + 0.027789726853370667, + 0.0351395420730114, + 0.03219665214419365, + 0.023487690836191177, + 0.02785329520702362, + 0.029714839532971382, + 0.033008087426424026, + 0.03116326965391636, + 0.03530624136328697, + 0.039742980152368546, + 0.04356298968195915, + 0.0275847427546978, + 0.03498651832342148, + 0.03999738767743111, + 0.04058942198753357, + 0.03527665138244629, + 0.04275305196642876, + 0.05535208806395531, + 0.0656047835946083, + 0.05633961409330368, + 0.03470388799905777, + 0.05883055925369263, + 0.038596417754888535, + 0.08146370947360992, + 0.05386771634221077, + 0.07517227530479431, + 0.08294828981161118, + 0.025813337415456772, + 0.04563282057642937, + 0.01574927195906639, + 0.051225993782281876, + 0.020291903987526894, + 0.06201229616999626, + 0.0159101914614439, + 0.07029138505458832, + 0.019816597923636436, + 0.058557167649269104, + 0.016239477321505547, + 0.08471964299678802, + 0.018217360600829124, + 0.10223932564258575, + 0.025022683665156364, + 0.1163979098200798 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00036256920211225226, + 0.00023909838852170657, + 0.0001716852117943585, + 0.0002052383424155029, + 0.00019955362064525106, + 0.00011611424862037749, + 0.0001510664219237857, + 0.0002532324829188919, + 0.00020113841347771863, + 0.00018872062409915142, + 0.00019658120746824132, + 0.0002259886807107496, + 0.0002054960612309121, + 0.0002559375115391091, + 0.00028068839576721085, + 0.00032177826777579085, + 0.0003211822309042471, + 0.00039299423268890756, + 0.0003600682070723913, + 0.0004692556196772698, + 0.00039470136296054477, + 0.00044148158098142955, + 0.000544243713345614, + 0.0005617089207073266, + 0.0010804933109559392, + 0.0008644261123878541, + 0.0007021988237719299, + 0.000534741558677252, + 0.00038869737773335774, + 0.0003095203408934536, + 0.00025890636537826935, + 0.00024310002856117336 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0002123197398690158, + "first_to_last_ratio": 0.38761419237907074, + "imbalance_abs_log_ratio": 0.9477447835908027, + "last_quartile_mean": 0.0005477604897949036, + "mean": 0.00035758146461618655, + "normalized": [ + 1.013948534780513, + 0.6686543128804092, + 0.48012894622107566, + 0.5739624749168625, + 0.5580647779365293, + 0.3247211058464952, + 0.42246714908988436, + 0.7081812341439492, + 0.5624967549523644, + 0.5277695931519171, + 0.5497522296890965, + 0.6319921558387168, + 0.5746832024738285, + 0.7157460239551904, + 0.7849634937551654, + 0.8998740136633608, + 0.8982071574906465, + 1.0990341267009789, + 1.0069543382481358, + 1.3123040932251613, + 1.1038082283828698, + 1.2346321738328858, + 1.5220132115342813, + 1.5708558085085373, + 3.021670354518227, + 2.4174242736985656, + 1.96374502947361, + 1.495439813277855, + 1.087017690222198, + 0.8655939178102539, + 0.7240486182810648, + 0.6798451614993721 + ], + "population_cv": 0.5852093791165481 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.25654762983322144, + "mean_weights": [ + 0.924528956413269, + 0.07547103613615036 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5859805345535278, + "mean_weights": [ + 0.6373459100723267, + 0.36265408992767334 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5292341709136963, + "mean_weights": [ + 0.7701468467712402, + 0.22985315322875977 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6888049840927124, + "mean_weights": [ + 0.47958698868751526, + 0.5204130411148071 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6710846424102783, + "mean_weights": [ + 0.5123445987701416, + 0.4876554012298584 + ], + "sources": 2 + }, + { + "entropy_mean": 0.686903178691864, + "mean_weights": [ + 0.46272584795951843, + 0.537274181842804 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6142507791519165, + "mean_weights": [ + 0.6628916263580322, + 0.3371083438396454 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6781005859375, + "mean_weights": [ + 0.5222160220146179, + 0.47778400778770447 + ], + "sources": 2 + }, + { + "entropy_mean": 0.7619183659553528, + "mean_weights": [ + 0.1484677940607071, + 0.13072103261947632, + 0.720811128616333 + ], + "sources": 3 + }, + { + "entropy_mean": 1.077111005783081, + "mean_weights": [ + 0.3123582601547241, + 0.2649896740913391, + 0.42265206575393677 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9107450246810913, + "mean_weights": [ + 0.21128304302692413, + 0.17685732245445251, + 0.6118596196174622 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0806925296783447, + "mean_weights": [ + 0.33169159293174744, + 0.2629653811454773, + 0.40534305572509766 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0238357782363892, + "mean_weights": [ + 0.2712686359882355, + 0.25061118602752686, + 0.47812020778656006 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0686618089675903, + "mean_weights": [ + 0.4034028947353363, + 0.22649148106575012, + 0.3701056241989136 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0412673950195312, + "mean_weights": [ + 0.3609567880630493, + 0.20633846521377563, + 0.43270474672317505 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0787726640701294, + "mean_weights": [ + 0.3905488848686218, + 0.2559053897857666, + 0.35354578495025635 + ], + "sources": 3 + }, + { + "entropy_mean": 1.322709560394287, + "mean_weights": [ + 0.24716800451278687, + 0.1372198760509491, + 0.2754514217376709, + 0.34016066789627075 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3715487718582153, + "mean_weights": [ + 0.2855983078479767, + 0.21622662246227264, + 0.21348713338375092, + 0.28468793630599976 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3351621627807617, + "mean_weights": [ + 0.30080950260162354, + 0.16193118691444397, + 0.24671727418899536, + 0.29054203629493713 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3677794933319092, + "mean_weights": [ + 0.2910999059677124, + 0.2107609510421753, + 0.20240460336208344, + 0.29573455452919006 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3212642669677734, + "mean_weights": [ + 0.2969600558280945, + 0.19084672629833221, + 0.2178654670715332, + 0.2943277657032013 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3620918989181519, + "mean_weights": [ + 0.2864762544631958, + 0.19820505380630493, + 0.20298032462596893, + 0.31233835220336914 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3292608261108398, + "mean_weights": [ + 0.345756471157074, + 0.16863605380058289, + 0.18931153416633606, + 0.2962959408760071 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3642041683197021, + "mean_weights": [ + 0.24937844276428223, + 0.20937480032444, + 0.2136491984128952, + 0.32759755849838257 + ], + "sources": 4 + }, + { + "entropy_mean": 1.596426248550415, + "mean_weights": [ + 0.21438473463058472, + 0.16552291810512543, + 0.2073245346546173, + 0.2162974327802658, + 0.19647037982940674 + ], + "sources": 5 + }, + { + "entropy_mean": 1.585208773612976, + "mean_weights": [ + 0.20197874307632446, + 0.16221341490745544, + 0.15755358338356018, + 0.20029191672801971, + 0.277962327003479 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5919675827026367, + "mean_weights": [ + 0.1935817152261734, + 0.16350679099559784, + 0.1888156235218048, + 0.19774335622787476, + 0.2563525140285492 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5852510929107666, + "mean_weights": [ + 0.20259243249893188, + 0.1777014434337616, + 0.15493032336235046, + 0.18478944897651672, + 0.27998632192611694 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5295538902282715, + "mean_weights": [ + 0.19488933682441711, + 0.13828939199447632, + 0.16142871975898743, + 0.17478890717029572, + 0.3306036591529846 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5924994945526123, + "mean_weights": [ + 0.19990921020507812, + 0.18152540922164917, + 0.16365136206150055, + 0.18954291939735413, + 0.26537108421325684 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5296177864074707, + "mean_weights": [ + 0.2050195187330246, + 0.14048808813095093, + 0.1550590991973877, + 0.1719839870929718, + 0.32744932174682617 + ], + "sources": 5 + }, + { + "entropy_mean": 1.585580825805664, + "mean_weights": [ + 0.20668278634548187, + 0.17811644077301025, + 0.15315905213356018, + 0.18701499700546265, + 0.27502670884132385 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7498698234558105, + "mean_weights": [ + 0.1553119570016861, + 0.11526855826377869, + 0.15232881903648376, + 0.14954189956188202, + 0.2558150291442871, + 0.17173375189304352 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7741074562072754, + "mean_weights": [ + 0.16851678490638733, + 0.14691884815692902, + 0.13236239552497864, + 0.14436805248260498, + 0.18533164262771606, + 0.22250229120254517 + ], + "sources": 6 + }, + { + "entropy_mean": 1.712247371673584, + "mean_weights": [ + 0.1308009922504425, + 0.11019439250230789, + 0.14377249777317047, + 0.1303066611289978, + 0.18232303857803345, + 0.3026024401187897 + ], + "sources": 6 + }, + { + "entropy_mean": 1.772972583770752, + "mean_weights": [ + 0.16003316640853882, + 0.1488170027732849, + 0.13321717083454132, + 0.1489730030298233, + 0.17728419601917267, + 0.23167544603347778 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6446837186813354, + "mean_weights": [ + 0.12725940346717834, + 0.10245729237794876, + 0.12447428703308105, + 0.11490653455257416, + 0.16374742984771729, + 0.3671550750732422 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7861535549163818, + "mean_weights": [ + 0.16482852399349213, + 0.15605950355529785, + 0.1530533730983734, + 0.15436525642871857, + 0.1708633154630661, + 0.20083001255989075 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6539713144302368, + "mean_weights": [ + 0.12785008549690247, + 0.1187845841050148, + 0.13559319078922272, + 0.11930277943611145, + 0.14155693352222443, + 0.35691240429878235 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9394373893737793, + "mean_weights": [ + 0.15110564231872559, + 0.16482676565647125, + 0.12966853380203247, + 0.15147733688354492, + 0.14320483803749084, + 0.13106408715248108, + 0.12865278124809265 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8596339225769043, + "mean_weights": [ + 0.07650366425514221, + 0.0843966081738472, + 0.08822818845510483, + 0.07480975240468979, + 0.07224889099597931, + 0.08143468201160431, + 0.17312690615653992, + 0.34925132989883423 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8722710609436035, + "mean_weights": [ + 0.10591275990009308, + 0.12091490626335144, + 0.08483774960041046, + 0.0984455868601799, + 0.09109606593847275, + 0.07786098122596741, + 0.06428129971027374, + 0.3566506505012512 + ], + "sources": 8 + }, + { + "entropy_mean": 1.679398775100708, + "mean_weights": [ + 0.07216669619083405, + 0.07139373570680618, + 0.07146680355072021, + 0.06084490939974785, + 0.06314180046319962, + 0.06778790801763535, + 0.11978112161159515, + 0.4734170436859131 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8958207368850708, + "mean_weights": [ + 0.10732583701610565, + 0.10933168232440948, + 0.09077270328998566, + 0.10575048625469208, + 0.09812489151954651, + 0.08500136435031891, + 0.06377172470092773, + 0.33992132544517517 + ], + "sources": 8 + }, + { + "entropy_mean": 1.672429084777832, + "mean_weights": [ + 0.07784943282604218, + 0.07624351978302002, + 0.07613217085599899, + 0.06677807122468948, + 0.0630829855799675, + 0.0645536482334137, + 0.09626906365156174, + 0.4790911078453064 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9662518501281738, + "mean_weights": [ + 0.11312246322631836, + 0.12436866760253906, + 0.10386338829994202, + 0.11864522099494934, + 0.10376769304275513, + 0.07764019817113876, + 0.08166413754224777, + 0.27692824602127075 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8468971252441406, + "mean_weights": [ + 0.08237694203853607, + 0.09629979729652405, + 0.08868251740932465, + 0.08482513576745987, + 0.08011087775230408, + 0.08185725659132004, + 0.12058405578136444, + 0.365263432264328 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0288567543029785, + "mean_weights": [ + 0.1121244728565216, + 0.12685340642929077, + 0.10875183343887329, + 0.13421036303043365, + 0.12144745886325836, + 0.1012754887342453, + 0.10525961220264435, + 0.19007734954357147 + ], + "sources": 8 + }, + { + "entropy_mean": 1.94183349609375, + "mean_weights": [ + 0.07930874079465866, + 0.07933436334133148, + 0.07289997488260269, + 0.07092124223709106, + 0.06372561305761337, + 0.06322522461414337, + 0.09236924350261688, + 0.19880786538124084, + 0.2794077396392822 + ], + "sources": 9 + }, + { + "entropy_mean": 2.107085704803467, + "mean_weights": [ + 0.09533785283565521, + 0.09835049510002136, + 0.08640751242637634, + 0.09882399439811707, + 0.09218519926071167, + 0.07835222780704498, + 0.08051788806915283, + 0.1838473230600357, + 0.18617749214172363 + ], + "sources": 9 + }, + { + "entropy_mean": 1.8828853368759155, + "mean_weights": [ + 0.07947726547718048, + 0.08156488090753555, + 0.07059291005134583, + 0.06569002568721771, + 0.054470956325531006, + 0.047015994787216187, + 0.08014394342899323, + 0.2728085517883301, + 0.24823549389839172 + ], + "sources": 9 + }, + { + "entropy_mean": 2.102785110473633, + "mean_weights": [ + 0.1035289317369461, + 0.10835491120815277, + 0.08945068717002869, + 0.12213703989982605, + 0.09505149722099304, + 0.07644975930452347, + 0.07881602644920349, + 0.16374477744102478, + 0.1624663770198822 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1009445190429688, + "mean_weights": [ + 0.0795045867562294, + 0.12555745244026184, + 0.10012176632881165, + 0.10409736633300781, + 0.0781051442027092, + 0.08643308281898499, + 0.14046895503997803, + 0.16726824641227722, + 0.11844340711832047 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1456127166748047, + "mean_weights": [ + 0.0994025245308876, + 0.13179463148117065, + 0.10713519155979156, + 0.15096381306648254, + 0.125204399228096, + 0.09563042223453522, + 0.0964394062757492, + 0.11412577331066132, + 0.07930384576320648 + ], + "sources": 9 + }, + { + "entropy_mean": 2.041757583618164, + "mean_weights": [ + 0.0862305611371994, + 0.1065414622426033, + 0.09442433714866638, + 0.09713789820671082, + 0.07178515195846558, + 0.07004362344741821, + 0.0896543636918068, + 0.2331295609474182, + 0.1510530412197113 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06233189254999161, + 0.05824589356780052, + 0.06423600763082504, + 0.07460645586252213, + 0.12711766362190247, + 0.12856779992580414, + 0.14220136404037476, + 0.1016618013381958, + 0.13257507979869843, + 0.08432134985923767, + 0.09803175181150436, + 0.08995504677295685, + 0.09734734147787094, + 0.09984444081783295, + 0.08901718258857727, + 0.08931032568216324, + 0.09509968012571335, + 0.060017094016075134, + 0.07334348559379578, + 0.0647977814078331, + 0.07226859778165817, + 0.07020145654678345, + 0.07193213701248169, + 0.06606151908636093, + 0.0799039825797081, + 0.06170506775379181, + 0.05847546458244324, + 0.06017538532614708, + 0.06195582076907158, + 0.057102326303720474, + 0.06530650705099106, + 0.061479777097702026, + 0.06964915990829468, + 0.054929498583078384, + 0.053711604326963425, + 0.04968518391251564, + 0.05636162683367729, + 0.05409017205238342, + 0.05905292183160782, + 0.06239638105034828, + 0.0629483163356781, + 0.054407235234975815, + 0.054735470563173294, + 0.05528644472360611, + 0.05611152946949005, + 0.05715258792042732, + 0.05868622660636902, + 0.058439724147319794, + 0.06209225580096245, + 0.048382628709077835, + 0.04395167529582977, + 0.045431699603796005, + 0.04854873940348625, + 0.05293087288737297, + 0.05332231894135475, + 0.05895760655403137, + 0.057735197246074677, + 0.04565506801009178, + 0.0508013553917408, + 0.05879933014512062, + 0.05774350091814995, + 0.06084565073251724, + 0.05919515714049339, + 0.07114052027463913 + ], + "loss_nats": 1.2959260940551758, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.6899330615997314, + "mean_weights": [ + 0.09075309336185455, + 0.06258825212717056, + 0.05776957795023918, + 0.046863071620464325, + 0.04274744912981987, + 0.039803266525268555, + 0.03252103179693222, + 0.2717641592025757, + 0.35519012808799744 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.05999157205224037, + 0.1091526672244072, + 0.21367698907852173, + 0.2257029414176941, + 0.24478206038475037, + 0.24690568447113037, + 0.24986231327056885, + 0.25746673345565796, + 0.08274894952774048, + 0.08463229238986969, + 0.09015513211488724, + 0.09073861688375473, + 0.09965787082910538, + 0.1009940654039383, + 0.11579012870788574, + 0.11929578334093094, + 0.03607209026813507, + 0.04301517829298973, + 0.05704672262072563, + 0.06715712696313858, + 0.08161647617816925, + 0.09583133459091187, + 0.11469274014234543, + 0.1316859871149063, + 0.023487690836191177, + 0.037839874625205994, + 0.050790462642908096, + 0.07029420882463455, + 0.08000095188617706, + 0.09966468811035156, + 0.11148769408464432, + 0.1356954276561737, + 0.0275847427546978, + 0.04636101424694061, + 0.06276838481426239, + 0.08232291042804718, + 0.08944281190633774, + 0.10929957777261734, + 0.12712964415550232, + 0.1586902141571045, + 0.05633961409330368, + 0.06434766203165054, + 0.09196646511554718, + 0.10402293503284454, + 0.14207474887371063, + 0.15958718955516815, + 0.18664227426052094, + 0.2186121791601181, + 0.025813337415456772, + 0.05484472215175629, + 0.05754614621400833, + 0.08092059940099716, + 0.08401846140623093, + 0.10955590009689331, + 0.11104470491409302, + 0.1442185938358307, + 0.019816597923636436, + 0.062700055539608, + 0.06522931158542633, + 0.12616725265979767, + 0.1279960423707962, + 0.20539627969264984, + 0.20786607265472412, + 0.3026390075683594 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 7.9557801867581395, + "cross_entropy_nats": 5.514526605606079, + "step": 0 + }, + { + "bits_per_byte": 5.165426814938083, + "cross_entropy_nats": 3.5804010331630707, + "step": 100 + }, + { + "bits_per_byte": 3.098621266601985, + "cross_entropy_nats": 2.1478005945682526, + "step": 500 + }, + { + "bits_per_byte": 1.9570974068919167, + "cross_entropy_nats": 1.356556549668312, + "step": 2000 + }, + { + "bits_per_byte": 1.8010449715327461, + "cross_entropy_nats": 1.2483892440795898, + "step": 4000 + }, + { + "bits_per_byte": 1.7188907940204046, + "cross_entropy_nats": 1.191444307565689, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.40.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.40.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "40": true, + "41": true, + "42": true, + "43": true, + "44": true, + "45": true, + "46": true, + "47": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.49.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.49.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.51.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.51.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.53.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.53.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.55.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.55.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_group_6_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "be0529390b0327603491a048f0bfde7f4c57fe4f1e65f331680027cefa1af30f", + "final_model_state": "0711b1a9cc4f6490a366f80a95c37ba3056e8f03a3b29e3b56389f1542a200fd", + "final_optimizer_state": "fc9d2a898b8bc489396f4c8597c5908d24c8b8fad3cea722aac7f929e9d85293", + "final_public_parameters": "ca339ade865bc682350744901bd259cdef104c752888ad11e332b95ad2d95431", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "73bbe569e1a47981386ebaf59ca891174746bf81897fe5403676ad6d50423c58" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "65136111a29a042e61a7909132560d95cd4bcf0f9b52f64d0fb2e57773856434", + "1": "d995676b4e7dec8f661cd8c2345fe7fc7a513c17f528c02fc946a441a6995a94", + "7999": "2345e7ac3decca2bdaebf13094fdc92fcabef42e3e461a2fefd6e8e7e76baccc" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "a7189cc54100edb7c1756c8d5f18a4cc288823c250af4ad694c8f25d86b247ae", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073001, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 212.70427767248702, + "measured_steps": 7980, + "median_ms": 212.7967414999148, + "p95_ms": 215.5133709369693, + "peak_allocated_bytes": 11759431680, + "peak_reserved_bytes": 12266242048, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 7.958777844362678, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.516604423522949, + "step": 1, + "unclipped_grad_norm": 21.32261848449707 + }, + { + "bits_per_byte": 7.247981115583328, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.0239176750183105, + "step": 10, + "unclipped_grad_norm": 11.154319763183594 + }, + { + "bits_per_byte": 6.733647048190599, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.667408466339111, + "step": 20, + "unclipped_grad_norm": 3.489363431930542 + }, + { + "bits_per_byte": 6.611874388281885, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.583002090454102, + "step": 30, + "unclipped_grad_norm": 3.0756163597106934 + }, + { + "bits_per_byte": 6.391837720976198, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.430484294891357, + "step": 40, + "unclipped_grad_norm": 3.2319676876068115 + }, + { + "bits_per_byte": 6.094095164427072, + "learning_rate": 3.75e-05, + "loss_nats": 4.224104881286621, + "step": 50, + "unclipped_grad_norm": 2.8724300861358643 + }, + { + "bits_per_byte": 5.886044999955918, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.079895496368408, + "step": 60, + "unclipped_grad_norm": 2.67099666595459 + }, + { + "bits_per_byte": 5.715541055925228, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.9617111682891846, + "step": 70, + "unclipped_grad_norm": 3.164740800857544 + }, + { + "bits_per_byte": 5.566060271136008, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.8580989837646484, + "step": 80, + "unclipped_grad_norm": 2.5551540851593018 + }, + { + "bits_per_byte": 5.3550279274988934, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.711822509765625, + "step": 90, + "unclipped_grad_norm": 2.4256324768066406 + }, + { + "bits_per_byte": 5.18111081577918, + "learning_rate": 7.5e-05, + "loss_nats": 3.5912723541259766, + "step": 100, + "unclipped_grad_norm": 2.2769131660461426 + }, + { + "bits_per_byte": 4.992225366024273, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.4603469371795654, + "step": 110, + "unclipped_grad_norm": 2.100472927093506 + }, + { + "bits_per_byte": 4.775879446406101, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.310387372970581, + "step": 120, + "unclipped_grad_norm": 1.8943732976913452 + }, + { + "bits_per_byte": 4.582284423739966, + "learning_rate": 9.75e-05, + "loss_nats": 3.1761975288391113, + "step": 130, + "unclipped_grad_norm": 1.6982649564743042 + }, + { + "bits_per_byte": 4.428253321974218, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.0694313049316406, + "step": 140, + "unclipped_grad_norm": 1.4454729557037354 + }, + { + "bits_per_byte": 4.271617370978501, + "learning_rate": 0.0001125, + "loss_nats": 2.960859537124634, + "step": 150, + "unclipped_grad_norm": 1.225760817527771 + }, + { + "bits_per_byte": 4.167454358656932, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.8886592388153076, + "step": 160, + "unclipped_grad_norm": 1.0610370635986328 + }, + { + "bits_per_byte": 4.0266731442789565, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.791077136993408, + "step": 170, + "unclipped_grad_norm": 0.9668865203857422 + }, + { + "bits_per_byte": 3.9477661282018928, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.7363829612731934, + "step": 180, + "unclipped_grad_norm": 0.8600083589553833 + }, + { + "bits_per_byte": 3.839628940784186, + "learning_rate": 0.0001425, + "loss_nats": 2.6614279747009277, + "step": 190, + "unclipped_grad_norm": 0.8320118188858032 + }, + { + "bits_per_byte": 3.7769852921033134, + "learning_rate": 0.00015, + "loss_nats": 2.618006706237793, + "step": 200, + "unclipped_grad_norm": 1.0159423351287842 + }, + { + "bits_per_byte": 3.730443691092122, + "learning_rate": 0.0001575, + "loss_nats": 2.5857465267181396, + "step": 210, + "unclipped_grad_norm": 0.7206029891967773 + }, + { + "bits_per_byte": 3.6333852820776955, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.5184707641601562, + "step": 220, + "unclipped_grad_norm": 0.7382317781448364 + }, + { + "bits_per_byte": 3.6166524020446498, + "learning_rate": 0.0001725, + "loss_nats": 2.5068724155426025, + "step": 230, + "unclipped_grad_norm": 0.780040442943573 + }, + { + "bits_per_byte": 3.5026894743116164, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.4278793334960938, + "step": 240, + "unclipped_grad_norm": 0.7174320816993713 + }, + { + "bits_per_byte": 3.535126778125549, + "learning_rate": 0.0001875, + "loss_nats": 2.4503631591796875, + "step": 250, + "unclipped_grad_norm": 1.0199596881866455 + }, + { + "bits_per_byte": 3.5347786852402026, + "learning_rate": 0.000195, + "loss_nats": 2.4501218795776367, + "step": 260, + "unclipped_grad_norm": 0.9765002131462097 + }, + { + "bits_per_byte": 3.5257592270983533, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.4438700675964355, + "step": 270, + "unclipped_grad_norm": 0.9917251467704773 + }, + { + "bits_per_byte": 3.467242130142283, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.4033091068267822, + "step": 280, + "unclipped_grad_norm": 1.2836586236953735 + }, + { + "bits_per_byte": 3.3672056695455166, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.3339691162109375, + "step": 290, + "unclipped_grad_norm": 0.7538573741912842 + }, + { + "bits_per_byte": 3.418632609737117, + "learning_rate": 0.000225, + "loss_nats": 2.3696155548095703, + "step": 300, + "unclipped_grad_norm": 1.7477567195892334 + }, + { + "bits_per_byte": 3.3640315577411926, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.3317689895629883, + "step": 310, + "unclipped_grad_norm": 1.0488182306289673 + }, + { + "bits_per_byte": 3.33978475569195, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.314962387084961, + "step": 320, + "unclipped_grad_norm": 0.8927777409553528 + }, + { + "bits_per_byte": 3.324816761622058, + "learning_rate": 0.0002475, + "loss_nats": 2.3045873641967773, + "step": 330, + "unclipped_grad_norm": 0.9596893191337585 + }, + { + "bits_per_byte": 3.412968533113601, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.365689516067505, + "step": 340, + "unclipped_grad_norm": 0.9264032244682312 + }, + { + "bits_per_byte": 3.3533469635736894, + "learning_rate": 0.0002625, + "loss_nats": 2.3243629932403564, + "step": 350, + "unclipped_grad_norm": 0.9232060313224792 + }, + { + "bits_per_byte": 3.296050599473443, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.2846481800079346, + "step": 360, + "unclipped_grad_norm": 1.2688182592391968 + }, + { + "bits_per_byte": 3.2850715710076632, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.277038097381592, + "step": 370, + "unclipped_grad_norm": 0.9678829908370972 + }, + { + "bits_per_byte": 3.3315584815358803, + "learning_rate": 0.000285, + "loss_nats": 2.309260368347168, + "step": 380, + "unclipped_grad_norm": 1.2885792255401611 + }, + { + "bits_per_byte": 3.2548115675536535, + "learning_rate": 0.0002925, + "loss_nats": 2.256063461303711, + "step": 390, + "unclipped_grad_norm": 1.3454504013061523 + }, + { + "bits_per_byte": 3.1982523210950053, + "learning_rate": 0.0003, + "loss_nats": 2.2168595790863037, + "step": 400, + "unclipped_grad_norm": 1.0475049018859863 + }, + { + "bits_per_byte": 3.2009875331743274, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.2187554836273193, + "step": 410, + "unclipped_grad_norm": 1.1378477811813354 + }, + { + "bits_per_byte": 3.2207376768341107, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.232445240020752, + "step": 420, + "unclipped_grad_norm": 1.0079383850097656 + }, + { + "bits_per_byte": 3.1851716646356816, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.2077927589416504, + "step": 430, + "unclipped_grad_norm": 0.9795085191726685 + }, + { + "bits_per_byte": 3.242960587047999, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.2478489875793457, + "step": 440, + "unclipped_grad_norm": 1.257104754447937 + }, + { + "bits_per_byte": 3.159943185615315, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.190305709838867, + "step": 450, + "unclipped_grad_norm": 1.2186659574508667 + }, + { + "bits_per_byte": 3.188852093364146, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.210343837738037, + "step": 460, + "unclipped_grad_norm": 1.2219805717468262 + }, + { + "bits_per_byte": 3.132021458282436, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.1709518432617188, + "step": 470, + "unclipped_grad_norm": 1.1065059900283813 + }, + { + "bits_per_byte": 3.1440973920954187, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.1793222427368164, + "step": 480, + "unclipped_grad_norm": 1.0026949644088745 + }, + { + "bits_per_byte": 3.0797861996317804, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.1347451210021973, + "step": 490, + "unclipped_grad_norm": 1.499160647392273 + }, + { + "bits_per_byte": 3.0383641781714728, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.1060335636138916, + "step": 500, + "unclipped_grad_norm": 1.293682336807251 + }, + { + "bits_per_byte": 3.0775493932746585, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.1331946849823, + "step": 510, + "unclipped_grad_norm": 1.2183760404586792 + }, + { + "bits_per_byte": 2.9871989955744853, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.070568561553955, + "step": 520, + "unclipped_grad_norm": 1.1598179340362549 + }, + { + "bits_per_byte": 3.0731270313903725, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.130129337310791, + "step": 530, + "unclipped_grad_norm": 1.284065842628479 + }, + { + "bits_per_byte": 3.010051362290532, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.0864086151123047, + "step": 540, + "unclipped_grad_norm": 0.9896875619888306 + }, + { + "bits_per_byte": 2.9835337013192973, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.068027973175049, + "step": 550, + "unclipped_grad_norm": 0.9373286366462708 + }, + { + "bits_per_byte": 2.9783886683360854, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.0644617080688477, + "step": 560, + "unclipped_grad_norm": 1.2158186435699463 + }, + { + "bits_per_byte": 2.916199741784094, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.021355628967285, + "step": 570, + "unclipped_grad_norm": 0.9061878323554993 + }, + { + "bits_per_byte": 2.9963629191430594, + "learning_rate": 0.000299626473885547, + "loss_nats": 2.076920509338379, + "step": 580, + "unclipped_grad_norm": 0.9649468660354614 + }, + { + "bits_per_byte": 2.8798801016095843, + "learning_rate": 0.00029958384005397226, + "loss_nats": 1.996180772781372, + "step": 590, + "unclipped_grad_norm": 1.1992034912109375 + }, + { + "bits_per_byte": 2.9395377875061772, + "learning_rate": 0.0002995389065559004, + "loss_nats": 2.037532329559326, + "step": 600, + "unclipped_grad_norm": 0.9665898680686951 + }, + { + "bits_per_byte": 2.883353979174362, + "learning_rate": 0.000299491674159121, + "loss_nats": 1.9985886812210083, + "step": 610, + "unclipped_grad_norm": 1.7724957466125488 + }, + { + "bits_per_byte": 2.887181453069314, + "learning_rate": 0.00029944214367070535, + "loss_nats": 2.001241683959961, + "step": 620, + "unclipped_grad_norm": 0.6843562722206116 + }, + { + "bits_per_byte": 2.834979043104973, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.9650577306747437, + "step": 630, + "unclipped_grad_norm": 0.9718148708343506 + }, + { + "bits_per_byte": 2.792751626890675, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.9357879161834717, + "step": 640, + "unclipped_grad_norm": 0.9564118385314941 + }, + { + "bits_per_byte": 2.744089823759647, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.9020581245422363, + "step": 650, + "unclipped_grad_norm": 0.9242531061172485 + }, + { + "bits_per_byte": 2.674752954272495, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.8539974689483643, + "step": 660, + "unclipped_grad_norm": 1.0690863132476807 + }, + { + "bits_per_byte": 2.722828468546732, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.887320876121521, + "step": 670, + "unclipped_grad_norm": 1.1513385772705078 + }, + { + "bits_per_byte": 2.640349372811248, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.8301507234573364, + "step": 680, + "unclipped_grad_norm": 0.9440202713012695 + }, + { + "bits_per_byte": 2.725472185855795, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.8891533613204956, + "step": 690, + "unclipped_grad_norm": 1.41117262840271 + }, + { + "bits_per_byte": 2.632447973882658, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.8246738910675049, + "step": 700, + "unclipped_grad_norm": 1.0918093919754028 + }, + { + "bits_per_byte": 2.6459176550982333, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.834010362625122, + "step": 710, + "unclipped_grad_norm": 1.155935287475586 + }, + { + "bits_per_byte": 2.6717876294063188, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.8519420623779297, + "step": 720, + "unclipped_grad_norm": 1.1452586650848389 + }, + { + "bits_per_byte": 2.631740953204961, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.8241838216781616, + "step": 730, + "unclipped_grad_norm": 1.3785425424575806 + }, + { + "bits_per_byte": 2.5635412330053144, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.7769113779067993, + "step": 740, + "unclipped_grad_norm": 0.9390878677368164 + }, + { + "bits_per_byte": 2.5502426745273477, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.7676935195922852, + "step": 750, + "unclipped_grad_norm": 1.7016096115112305 + }, + { + "bits_per_byte": 2.4747660244043934, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.7153770923614502, + "step": 760, + "unclipped_grad_norm": 1.2574355602264404 + }, + { + "bits_per_byte": 2.617683607271388, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.8144400119781494, + "step": 770, + "unclipped_grad_norm": 0.8545989990234375 + }, + { + "bits_per_byte": 2.5068349413801783, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.7376055717468262, + "step": 780, + "unclipped_grad_norm": 0.8761343359947205 + }, + { + "bits_per_byte": 2.4782548644597964, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.7177953720092773, + "step": 790, + "unclipped_grad_norm": 1.5282965898513794 + }, + { + "bits_per_byte": 2.424481392979824, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.6805224418640137, + "step": 800, + "unclipped_grad_norm": 1.1385701894760132 + }, + { + "bits_per_byte": 2.479616623089328, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7187392711639404, + "step": 810, + "unclipped_grad_norm": 1.2494919300079346 + }, + { + "bits_per_byte": 2.4295225484419145, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.6840167045593262, + "step": 820, + "unclipped_grad_norm": 0.9844147562980652 + }, + { + "bits_per_byte": 2.4738269991306825, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.714726209640503, + "step": 830, + "unclipped_grad_norm": 1.1955569982528687 + }, + { + "bits_per_byte": 2.54569648513444, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.7645423412322998, + "step": 840, + "unclipped_grad_norm": 0.9406070709228516 + }, + { + "bits_per_byte": 2.442049248782809, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.6926995515823364, + "step": 850, + "unclipped_grad_norm": 1.05335533618927 + }, + { + "bits_per_byte": 2.4350146424145676, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.6878235340118408, + "step": 860, + "unclipped_grad_norm": 1.0259432792663574 + }, + { + "bits_per_byte": 2.339228045103272, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.6214293241500854, + "step": 870, + "unclipped_grad_norm": 0.9182549715042114 + }, + { + "bits_per_byte": 2.3826982079527044, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.6515605449676514, + "step": 880, + "unclipped_grad_norm": 1.0221705436706543 + }, + { + "bits_per_byte": 2.300941783316146, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.5948913097381592, + "step": 890, + "unclipped_grad_norm": 0.8971023559570312 + }, + { + "bits_per_byte": 2.322657688675291, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.6099436283111572, + "step": 900, + "unclipped_grad_norm": 0.8506956100463867 + }, + { + "bits_per_byte": 2.3390856434683576, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.6213306188583374, + "step": 910, + "unclipped_grad_norm": 0.98213130235672 + }, + { + "bits_per_byte": 2.3215339540345536, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.6091647148132324, + "step": 920, + "unclipped_grad_norm": 0.9973281025886536 + }, + { + "bits_per_byte": 2.345137540969569, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.6255254745483398, + "step": 930, + "unclipped_grad_norm": 0.9733822345733643 + }, + { + "bits_per_byte": 2.3491168755452327, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.6282837390899658, + "step": 940, + "unclipped_grad_norm": 0.919996976852417 + }, + { + "bits_per_byte": 2.2933358505817383, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.5896192789077759, + "step": 950, + "unclipped_grad_norm": 0.9114649891853333 + }, + { + "bits_per_byte": 2.3501374205954524, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.6289911270141602, + "step": 960, + "unclipped_grad_norm": 0.8965623378753662 + }, + { + "bits_per_byte": 2.246712557811387, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.557302474975586, + "step": 970, + "unclipped_grad_norm": 0.8982234001159668 + }, + { + "bits_per_byte": 2.3250792043994397, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.6116220951080322, + "step": 980, + "unclipped_grad_norm": 0.8211364150047302 + }, + { + "bits_per_byte": 2.304286157944824, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.5972094535827637, + "step": 990, + "unclipped_grad_norm": 0.9685729742050171 + }, + { + "bits_per_byte": 2.273323949327342, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.575748085975647, + "step": 1000, + "unclipped_grad_norm": 0.8820132613182068 + }, + { + "bits_per_byte": 2.249210949780471, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.5590342283248901, + "step": 1010, + "unclipped_grad_norm": 1.051476001739502 + }, + { + "bits_per_byte": 2.211575126378908, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.532947063446045, + "step": 1020, + "unclipped_grad_norm": 0.9651745557785034 + }, + { + "bits_per_byte": 2.2227080793172105, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.5406638383865356, + "step": 1030, + "unclipped_grad_norm": 0.8863841891288757 + }, + { + "bits_per_byte": 2.2766055502884552, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.5780227184295654, + "step": 1040, + "unclipped_grad_norm": 1.1099046468734741 + }, + { + "bits_per_byte": 2.251583794414425, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5606789588928223, + "step": 1050, + "unclipped_grad_norm": 0.9207155704498291 + }, + { + "bits_per_byte": 2.1771259695151826, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.5090687274932861, + "step": 1060, + "unclipped_grad_norm": 1.1328914165496826 + }, + { + "bits_per_byte": 2.2287740793957926, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.5448684692382812, + "step": 1070, + "unclipped_grad_norm": 0.8940086364746094 + }, + { + "bits_per_byte": 2.210852283297332, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.532446026802063, + "step": 1080, + "unclipped_grad_norm": 0.7960404753684998 + }, + { + "bits_per_byte": 2.195113119020971, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.5215364694595337, + "step": 1090, + "unclipped_grad_norm": 1.0410032272338867 + }, + { + "bits_per_byte": 2.2582911177980747, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5653281211853027, + "step": 1100, + "unclipped_grad_norm": 0.9107176661491394 + }, + { + "bits_per_byte": 2.116759715097035, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.4672260284423828, + "step": 1110, + "unclipped_grad_norm": 0.8636605143547058 + }, + { + "bits_per_byte": 2.1568354563663883, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.495004415512085, + "step": 1120, + "unclipped_grad_norm": 0.8329456448554993 + }, + { + "bits_per_byte": 2.090363129912008, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.4489293098449707, + "step": 1130, + "unclipped_grad_norm": 0.8071860074996948 + }, + { + "bits_per_byte": 2.2289398706712245, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.5449833869934082, + "step": 1140, + "unclipped_grad_norm": 0.8704212307929993 + }, + { + "bits_per_byte": 2.081553662586862, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.442823052406311, + "step": 1150, + "unclipped_grad_norm": 0.7851199507713318 + }, + { + "bits_per_byte": 2.11072106025994, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.4630403518676758, + "step": 1160, + "unclipped_grad_norm": 0.8752713799476624 + }, + { + "bits_per_byte": 2.1676478336434823, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.502498984336853, + "step": 1170, + "unclipped_grad_norm": 0.8719432950019836 + }, + { + "bits_per_byte": 2.148015670082191, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.4888910055160522, + "step": 1180, + "unclipped_grad_norm": 1.007690668106079 + }, + { + "bits_per_byte": 2.061155144333326, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.428683876991272, + "step": 1190, + "unclipped_grad_norm": 0.8875235915184021 + }, + { + "bits_per_byte": 2.036161937579339, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.4113599061965942, + "step": 1200, + "unclipped_grad_norm": 0.8464028239250183 + }, + { + "bits_per_byte": 2.096879896518501, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.453446388244629, + "step": 1210, + "unclipped_grad_norm": 0.8538656234741211 + }, + { + "bits_per_byte": 2.112071468034515, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4639763832092285, + "step": 1220, + "unclipped_grad_norm": 0.8605926036834717 + }, + { + "bits_per_byte": 2.088967318717605, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.4479618072509766, + "step": 1230, + "unclipped_grad_norm": 0.7967903017997742 + }, + { + "bits_per_byte": 2.015986480823972, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.3973753452301025, + "step": 1240, + "unclipped_grad_norm": 0.821725070476532 + }, + { + "bits_per_byte": 2.1025407054716503, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.4573701620101929, + "step": 1250, + "unclipped_grad_norm": 0.8157300353050232 + }, + { + "bits_per_byte": 2.0965937173874654, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.4532480239868164, + "step": 1260, + "unclipped_grad_norm": 0.865153431892395 + }, + { + "bits_per_byte": 2.0842106225600414, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.444664716720581, + "step": 1270, + "unclipped_grad_norm": 0.866759181022644 + }, + { + "bits_per_byte": 1.978468981486203, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.3713701963424683, + "step": 1280, + "unclipped_grad_norm": 0.8000590205192566 + }, + { + "bits_per_byte": 2.094408505825603, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4517333507537842, + "step": 1290, + "unclipped_grad_norm": 0.845427930355072 + }, + { + "bits_per_byte": 2.0327191888743674, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.4089735746383667, + "step": 1300, + "unclipped_grad_norm": 0.7450453639030457 + }, + { + "bits_per_byte": 2.0503357157675466, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.4211844205856323, + "step": 1310, + "unclipped_grad_norm": 0.8611158132553101 + }, + { + "bits_per_byte": 2.0324056645018445, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.4087562561035156, + "step": 1320, + "unclipped_grad_norm": 0.8044684529304504 + }, + { + "bits_per_byte": 2.095409100888322, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.4524269104003906, + "step": 1330, + "unclipped_grad_norm": 0.8400101661682129 + }, + { + "bits_per_byte": 2.034540141181703, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.4102357625961304, + "step": 1340, + "unclipped_grad_norm": 0.737845778465271 + }, + { + "bits_per_byte": 2.042799607989389, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.4159607887268066, + "step": 1350, + "unclipped_grad_norm": 0.7174865007400513 + }, + { + "bits_per_byte": 2.0930082230822777, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.4507627487182617, + "step": 1360, + "unclipped_grad_norm": 0.8992759585380554 + }, + { + "bits_per_byte": 2.094602158290474, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4518675804138184, + "step": 1370, + "unclipped_grad_norm": 0.7615582942962646 + }, + { + "bits_per_byte": 2.0322003172167142, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.408613920211792, + "step": 1380, + "unclipped_grad_norm": 0.7762367725372314 + }, + { + "bits_per_byte": 2.0570738240457005, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.4258549213409424, + "step": 1390, + "unclipped_grad_norm": 0.8229320645332336 + }, + { + "bits_per_byte": 1.9650961265030638, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.3621008396148682, + "step": 1400, + "unclipped_grad_norm": 0.7679391503334045 + }, + { + "bits_per_byte": 2.00868065781532, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.3923113346099854, + "step": 1410, + "unclipped_grad_norm": 0.7845778465270996 + }, + { + "bits_per_byte": 2.043542057093164, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.4164754152297974, + "step": 1420, + "unclipped_grad_norm": 0.834043025970459 + }, + { + "bits_per_byte": 2.0733779513301456, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.437156081199646, + "step": 1430, + "unclipped_grad_norm": 0.8110694289207458 + }, + { + "bits_per_byte": 2.014843656108989, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.3965831995010376, + "step": 1440, + "unclipped_grad_norm": 0.8650574684143066 + }, + { + "bits_per_byte": 1.9678162041091098, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.3639862537384033, + "step": 1450, + "unclipped_grad_norm": 0.768469512462616 + }, + { + "bits_per_byte": 2.0038902530581875, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.388990879058838, + "step": 1460, + "unclipped_grad_norm": 0.8076180815696716 + }, + { + "bits_per_byte": 1.9444059256736643, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.347759485244751, + "step": 1470, + "unclipped_grad_norm": 0.9228054285049438 + }, + { + "bits_per_byte": 2.0012434400614088, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.3871562480926514, + "step": 1480, + "unclipped_grad_norm": 0.7476006746292114 + }, + { + "bits_per_byte": 2.0296324442566815, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.4068340063095093, + "step": 1490, + "unclipped_grad_norm": 0.8021265268325806 + }, + { + "bits_per_byte": 2.0030176130877093, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.3883860111236572, + "step": 1500, + "unclipped_grad_norm": 0.7847866415977478 + }, + { + "bits_per_byte": 2.0241584084623696, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.4030396938323975, + "step": 1510, + "unclipped_grad_norm": 0.7436230182647705 + }, + { + "bits_per_byte": 1.9817667488164974, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.3736560344696045, + "step": 1520, + "unclipped_grad_norm": 0.7735458612442017 + }, + { + "bits_per_byte": 2.0356915650292295, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.4110338687896729, + "step": 1530, + "unclipped_grad_norm": 0.7717568874359131 + }, + { + "bits_per_byte": 1.9954434971437114, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3831360340118408, + "step": 1540, + "unclipped_grad_norm": 0.7771808505058289 + }, + { + "bits_per_byte": 2.0515652197385648, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.4220366477966309, + "step": 1550, + "unclipped_grad_norm": 0.7802103161811829 + }, + { + "bits_per_byte": 1.9670068537541494, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.3634252548217773, + "step": 1560, + "unclipped_grad_norm": 0.84665447473526 + }, + { + "bits_per_byte": 2.0517287752395355, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.4221500158309937, + "step": 1570, + "unclipped_grad_norm": 0.7568546533584595 + }, + { + "bits_per_byte": 1.9842876705128445, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.3754034042358398, + "step": 1580, + "unclipped_grad_norm": 0.7264503240585327 + }, + { + "bits_per_byte": 1.9921176966413265, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3808307647705078, + "step": 1590, + "unclipped_grad_norm": 0.7208127379417419 + }, + { + "bits_per_byte": 2.0161361057302227, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.3974790573120117, + "step": 1600, + "unclipped_grad_norm": 0.7540138959884644 + }, + { + "bits_per_byte": 1.95655615599182, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.3561813831329346, + "step": 1610, + "unclipped_grad_norm": 0.7181152701377869 + }, + { + "bits_per_byte": 1.9151803978645947, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.327501893043518, + "step": 1620, + "unclipped_grad_norm": 0.7724239230155945 + }, + { + "bits_per_byte": 1.9536489612616352, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.3541662693023682, + "step": 1630, + "unclipped_grad_norm": 0.786527156829834 + }, + { + "bits_per_byte": 1.9807548028988207, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.3729546070098877, + "step": 1640, + "unclipped_grad_norm": 0.7289682030677795 + }, + { + "bits_per_byte": 2.047440731805573, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.419177770614624, + "step": 1650, + "unclipped_grad_norm": 0.7997592687606812 + }, + { + "bits_per_byte": 1.9918996226400325, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.3806796073913574, + "step": 1660, + "unclipped_grad_norm": 0.7737360596656799 + }, + { + "bits_per_byte": 1.9986358391090269, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.3853487968444824, + "step": 1670, + "unclipped_grad_norm": 0.7341009974479675 + }, + { + "bits_per_byte": 2.006157672327163, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.3905625343322754, + "step": 1680, + "unclipped_grad_norm": 0.7726286053657532 + }, + { + "bits_per_byte": 2.059701031020281, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.4276759624481201, + "step": 1690, + "unclipped_grad_norm": 0.7421797513961792 + }, + { + "bits_per_byte": 1.945255176003625, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.3483481407165527, + "step": 1700, + "unclipped_grad_norm": 0.7020687460899353 + }, + { + "bits_per_byte": 1.883931494168124, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3058418035507202, + "step": 1710, + "unclipped_grad_norm": 0.6953147649765015 + }, + { + "bits_per_byte": 1.8780419482893271, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.3017594814300537, + "step": 1720, + "unclipped_grad_norm": 0.7373859882354736 + }, + { + "bits_per_byte": 1.871794162548783, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.297428846359253, + "step": 1730, + "unclipped_grad_norm": 0.7177160978317261 + }, + { + "bits_per_byte": 1.9813023956591678, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.3733341693878174, + "step": 1740, + "unclipped_grad_norm": 0.7366491556167603 + }, + { + "bits_per_byte": 1.992689710938096, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.3812272548675537, + "step": 1750, + "unclipped_grad_norm": 0.7558485865592957 + }, + { + "bits_per_byte": 2.003813376813252, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.3889375925064087, + "step": 1760, + "unclipped_grad_norm": 0.8901766538619995 + }, + { + "bits_per_byte": 2.007878186766394, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.3917551040649414, + "step": 1770, + "unclipped_grad_norm": 0.7483155131340027 + }, + { + "bits_per_byte": 1.858201685720492, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.2880072593688965, + "step": 1780, + "unclipped_grad_norm": 0.6878527402877808 + }, + { + "bits_per_byte": 1.9582806260320214, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3573766946792603, + "step": 1790, + "unclipped_grad_norm": 0.7255557775497437 + }, + { + "bits_per_byte": 1.8901213217553274, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.3101322650909424, + "step": 1800, + "unclipped_grad_norm": 0.7484555244445801 + }, + { + "bits_per_byte": 1.931610932397424, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.3388906717300415, + "step": 1810, + "unclipped_grad_norm": 0.6824329495429993 + }, + { + "bits_per_byte": 1.8951939500425259, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.3136483430862427, + "step": 1820, + "unclipped_grad_norm": 0.7226071953773499 + }, + { + "bits_per_byte": 1.9011990682627027, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.3178107738494873, + "step": 1830, + "unclipped_grad_norm": 0.6963138580322266 + }, + { + "bits_per_byte": 1.897375206003419, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.3151602745056152, + "step": 1840, + "unclipped_grad_norm": 0.7190924286842346 + }, + { + "bits_per_byte": 1.9094581911050874, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.3235355615615845, + "step": 1850, + "unclipped_grad_norm": 0.8129149079322815 + }, + { + "bits_per_byte": 1.9019098725587187, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.3183034658432007, + "step": 1860, + "unclipped_grad_norm": 0.7174298167228699 + }, + { + "bits_per_byte": 1.8757370368024637, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.3001618385314941, + "step": 1870, + "unclipped_grad_norm": 0.698638379573822 + }, + { + "bits_per_byte": 1.8930797673154696, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.312182903289795, + "step": 1880, + "unclipped_grad_norm": 0.6759474873542786 + }, + { + "bits_per_byte": 1.9074846901864366, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3221676349639893, + "step": 1890, + "unclipped_grad_norm": 0.7051205039024353 + }, + { + "bits_per_byte": 1.918608528044243, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.3298780918121338, + "step": 1900, + "unclipped_grad_norm": 0.764670193195343 + }, + { + "bits_per_byte": 1.853195958684479, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.2845375537872314, + "step": 1910, + "unclipped_grad_norm": 0.7271832227706909 + }, + { + "bits_per_byte": 1.8660571652813016, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.293452262878418, + "step": 1920, + "unclipped_grad_norm": 0.736711859703064 + }, + { + "bits_per_byte": 1.9508852000622698, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.352250576019287, + "step": 1930, + "unclipped_grad_norm": 0.7282419204711914 + }, + { + "bits_per_byte": 1.8198199735621372, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.2614030838012695, + "step": 1940, + "unclipped_grad_norm": 0.6833932399749756 + }, + { + "bits_per_byte": 1.9028743512647572, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.3189719915390015, + "step": 1950, + "unclipped_grad_norm": 0.6922848224639893 + }, + { + "bits_per_byte": 1.8213626579403766, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.26247239112854, + "step": 1960, + "unclipped_grad_norm": 0.692038357257843 + }, + { + "bits_per_byte": 1.9041305125466592, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.3198426961898804, + "step": 1970, + "unclipped_grad_norm": 0.6990416049957275 + }, + { + "bits_per_byte": 1.9475050530422122, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.349907636642456, + "step": 1980, + "unclipped_grad_norm": 0.7090837955474854 + }, + { + "bits_per_byte": 1.9349700975340764, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.3412190675735474, + "step": 1990, + "unclipped_grad_norm": 0.7101387977600098 + }, + { + "bits_per_byte": 1.907399042826307, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.322108268737793, + "step": 2000, + "unclipped_grad_norm": 0.7119353413581848 + }, + { + "bits_per_byte": 1.8787948883348047, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.302281379699707, + "step": 2010, + "unclipped_grad_norm": 0.6490131616592407 + }, + { + "bits_per_byte": 1.9006151871630235, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.3174060583114624, + "step": 2020, + "unclipped_grad_norm": 0.6715216636657715 + }, + { + "bits_per_byte": 1.977168964628331, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.370469093322754, + "step": 2030, + "unclipped_grad_norm": 0.7426184415817261 + }, + { + "bits_per_byte": 1.9180440809841113, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.3294868469238281, + "step": 2040, + "unclipped_grad_norm": 0.6992321610450745 + }, + { + "bits_per_byte": 1.9641632926047838, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.3614542484283447, + "step": 2050, + "unclipped_grad_norm": 0.7103733420372009 + }, + { + "bits_per_byte": 1.925689053780265, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.3347859382629395, + "step": 2060, + "unclipped_grad_norm": 0.7206700444221497 + }, + { + "bits_per_byte": 1.875058737227461, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.2996916770935059, + "step": 2070, + "unclipped_grad_norm": 0.7435591816902161 + }, + { + "bits_per_byte": 1.8684743814391782, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.2951277494430542, + "step": 2080, + "unclipped_grad_norm": 0.7046454548835754 + }, + { + "bits_per_byte": 1.8726170995331617, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.2979992628097534, + "step": 2090, + "unclipped_grad_norm": 0.7304298877716064 + }, + { + "bits_per_byte": 1.867644909114067, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.2945528030395508, + "step": 2100, + "unclipped_grad_norm": 0.6664448976516724 + }, + { + "bits_per_byte": 1.9067300303144505, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.3216445446014404, + "step": 2110, + "unclipped_grad_norm": 0.7594506740570068 + }, + { + "bits_per_byte": 1.8571168191588492, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.2872552871704102, + "step": 2120, + "unclipped_grad_norm": 0.6419799327850342 + }, + { + "bits_per_byte": 1.824252998370777, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.2644758224487305, + "step": 2130, + "unclipped_grad_norm": 0.6938155889511108 + }, + { + "bits_per_byte": 1.9548898160876103, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.355026364326477, + "step": 2140, + "unclipped_grad_norm": 0.708057165145874 + }, + { + "bits_per_byte": 1.8217463512344514, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2627383470535278, + "step": 2150, + "unclipped_grad_norm": 0.6769831776618958 + }, + { + "bits_per_byte": 1.8793304423095913, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.3026525974273682, + "step": 2160, + "unclipped_grad_norm": 0.8171074390411377 + }, + { + "bits_per_byte": 1.9571610189749045, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.3566006422042847, + "step": 2170, + "unclipped_grad_norm": 0.7180595993995667 + }, + { + "bits_per_byte": 1.832680836193658, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.270317554473877, + "step": 2180, + "unclipped_grad_norm": 0.7150953412055969 + }, + { + "bits_per_byte": 1.8434720596046958, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.2777974605560303, + "step": 2190, + "unclipped_grad_norm": 0.7034876346588135 + }, + { + "bits_per_byte": 1.8713239619813242, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.297102928161621, + "step": 2200, + "unclipped_grad_norm": 0.653628408908844 + }, + { + "bits_per_byte": 1.8320730495055089, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.2698962688446045, + "step": 2210, + "unclipped_grad_norm": 0.658699095249176 + }, + { + "bits_per_byte": 1.8208840302230254, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2621406316757202, + "step": 2220, + "unclipped_grad_norm": 0.6561844348907471 + }, + { + "bits_per_byte": 1.9036004620167, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.3194752931594849, + "step": 2230, + "unclipped_grad_norm": 0.7072258591651917 + }, + { + "bits_per_byte": 1.8385222269302133, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.2743664979934692, + "step": 2240, + "unclipped_grad_norm": 0.7004716992378235 + }, + { + "bits_per_byte": 1.8586491845780369, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2883174419403076, + "step": 2250, + "unclipped_grad_norm": 0.729771077632904 + }, + { + "bits_per_byte": 1.8475716100533126, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.2806390523910522, + "step": 2260, + "unclipped_grad_norm": 0.6963329911231995 + }, + { + "bits_per_byte": 1.8137182010921755, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.2571736574172974, + "step": 2270, + "unclipped_grad_norm": 0.7027154564857483 + }, + { + "bits_per_byte": 1.8373921289313937, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.273583173751831, + "step": 2280, + "unclipped_grad_norm": 0.7033227682113647 + }, + { + "bits_per_byte": 1.8683823707209668, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.2950639724731445, + "step": 2290, + "unclipped_grad_norm": 0.6793314814567566 + }, + { + "bits_per_byte": 1.7955752353047052, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.2445979118347168, + "step": 2300, + "unclipped_grad_norm": 0.6556612849235535 + }, + { + "bits_per_byte": 1.8239810937997627, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.2642873525619507, + "step": 2310, + "unclipped_grad_norm": 0.6811193823814392 + }, + { + "bits_per_byte": 1.8029698013612026, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.2497234344482422, + "step": 2320, + "unclipped_grad_norm": 0.676766037940979 + }, + { + "bits_per_byte": 1.8823826184144536, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.3047682046890259, + "step": 2330, + "unclipped_grad_norm": 0.7007371187210083 + }, + { + "bits_per_byte": 1.7997184693466413, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.2474697828292847, + "step": 2340, + "unclipped_grad_norm": 0.6601847410202026 + }, + { + "bits_per_byte": 1.852642862479304, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2841541767120361, + "step": 2350, + "unclipped_grad_norm": 0.7048074007034302 + }, + { + "bits_per_byte": 1.8561036693626165, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.2865530252456665, + "step": 2360, + "unclipped_grad_norm": 0.6701161861419678 + }, + { + "bits_per_byte": 1.8763671812352252, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.3005986213684082, + "step": 2370, + "unclipped_grad_norm": 0.6814139485359192 + }, + { + "bits_per_byte": 1.8280689494281208, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.2671208381652832, + "step": 2380, + "unclipped_grad_norm": 0.6990898847579956 + }, + { + "bits_per_byte": 1.890114270466642, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.3101273775100708, + "step": 2390, + "unclipped_grad_norm": 0.6828675866127014 + }, + { + "bits_per_byte": 1.833512028345278, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.2708936929702759, + "step": 2400, + "unclipped_grad_norm": 0.6967778205871582 + }, + { + "bits_per_byte": 1.8236293892787483, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2640435695648193, + "step": 2410, + "unclipped_grad_norm": 0.7067846655845642 + }, + { + "bits_per_byte": 1.8038049491137924, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.2503023147583008, + "step": 2420, + "unclipped_grad_norm": 0.6867150068283081 + }, + { + "bits_per_byte": 1.822578231316675, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.263314962387085, + "step": 2430, + "unclipped_grad_norm": 0.6713313460350037 + }, + { + "bits_per_byte": 1.8847618264064898, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.3064173460006714, + "step": 2440, + "unclipped_grad_norm": 0.7129595279693604 + }, + { + "bits_per_byte": 1.83070389562199, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.2689472436904907, + "step": 2450, + "unclipped_grad_norm": 0.6942421793937683 + }, + { + "bits_per_byte": 1.825779860345137, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.2655341625213623, + "step": 2460, + "unclipped_grad_norm": 0.6401145458221436 + }, + { + "bits_per_byte": 1.8126027216186793, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.256400465965271, + "step": 2470, + "unclipped_grad_norm": 0.6519253849983215 + }, + { + "bits_per_byte": 1.8299296297278056, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.268410563468933, + "step": 2480, + "unclipped_grad_norm": 0.6623149514198303 + }, + { + "bits_per_byte": 1.812116870629992, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.25606369972229, + "step": 2490, + "unclipped_grad_norm": 0.6886719465255737 + }, + { + "bits_per_byte": 1.8384610011065061, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2743240594863892, + "step": 2500, + "unclipped_grad_norm": 0.6929810047149658 + }, + { + "bits_per_byte": 1.7915224641371232, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.2417887449264526, + "step": 2510, + "unclipped_grad_norm": 0.7109444737434387 + }, + { + "bits_per_byte": 1.860673420378692, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.2897205352783203, + "step": 2520, + "unclipped_grad_norm": 0.7332566976547241 + }, + { + "bits_per_byte": 1.8426176497952094, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.277205228805542, + "step": 2530, + "unclipped_grad_norm": 0.7004513144493103 + }, + { + "bits_per_byte": 1.8069439764573403, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.2524781227111816, + "step": 2540, + "unclipped_grad_norm": 0.6615054607391357 + }, + { + "bits_per_byte": 1.8029233660454695, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.2496912479400635, + "step": 2550, + "unclipped_grad_norm": 0.7023327946662903 + }, + { + "bits_per_byte": 1.832715920654434, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.2703418731689453, + "step": 2560, + "unclipped_grad_norm": 0.7012928128242493 + }, + { + "bits_per_byte": 1.8136820847354944, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2571486234664917, + "step": 2570, + "unclipped_grad_norm": 0.6797406673431396 + }, + { + "bits_per_byte": 1.8463489853883288, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.2797915935516357, + "step": 2580, + "unclipped_grad_norm": 0.6952629685401917 + }, + { + "bits_per_byte": 1.800200192751708, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.2478036880493164, + "step": 2590, + "unclipped_grad_norm": 0.6217628121376038 + }, + { + "bits_per_byte": 1.8021198631006385, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.2491343021392822, + "step": 2600, + "unclipped_grad_norm": 0.6944329142570496 + }, + { + "bits_per_byte": 1.737192972746911, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.2041304111480713, + "step": 2610, + "unclipped_grad_norm": 0.6685537695884705 + }, + { + "bits_per_byte": 1.8099602081881723, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.2545688152313232, + "step": 2620, + "unclipped_grad_norm": 0.6587880253791809 + }, + { + "bits_per_byte": 1.8246394433872657, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.264743685722351, + "step": 2630, + "unclipped_grad_norm": 0.7501751184463501 + }, + { + "bits_per_byte": 1.6850992558179154, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.16802179813385, + "step": 2640, + "unclipped_grad_norm": 0.6495976448059082 + }, + { + "bits_per_byte": 1.766604757766865, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2245171070098877, + "step": 2650, + "unclipped_grad_norm": 0.6898009777069092 + }, + { + "bits_per_byte": 1.8197308865489903, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.2613413333892822, + "step": 2660, + "unclipped_grad_norm": 0.6980257630348206 + }, + { + "bits_per_byte": 1.8174865129552304, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.2597856521606445, + "step": 2670, + "unclipped_grad_norm": 0.7108778953552246 + }, + { + "bits_per_byte": 1.7355491625699644, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.202991008758545, + "step": 2680, + "unclipped_grad_norm": 0.6760063171386719 + }, + { + "bits_per_byte": 1.909992713183969, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.3239060640335083, + "step": 2690, + "unclipped_grad_norm": 0.6997917890548706 + }, + { + "bits_per_byte": 1.816129569839923, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.2588450908660889, + "step": 2700, + "unclipped_grad_norm": 0.7134007215499878 + }, + { + "bits_per_byte": 1.7323661076677954, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.200784683227539, + "step": 2710, + "unclipped_grad_norm": 0.6671409606933594 + }, + { + "bits_per_byte": 1.7415520450156798, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.2071518898010254, + "step": 2720, + "unclipped_grad_norm": 0.6266056299209595 + }, + { + "bits_per_byte": 1.7764865369201404, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.2313666343688965, + "step": 2730, + "unclipped_grad_norm": 0.6838699579238892 + }, + { + "bits_per_byte": 1.7813295683684371, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.2347235679626465, + "step": 2740, + "unclipped_grad_norm": 0.6395838260650635 + }, + { + "bits_per_byte": 1.7931877721454277, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.2429430484771729, + "step": 2750, + "unclipped_grad_norm": 0.6987690925598145 + }, + { + "bits_per_byte": 1.8191729748295913, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.2609546184539795, + "step": 2760, + "unclipped_grad_norm": 0.6874120831489563 + }, + { + "bits_per_byte": 1.712008349302514, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.1866737604141235, + "step": 2770, + "unclipped_grad_norm": 0.6818541884422302 + }, + { + "bits_per_byte": 1.7247985270645303, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.1955392360687256, + "step": 2780, + "unclipped_grad_norm": 0.6143966317176819 + }, + { + "bits_per_byte": 1.8109692304007845, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2552682161331177, + "step": 2790, + "unclipped_grad_norm": 0.6894205808639526 + }, + { + "bits_per_byte": 1.856113128408414, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.2865595817565918, + "step": 2800, + "unclipped_grad_norm": 0.7088111639022827 + }, + { + "bits_per_byte": 1.7908735735954178, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.2413389682769775, + "step": 2810, + "unclipped_grad_norm": 0.6977660655975342 + }, + { + "bits_per_byte": 1.7772083481058116, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2318669557571411, + "step": 2820, + "unclipped_grad_norm": 0.6387017965316772 + }, + { + "bits_per_byte": 1.8735717752281011, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.2986609935760498, + "step": 2830, + "unclipped_grad_norm": 0.6948992013931274 + }, + { + "bits_per_byte": 1.7648818355705216, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.223322868347168, + "step": 2840, + "unclipped_grad_norm": 0.6597858667373657 + }, + { + "bits_per_byte": 1.7594974027373087, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.219590663909912, + "step": 2850, + "unclipped_grad_norm": 0.6552638411521912 + }, + { + "bits_per_byte": 1.742559519384434, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.2078502178192139, + "step": 2860, + "unclipped_grad_norm": 0.6600345373153687 + }, + { + "bits_per_byte": 1.8186050881164422, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.2605609893798828, + "step": 2870, + "unclipped_grad_norm": 0.6798568367958069 + }, + { + "bits_per_byte": 1.8321973929620827, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2699824571609497, + "step": 2880, + "unclipped_grad_norm": 0.7076214551925659 + }, + { + "bits_per_byte": 1.7303614778893386, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.1993951797485352, + "step": 2890, + "unclipped_grad_norm": 0.6557732224464417 + }, + { + "bits_per_byte": 1.7005912810249864, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.178760051727295, + "step": 2900, + "unclipped_grad_norm": 0.640516996383667 + }, + { + "bits_per_byte": 1.7585642248737272, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.2189438343048096, + "step": 2910, + "unclipped_grad_norm": 0.6746198534965515 + }, + { + "bits_per_byte": 1.7486867452867234, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.2120972871780396, + "step": 2920, + "unclipped_grad_norm": 0.7077968716621399 + }, + { + "bits_per_byte": 1.7558682248388018, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.2170751094818115, + "step": 2930, + "unclipped_grad_norm": 0.6616460084915161 + }, + { + "bits_per_byte": 1.7747749655787537, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.230180263519287, + "step": 2940, + "unclipped_grad_norm": 0.6763441562652588 + }, + { + "bits_per_byte": 1.7743027012194845, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2298529148101807, + "step": 2950, + "unclipped_grad_norm": 0.7340145707130432 + }, + { + "bits_per_byte": 1.8121743128353802, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.256103515625, + "step": 2960, + "unclipped_grad_norm": 0.7379416227340698 + }, + { + "bits_per_byte": 1.7582642871306224, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.218735933303833, + "step": 2970, + "unclipped_grad_norm": 0.6506208777427673 + }, + { + "bits_per_byte": 1.748723549574008, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.2121227979660034, + "step": 2980, + "unclipped_grad_norm": 0.6919231414794922 + }, + { + "bits_per_byte": 1.7729046542506206, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.2288838624954224, + "step": 2990, + "unclipped_grad_norm": 0.7055864334106445 + }, + { + "bits_per_byte": 1.7846963007417274, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.2370572090148926, + "step": 3000, + "unclipped_grad_norm": 0.646892786026001 + }, + { + "bits_per_byte": 1.7670305868104015, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.2248122692108154, + "step": 3010, + "unclipped_grad_norm": 0.6672993898391724 + }, + { + "bits_per_byte": 1.72469482152606, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.1954673528671265, + "step": 3020, + "unclipped_grad_norm": 0.6979844570159912 + }, + { + "bits_per_byte": 1.806369038455506, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.2520796060562134, + "step": 3030, + "unclipped_grad_norm": 0.6935514211654663 + }, + { + "bits_per_byte": 1.7328426715933363, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.2011150121688843, + "step": 3040, + "unclipped_grad_norm": 0.7386679649353027 + }, + { + "bits_per_byte": 1.747656741190706, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.21138334274292, + "step": 3050, + "unclipped_grad_norm": 0.7208184003829956 + }, + { + "bits_per_byte": 1.7560739160892336, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.2172176837921143, + "step": 3060, + "unclipped_grad_norm": 0.7037277817726135 + }, + { + "bits_per_byte": 1.6974938734829472, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.1766130924224854, + "step": 3070, + "unclipped_grad_norm": 0.6785590052604675 + }, + { + "bits_per_byte": 1.7290394472521562, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.1984788179397583, + "step": 3080, + "unclipped_grad_norm": 0.6929227709770203 + }, + { + "bits_per_byte": 1.7905583293963863, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.241120457649231, + "step": 3090, + "unclipped_grad_norm": 0.7170301079750061 + }, + { + "bits_per_byte": 1.6955965608786288, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.1752979755401611, + "step": 3100, + "unclipped_grad_norm": 0.6704404950141907 + }, + { + "bits_per_byte": 1.7847881394772882, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.2371208667755127, + "step": 3110, + "unclipped_grad_norm": 0.6637366414070129 + }, + { + "bits_per_byte": 1.7441582701068548, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.208958387374878, + "step": 3120, + "unclipped_grad_norm": 0.6746242642402649 + }, + { + "bits_per_byte": 1.798037682899759, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.2463047504425049, + "step": 3130, + "unclipped_grad_norm": 0.6646231412887573 + }, + { + "bits_per_byte": 1.760276828111019, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.2201309204101562, + "step": 3140, + "unclipped_grad_norm": 0.6572377681732178 + }, + { + "bits_per_byte": 1.7248595809055864, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.1955815553665161, + "step": 3150, + "unclipped_grad_norm": 0.703731894493103 + }, + { + "bits_per_byte": 1.7581573139217856, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.2186617851257324, + "step": 3160, + "unclipped_grad_norm": 0.6984030604362488 + }, + { + "bits_per_byte": 1.6632455923554292, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.1528739929199219, + "step": 3170, + "unclipped_grad_norm": 0.6426517963409424 + }, + { + "bits_per_byte": 1.7557653792135859, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.2170038223266602, + "step": 3180, + "unclipped_grad_norm": 0.7004299163818359 + }, + { + "bits_per_byte": 1.745408927943929, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.2098252773284912, + "step": 3190, + "unclipped_grad_norm": 0.6766098737716675 + }, + { + "bits_per_byte": 1.6974529416120419, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.1765847206115723, + "step": 3200, + "unclipped_grad_norm": 0.6610512733459473 + }, + { + "bits_per_byte": 1.6953965450556754, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.1751593351364136, + "step": 3210, + "unclipped_grad_norm": 0.6341917514801025 + }, + { + "bits_per_byte": 1.7080972918392427, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.1839628219604492, + "step": 3220, + "unclipped_grad_norm": 0.6630572080612183 + }, + { + "bits_per_byte": 1.8024077620581829, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.2493338584899902, + "step": 3230, + "unclipped_grad_norm": 0.7526578903198242 + }, + { + "bits_per_byte": 1.6841965188835366, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.167396068572998, + "step": 3240, + "unclipped_grad_norm": 0.6205268502235413 + }, + { + "bits_per_byte": 1.8400036854847455, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.2753933668136597, + "step": 3250, + "unclipped_grad_norm": 0.7234300374984741 + }, + { + "bits_per_byte": 1.6895671931046803, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.1711187362670898, + "step": 3260, + "unclipped_grad_norm": 0.6443334221839905 + }, + { + "bits_per_byte": 1.7202893139415583, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.1924136877059937, + "step": 3270, + "unclipped_grad_norm": 0.6670138835906982 + }, + { + "bits_per_byte": 1.7661641382153546, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.2242116928100586, + "step": 3280, + "unclipped_grad_norm": 0.6374489068984985 + }, + { + "bits_per_byte": 1.7692323087067472, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.2263383865356445, + "step": 3290, + "unclipped_grad_norm": 0.705141007900238 + }, + { + "bits_per_byte": 1.7004616061062359, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.1786701679229736, + "step": 3300, + "unclipped_grad_norm": 0.6657145619392395 + }, + { + "bits_per_byte": 1.7267009991483746, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.1968579292297363, + "step": 3310, + "unclipped_grad_norm": 0.6895747780799866 + }, + { + "bits_per_byte": 1.7227281999134436, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.1941041946411133, + "step": 3320, + "unclipped_grad_norm": 0.6446650624275208 + }, + { + "bits_per_byte": 1.733840858898944, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.201806902885437, + "step": 3330, + "unclipped_grad_norm": 0.694030225276947 + }, + { + "bits_per_byte": 1.710304861145718, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.185492992401123, + "step": 3340, + "unclipped_grad_norm": 0.6796198487281799 + }, + { + "bits_per_byte": 1.7090946192315963, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.1846541166305542, + "step": 3350, + "unclipped_grad_norm": 0.6426836252212524 + }, + { + "bits_per_byte": 1.7316100719346026, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.2002606391906738, + "step": 3360, + "unclipped_grad_norm": 0.6927606463432312 + }, + { + "bits_per_byte": 1.6758378180863023, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.161602258682251, + "step": 3370, + "unclipped_grad_norm": 0.6815332174301147 + }, + { + "bits_per_byte": 1.719422005433257, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.191812515258789, + "step": 3380, + "unclipped_grad_norm": 0.6537415385246277 + }, + { + "bits_per_byte": 1.7523346692941746, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.2146258354187012, + "step": 3390, + "unclipped_grad_norm": 0.6900944113731384 + }, + { + "bits_per_byte": 1.8003312435316654, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.247894525527954, + "step": 3400, + "unclipped_grad_norm": 0.6918292045593262 + }, + { + "bits_per_byte": 1.832252083445057, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.2700203657150269, + "step": 3410, + "unclipped_grad_norm": 0.7197720408439636 + }, + { + "bits_per_byte": 1.684495596713387, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.1676033735275269, + "step": 3420, + "unclipped_grad_norm": 0.6265860199928284 + }, + { + "bits_per_byte": 1.634675146463496, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.133070468902588, + "step": 3430, + "unclipped_grad_norm": 0.6592465043067932 + }, + { + "bits_per_byte": 1.7145586800321586, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.188441514968872, + "step": 3440, + "unclipped_grad_norm": 0.7312958240509033 + }, + { + "bits_per_byte": 1.7735012620664639, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.229297399520874, + "step": 3450, + "unclipped_grad_norm": 0.7026493549346924 + }, + { + "bits_per_byte": 1.6997915616984745, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.1782057285308838, + "step": 3460, + "unclipped_grad_norm": 0.7046605944633484 + }, + { + "bits_per_byte": 1.7120762824496047, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.186720848083496, + "step": 3470, + "unclipped_grad_norm": 0.6958264112472534 + }, + { + "bits_per_byte": 1.6523599504690678, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.1453286409378052, + "step": 3480, + "unclipped_grad_norm": 0.6580730676651001 + }, + { + "bits_per_byte": 1.845943794262896, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.279510736465454, + "step": 3490, + "unclipped_grad_norm": 0.7008020281791687 + }, + { + "bits_per_byte": 1.698997173834139, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1776551008224487, + "step": 3500, + "unclipped_grad_norm": 0.6716367602348328 + }, + { + "bits_per_byte": 1.741104374175484, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.2068415880203247, + "step": 3510, + "unclipped_grad_norm": 0.6979703307151794 + }, + { + "bits_per_byte": 1.7289997192598068, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.198451280593872, + "step": 3520, + "unclipped_grad_norm": 0.6473463773727417 + }, + { + "bits_per_byte": 1.7045809345597018, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.181525468826294, + "step": 3530, + "unclipped_grad_norm": 0.6870424151420593 + }, + { + "bits_per_byte": 1.715891889541647, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.1893656253814697, + "step": 3540, + "unclipped_grad_norm": 0.6573967337608337 + }, + { + "bits_per_byte": 1.7137861339644829, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.18790602684021, + "step": 3550, + "unclipped_grad_norm": 0.6936417818069458 + }, + { + "bits_per_byte": 1.664749064689272, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.1539161205291748, + "step": 3560, + "unclipped_grad_norm": 0.65703946352005 + }, + { + "bits_per_byte": 1.677199576715834, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.162546157836914, + "step": 3570, + "unclipped_grad_norm": 0.6742871403694153 + }, + { + "bits_per_byte": 1.701724990659474, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.1795458793640137, + "step": 3580, + "unclipped_grad_norm": 0.7284191846847534 + }, + { + "bits_per_byte": 1.7356530400910855, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.2030630111694336, + "step": 3590, + "unclipped_grad_norm": 0.7113268971443176 + }, + { + "bits_per_byte": 1.7450689182431733, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.2095896005630493, + "step": 3600, + "unclipped_grad_norm": 0.7258033752441406 + }, + { + "bits_per_byte": 1.6724920675964172, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.15928316116333, + "step": 3610, + "unclipped_grad_norm": 0.6585474610328674 + }, + { + "bits_per_byte": 1.6457085214469487, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.1407182216644287, + "step": 3620, + "unclipped_grad_norm": 0.668692946434021 + }, + { + "bits_per_byte": 1.6987481429556897, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.1774824857711792, + "step": 3630, + "unclipped_grad_norm": 0.6788787245750427 + }, + { + "bits_per_byte": 1.6773617563555978, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.1626585721969604, + "step": 3640, + "unclipped_grad_norm": 0.651864767074585 + }, + { + "bits_per_byte": 1.727806159662819, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.1976239681243896, + "step": 3650, + "unclipped_grad_norm": 0.6738831400871277 + }, + { + "bits_per_byte": 1.631005896607338, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.130527138710022, + "step": 3660, + "unclipped_grad_norm": 0.651749849319458 + }, + { + "bits_per_byte": 1.7126844131030556, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.1871423721313477, + "step": 3670, + "unclipped_grad_norm": 0.6611043810844421 + }, + { + "bits_per_byte": 1.6572344547424724, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.148707389831543, + "step": 3680, + "unclipped_grad_norm": 0.6435372233390808 + }, + { + "bits_per_byte": 1.6328050071180138, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.1317741870880127, + "step": 3690, + "unclipped_grad_norm": 0.6872193813323975 + }, + { + "bits_per_byte": 1.6413826418297965, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.137719750404358, + "step": 3700, + "unclipped_grad_norm": 0.6756983399391174 + }, + { + "bits_per_byte": 1.6980223761690487, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.176979422569275, + "step": 3710, + "unclipped_grad_norm": 0.6636074185371399 + }, + { + "bits_per_byte": 1.679294497385994, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.1639982461929321, + "step": 3720, + "unclipped_grad_norm": 0.6621697545051575 + }, + { + "bits_per_byte": 1.6224034963938312, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.1245644092559814, + "step": 3730, + "unclipped_grad_norm": 0.64894038438797 + }, + { + "bits_per_byte": 1.6713036674789552, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.1584594249725342, + "step": 3740, + "unclipped_grad_norm": 0.6720901727676392 + }, + { + "bits_per_byte": 1.689095100728062, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.170791506767273, + "step": 3750, + "unclipped_grad_norm": 0.6983838677406311 + }, + { + "bits_per_byte": 1.7662748950425102, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.2242884635925293, + "step": 3760, + "unclipped_grad_norm": 0.6903348565101624 + }, + { + "bits_per_byte": 1.733212950240644, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.201371669769287, + "step": 3770, + "unclipped_grad_norm": 0.6828967928886414 + }, + { + "bits_per_byte": 1.564032584690994, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.0841047763824463, + "step": 3780, + "unclipped_grad_norm": 0.6709024310112 + }, + { + "bits_per_byte": 1.6655185150692322, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.154449462890625, + "step": 3790, + "unclipped_grad_norm": 0.6959334015846252 + }, + { + "bits_per_byte": 1.6583502781812702, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.1494808197021484, + "step": 3800, + "unclipped_grad_norm": 0.6801181435585022 + }, + { + "bits_per_byte": 1.6910672257855057, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.1721584796905518, + "step": 3810, + "unclipped_grad_norm": 0.670138418674469 + }, + { + "bits_per_byte": 1.6633855862332316, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1529710292816162, + "step": 3820, + "unclipped_grad_norm": 0.6844092011451721 + }, + { + "bits_per_byte": 1.7196521182201114, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.191972017288208, + "step": 3830, + "unclipped_grad_norm": 0.6983829736709595 + }, + { + "bits_per_byte": 1.648461103774008, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.142626166343689, + "step": 3840, + "unclipped_grad_norm": 0.6881207823753357 + }, + { + "bits_per_byte": 1.6013098242655077, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1099433898925781, + "step": 3850, + "unclipped_grad_norm": 0.6545227766036987 + }, + { + "bits_per_byte": 1.6583507941292228, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.149481177330017, + "step": 3860, + "unclipped_grad_norm": 0.7051569223403931 + }, + { + "bits_per_byte": 1.7379079046265478, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.2046259641647339, + "step": 3870, + "unclipped_grad_norm": 0.670116662979126 + }, + { + "bits_per_byte": 1.6388226800717036, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1359453201293945, + "step": 3880, + "unclipped_grad_norm": 0.6294643878936768 + }, + { + "bits_per_byte": 1.6008186418146435, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.109602928161621, + "step": 3890, + "unclipped_grad_norm": 0.678223729133606 + }, + { + "bits_per_byte": 1.681825566058742, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.165752649307251, + "step": 3900, + "unclipped_grad_norm": 0.7286859750747681 + }, + { + "bits_per_byte": 1.6862455201859161, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.168816328048706, + "step": 3910, + "unclipped_grad_norm": 0.7169060707092285 + }, + { + "bits_per_byte": 1.6509453931657212, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.14434814453125, + "step": 3920, + "unclipped_grad_norm": 0.6796501278877258 + }, + { + "bits_per_byte": 1.6775789704436375, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.162809133529663, + "step": 3930, + "unclipped_grad_norm": 0.665902853012085 + }, + { + "bits_per_byte": 1.686020050930635, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.168660044670105, + "step": 3940, + "unclipped_grad_norm": 0.6695700883865356 + }, + { + "bits_per_byte": 1.7230451639389839, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1943238973617554, + "step": 3950, + "unclipped_grad_norm": 0.68856281042099 + }, + { + "bits_per_byte": 1.7090720895043332, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.184638500213623, + "step": 3960, + "unclipped_grad_norm": 0.7034095525741577 + }, + { + "bits_per_byte": 1.6482383862411405, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1424717903137207, + "step": 3970, + "unclipped_grad_norm": 0.7339553236961365 + }, + { + "bits_per_byte": 1.6858351695809575, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.168531894683838, + "step": 3980, + "unclipped_grad_norm": 0.7064217925071716 + }, + { + "bits_per_byte": 1.6987933743928667, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.177513837814331, + "step": 3990, + "unclipped_grad_norm": 0.7165923118591309 + }, + { + "bits_per_byte": 1.5830622930263127, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.0972951650619507, + "step": 4000, + "unclipped_grad_norm": 0.6805729866027832 + }, + { + "bits_per_byte": 1.6785719982697194, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.1634974479675293, + "step": 4010, + "unclipped_grad_norm": 0.7209922671318054 + }, + { + "bits_per_byte": 1.6429657421309887, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.1388170719146729, + "step": 4020, + "unclipped_grad_norm": 0.6874703764915466 + }, + { + "bits_per_byte": 1.6853649690134984, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.168205976486206, + "step": 4030, + "unclipped_grad_norm": 0.6872043609619141 + }, + { + "bits_per_byte": 1.634811872670932, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1331652402877808, + "step": 4040, + "unclipped_grad_norm": 0.7068036794662476 + }, + { + "bits_per_byte": 1.6567664899494747, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.1483830213546753, + "step": 4050, + "unclipped_grad_norm": 0.6833421587944031 + }, + { + "bits_per_byte": 1.6509471129922297, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1443493366241455, + "step": 4060, + "unclipped_grad_norm": 0.6590486764907837 + }, + { + "bits_per_byte": 1.6867212241982026, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.1691460609436035, + "step": 4070, + "unclipped_grad_norm": 0.7028141617774963 + }, + { + "bits_per_byte": 1.752652321250318, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.2148460149765015, + "step": 4080, + "unclipped_grad_norm": 0.7302015423774719 + }, + { + "bits_per_byte": 1.6524000224267188, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.1453564167022705, + "step": 4090, + "unclipped_grad_norm": 0.7509828209877014 + }, + { + "bits_per_byte": 1.64395670616526, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.1395039558410645, + "step": 4100, + "unclipped_grad_norm": 0.7001054883003235 + }, + { + "bits_per_byte": 1.6262314862367355, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.1272177696228027, + "step": 4110, + "unclipped_grad_norm": 0.6860130429267883 + }, + { + "bits_per_byte": 1.6751691495397478, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.1611387729644775, + "step": 4120, + "unclipped_grad_norm": 0.6895889043807983 + }, + { + "bits_per_byte": 1.6342847458460374, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.1327998638153076, + "step": 4130, + "unclipped_grad_norm": 0.704081654548645 + }, + { + "bits_per_byte": 1.6789816609440744, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.1637814044952393, + "step": 4140, + "unclipped_grad_norm": 0.7369034290313721 + }, + { + "bits_per_byte": 1.6230910830319807, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1250410079956055, + "step": 4150, + "unclipped_grad_norm": 0.6953556537628174 + }, + { + "bits_per_byte": 1.6572998081498003, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.1487526893615723, + "step": 4160, + "unclipped_grad_norm": 0.6883883476257324 + }, + { + "bits_per_byte": 1.6989230493116172, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.1776037216186523, + "step": 4170, + "unclipped_grad_norm": 0.7407701015472412 + }, + { + "bits_per_byte": 1.7419734025102938, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.2074439525604248, + "step": 4180, + "unclipped_grad_norm": 0.7870222330093384 + }, + { + "bits_per_byte": 1.6457745627848799, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.1407639980316162, + "step": 4190, + "unclipped_grad_norm": 0.6840526461601257 + }, + { + "bits_per_byte": 1.6703034163815371, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1577661037445068, + "step": 4200, + "unclipped_grad_norm": 0.6972874402999878 + }, + { + "bits_per_byte": 1.6585556254664005, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.149623155593872, + "step": 4210, + "unclipped_grad_norm": 0.6793850660324097 + }, + { + "bits_per_byte": 1.6884861101613569, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.1703693866729736, + "step": 4220, + "unclipped_grad_norm": 0.6979770064353943 + }, + { + "bits_per_byte": 1.6718698343655956, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.1588518619537354, + "step": 4230, + "unclipped_grad_norm": 0.7318124175071716 + }, + { + "bits_per_byte": 1.6108909777450748, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.1165845394134521, + "step": 4240, + "unclipped_grad_norm": 0.6801798343658447 + }, + { + "bits_per_byte": 1.6507507088049445, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1442131996154785, + "step": 4250, + "unclipped_grad_norm": 0.7269574999809265 + }, + { + "bits_per_byte": 1.621517441776586, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1239502429962158, + "step": 4260, + "unclipped_grad_norm": 0.705011248588562 + }, + { + "bits_per_byte": 1.6128925118358162, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.1179718971252441, + "step": 4270, + "unclipped_grad_norm": 0.6755602359771729 + }, + { + "bits_per_byte": 1.6190646252099805, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1222500801086426, + "step": 4280, + "unclipped_grad_norm": 0.6929484009742737 + }, + { + "bits_per_byte": 1.686737046602082, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.1691570281982422, + "step": 4290, + "unclipped_grad_norm": 0.6927577257156372 + }, + { + "bits_per_byte": 1.595531379179168, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.1059380769729614, + "step": 4300, + "unclipped_grad_norm": 0.7213481664657593 + }, + { + "bits_per_byte": 1.6835359335215725, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.1669381856918335, + "step": 4310, + "unclipped_grad_norm": 0.7051133513450623 + }, + { + "bits_per_byte": 1.653136968085666, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.1458672285079956, + "step": 4320, + "unclipped_grad_norm": 0.7240222096443176 + }, + { + "bits_per_byte": 1.6286263446500002, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.1288777589797974, + "step": 4330, + "unclipped_grad_norm": 0.6964207887649536 + }, + { + "bits_per_byte": 1.647465668190814, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.141936182975769, + "step": 4340, + "unclipped_grad_norm": 0.7361802458763123 + }, + { + "bits_per_byte": 1.6392404259306492, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.1362348794937134, + "step": 4350, + "unclipped_grad_norm": 0.7231231927871704 + }, + { + "bits_per_byte": 1.6254455255222926, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.1266729831695557, + "step": 4360, + "unclipped_grad_norm": 0.6956731081008911 + }, + { + "bits_per_byte": 1.6382018227020887, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.1355149745941162, + "step": 4370, + "unclipped_grad_norm": 0.7524617910385132 + }, + { + "bits_per_byte": 1.6896617835626548, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.1711843013763428, + "step": 4380, + "unclipped_grad_norm": 0.741474986076355 + }, + { + "bits_per_byte": 1.6504019279889948, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1439714431762695, + "step": 4390, + "unclipped_grad_norm": 0.7222744226455688 + }, + { + "bits_per_byte": 1.6090913512864464, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.1153371334075928, + "step": 4400, + "unclipped_grad_norm": 0.7083251476287842 + }, + { + "bits_per_byte": 1.6812045367064763, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.165322184562683, + "step": 4410, + "unclipped_grad_norm": 0.7132989168167114 + }, + { + "bits_per_byte": 1.6192730681828262, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.1223945617675781, + "step": 4420, + "unclipped_grad_norm": 0.6899346709251404 + }, + { + "bits_per_byte": 1.6722557634341317, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.1591193675994873, + "step": 4430, + "unclipped_grad_norm": 0.7278249859809875 + }, + { + "bits_per_byte": 1.6107358493939965, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.1164770126342773, + "step": 4440, + "unclipped_grad_norm": 0.7143073081970215 + }, + { + "bits_per_byte": 1.6219625328770189, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.1242587566375732, + "step": 4450, + "unclipped_grad_norm": 0.6768578290939331 + }, + { + "bits_per_byte": 1.5829671866203856, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.097229242324829, + "step": 4460, + "unclipped_grad_norm": 0.6797702312469482 + }, + { + "bits_per_byte": 1.6406014966295777, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1371783018112183, + "step": 4470, + "unclipped_grad_norm": 0.7112294435501099 + }, + { + "bits_per_byte": 1.6555956320624006, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.1475714445114136, + "step": 4480, + "unclipped_grad_norm": 0.7595873475074768 + }, + { + "bits_per_byte": 1.632986620797325, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.1319000720977783, + "step": 4490, + "unclipped_grad_norm": 0.7099776268005371 + }, + { + "bits_per_byte": 1.657930640513165, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.1491899490356445, + "step": 4500, + "unclipped_grad_norm": 0.7175009250640869 + }, + { + "bits_per_byte": 1.605070224926623, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.112549901008606, + "step": 4510, + "unclipped_grad_norm": 0.7242584228515625 + }, + { + "bits_per_byte": 1.6428085499881, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.1387081146240234, + "step": 4520, + "unclipped_grad_norm": 0.7148064970970154 + }, + { + "bits_per_byte": 1.6529692850010747, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.1457509994506836, + "step": 4530, + "unclipped_grad_norm": 0.6877104043960571 + }, + { + "bits_per_byte": 1.7035837791499993, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.1808342933654785, + "step": 4540, + "unclipped_grad_norm": 0.749001145362854 + }, + { + "bits_per_byte": 1.5467483282792818, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.0721242427825928, + "step": 4550, + "unclipped_grad_norm": 0.7023916840553284 + }, + { + "bits_per_byte": 1.6346574322504572, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.1330581903457642, + "step": 4560, + "unclipped_grad_norm": 0.7961910367012024 + }, + { + "bits_per_byte": 1.6243988391091415, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.1259474754333496, + "step": 4570, + "unclipped_grad_norm": 0.7217918634414673 + }, + { + "bits_per_byte": 1.5716245868306817, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.089367151260376, + "step": 4580, + "unclipped_grad_norm": 0.7096751928329468 + }, + { + "bits_per_byte": 1.6213279168953352, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.1238188743591309, + "step": 4590, + "unclipped_grad_norm": 0.7367755770683289 + }, + { + "bits_per_byte": 1.6127642127782724, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.1178829669952393, + "step": 4600, + "unclipped_grad_norm": 0.6875633597373962 + }, + { + "bits_per_byte": 1.663166824301334, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.1528193950653076, + "step": 4610, + "unclipped_grad_norm": 0.7382799386978149 + }, + { + "bits_per_byte": 1.6617116790923838, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.1518107652664185, + "step": 4620, + "unclipped_grad_norm": 0.7443722486495972 + }, + { + "bits_per_byte": 1.6398533720983244, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.1366597414016724, + "step": 4630, + "unclipped_grad_norm": 0.7211660146713257 + }, + { + "bits_per_byte": 1.6973463123685069, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.1765108108520508, + "step": 4640, + "unclipped_grad_norm": 0.7406770586967468 + }, + { + "bits_per_byte": 1.5999324152147474, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.108988642692566, + "step": 4650, + "unclipped_grad_norm": 0.7081073522567749 + }, + { + "bits_per_byte": 1.582279599982236, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.096752643585205, + "step": 4660, + "unclipped_grad_norm": 0.7577162981033325 + }, + { + "bits_per_byte": 1.7184687055995245, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.191151738166809, + "step": 4670, + "unclipped_grad_norm": 0.7871964573860168 + }, + { + "bits_per_byte": 1.7093806263799811, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.1848523616790771, + "step": 4680, + "unclipped_grad_norm": 0.7817367911338806 + }, + { + "bits_per_byte": 1.683371174142046, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.1668239831924438, + "step": 4690, + "unclipped_grad_norm": 0.7802981734275818 + }, + { + "bits_per_byte": 1.6031840911946107, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.1112425327301025, + "step": 4700, + "unclipped_grad_norm": 0.6893032193183899 + }, + { + "bits_per_byte": 1.6654767232850725, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.1544204950332642, + "step": 4710, + "unclipped_grad_norm": 0.7572650909423828 + }, + { + "bits_per_byte": 1.6104028909819261, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.116246223449707, + "step": 4720, + "unclipped_grad_norm": 0.7252569198608398 + }, + { + "bits_per_byte": 1.5375618749834448, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.0657566785812378, + "step": 4730, + "unclipped_grad_norm": 0.6874421834945679 + }, + { + "bits_per_byte": 1.5859369830354846, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.099287748336792, + "step": 4740, + "unclipped_grad_norm": 0.7203320860862732 + }, + { + "bits_per_byte": 1.6280593178501055, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.1284847259521484, + "step": 4750, + "unclipped_grad_norm": 0.7215160727500916 + }, + { + "bits_per_byte": 1.61023022040046, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.116126537322998, + "step": 4760, + "unclipped_grad_norm": 0.7411720156669617 + }, + { + "bits_per_byte": 1.578535193707651, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.0941572189331055, + "step": 4770, + "unclipped_grad_norm": 0.7336283326148987 + }, + { + "bits_per_byte": 1.6225285277810084, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.1246510744094849, + "step": 4780, + "unclipped_grad_norm": 0.7501600980758667 + }, + { + "bits_per_byte": 1.568786185160842, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.0873997211456299, + "step": 4790, + "unclipped_grad_norm": 0.7482179999351501 + }, + { + "bits_per_byte": 1.5826381838092851, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.0970011949539185, + "step": 4800, + "unclipped_grad_norm": 0.7566383481025696 + }, + { + "bits_per_byte": 1.6380320758256872, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.1353973150253296, + "step": 4810, + "unclipped_grad_norm": 0.7491005659103394 + }, + { + "bits_per_byte": 1.5735991196452377, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.0907357931137085, + "step": 4820, + "unclipped_grad_norm": 0.723125159740448 + }, + { + "bits_per_byte": 1.6129458264575836, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.1180088520050049, + "step": 4830, + "unclipped_grad_norm": 0.7223263382911682 + }, + { + "bits_per_byte": 1.6062945694181154, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.113398551940918, + "step": 4840, + "unclipped_grad_norm": 0.7777099609375 + }, + { + "bits_per_byte": 1.6282632892740287, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.1286261081695557, + "step": 4850, + "unclipped_grad_norm": 0.7160045504570007 + }, + { + "bits_per_byte": 1.5516071821314577, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.0754921436309814, + "step": 4860, + "unclipped_grad_norm": 0.7151633501052856 + }, + { + "bits_per_byte": 1.6273311433063524, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.1279799938201904, + "step": 4870, + "unclipped_grad_norm": 0.758723795413971 + }, + { + "bits_per_byte": 1.4805921059371867, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.0262682437896729, + "step": 4880, + "unclipped_grad_norm": 0.7258312106132507 + }, + { + "bits_per_byte": 1.5465180435097763, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.0719646215438843, + "step": 4890, + "unclipped_grad_norm": 0.7507379651069641 + }, + { + "bits_per_byte": 1.5897142379963844, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.1019059419631958, + "step": 4900, + "unclipped_grad_norm": 0.7564383149147034 + }, + { + "bits_per_byte": 1.6788098502758626, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.163662314414978, + "step": 4910, + "unclipped_grad_norm": 0.7541764378547668 + }, + { + "bits_per_byte": 1.6270172749685277, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.1277624368667603, + "step": 4920, + "unclipped_grad_norm": 0.7622291445732117 + }, + { + "bits_per_byte": 1.5870263211460498, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.1000428199768066, + "step": 4930, + "unclipped_grad_norm": 0.7018565535545349 + }, + { + "bits_per_byte": 1.5775168844318925, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.0934513807296753, + "step": 4940, + "unclipped_grad_norm": 0.7143231630325317 + }, + { + "bits_per_byte": 1.5298193880242519, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.0603899955749512, + "step": 4950, + "unclipped_grad_norm": 0.6924887299537659 + }, + { + "bits_per_byte": 1.5264027806822114, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.0580217838287354, + "step": 4960, + "unclipped_grad_norm": 0.7444772720336914 + }, + { + "bits_per_byte": 1.6129908859121096, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.1180400848388672, + "step": 4970, + "unclipped_grad_norm": 0.7593606114387512 + }, + { + "bits_per_byte": 1.6620138526099497, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.1520202159881592, + "step": 4980, + "unclipped_grad_norm": 0.7750640511512756 + }, + { + "bits_per_byte": 1.59956213656744, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.108731985092163, + "step": 4990, + "unclipped_grad_norm": 0.8498972654342651 + }, + { + "bits_per_byte": 1.5537573092325445, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.0769824981689453, + "step": 5000, + "unclipped_grad_norm": 0.7260940074920654 + }, + { + "bits_per_byte": 1.559745057204984, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.0811328887939453, + "step": 5010, + "unclipped_grad_norm": 0.7351677417755127 + }, + { + "bits_per_byte": 1.596499469520875, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.1066091060638428, + "step": 5020, + "unclipped_grad_norm": 0.6992141604423523 + }, + { + "bits_per_byte": 1.582644719150018, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.0970057249069214, + "step": 5030, + "unclipped_grad_norm": 0.7437629699707031 + }, + { + "bits_per_byte": 1.5447051743870315, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.0707080364227295, + "step": 5040, + "unclipped_grad_norm": 0.7290279269218445 + }, + { + "bits_per_byte": 1.5160294751127703, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.0508315563201904, + "step": 5050, + "unclipped_grad_norm": 0.6956131458282471 + }, + { + "bits_per_byte": 1.5367069492260057, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.0651640892028809, + "step": 5060, + "unclipped_grad_norm": 0.7014427781105042 + }, + { + "bits_per_byte": 1.6123297846021931, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.117581844329834, + "step": 5070, + "unclipped_grad_norm": 0.7479101419448853 + }, + { + "bits_per_byte": 1.5785757816132548, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.0941853523254395, + "step": 5080, + "unclipped_grad_norm": 0.7811676859855652 + }, + { + "bits_per_byte": 1.5648066786025274, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.0846413373947144, + "step": 5090, + "unclipped_grad_norm": 0.7735984921455383 + }, + { + "bits_per_byte": 1.5661742846421884, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.0855892896652222, + "step": 5100, + "unclipped_grad_norm": 0.7115684151649475 + }, + { + "bits_per_byte": 1.4899830466048996, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.0327775478363037, + "step": 5110, + "unclipped_grad_norm": 0.710027277469635 + }, + { + "bits_per_byte": 1.5823490809731848, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0968008041381836, + "step": 5120, + "unclipped_grad_norm": 0.6991461515426636 + }, + { + "bits_per_byte": 1.5920475266206402, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.1035232543945312, + "step": 5130, + "unclipped_grad_norm": 0.7620006203651428 + }, + { + "bits_per_byte": 1.5501119649648565, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.074455738067627, + "step": 5140, + "unclipped_grad_norm": 0.7136743664741516 + }, + { + "bits_per_byte": 1.6282715444412703, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.128631830215454, + "step": 5150, + "unclipped_grad_norm": 0.7847793698310852 + }, + { + "bits_per_byte": 1.5038620465295283, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.0423977375030518, + "step": 5160, + "unclipped_grad_norm": 0.7440685033798218 + }, + { + "bits_per_byte": 1.6924628649972575, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.1731258630752563, + "step": 5170, + "unclipped_grad_norm": 0.807824432849884 + }, + { + "bits_per_byte": 1.58750477688075, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.100374460220337, + "step": 5180, + "unclipped_grad_norm": 0.7542273998260498 + }, + { + "bits_per_byte": 1.5365982561906604, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.0650887489318848, + "step": 5190, + "unclipped_grad_norm": 0.7304221391677856 + }, + { + "bits_per_byte": 1.554339642488366, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.0773861408233643, + "step": 5200, + "unclipped_grad_norm": 0.7321885228157043 + }, + { + "bits_per_byte": 1.5664410297336766, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.0857741832733154, + "step": 5210, + "unclipped_grad_norm": 0.7158872485160828 + }, + { + "bits_per_byte": 1.6456338809764741, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.1406664848327637, + "step": 5220, + "unclipped_grad_norm": 0.7951130270957947 + }, + { + "bits_per_byte": 1.5676901397268932, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.0866400003433228, + "step": 5230, + "unclipped_grad_norm": 0.7635700702667236 + }, + { + "bits_per_byte": 1.5527904227693938, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0763123035430908, + "step": 5240, + "unclipped_grad_norm": 0.7247587442398071 + }, + { + "bits_per_byte": 1.5268999825258558, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0583664178848267, + "step": 5250, + "unclipped_grad_norm": 0.7301450371742249 + }, + { + "bits_per_byte": 1.5429303134301273, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.0694777965545654, + "step": 5260, + "unclipped_grad_norm": 0.7427898049354553 + }, + { + "bits_per_byte": 1.5453296433923143, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.0711408853530884, + "step": 5270, + "unclipped_grad_norm": 0.738189160823822 + }, + { + "bits_per_byte": 1.5520582926246709, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.0758048295974731, + "step": 5280, + "unclipped_grad_norm": 0.7907446026802063 + }, + { + "bits_per_byte": 1.5482315066603225, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.0731523036956787, + "step": 5290, + "unclipped_grad_norm": 0.7516751885414124 + }, + { + "bits_per_byte": 1.5624888684168494, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.0830347537994385, + "step": 5300, + "unclipped_grad_norm": 0.7457202672958374 + }, + { + "bits_per_byte": 1.5601533440181323, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.0814158916473389, + "step": 5310, + "unclipped_grad_norm": 0.7586469650268555 + }, + { + "bits_per_byte": 1.5539213806814678, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0770962238311768, + "step": 5320, + "unclipped_grad_norm": 0.7501583099365234 + }, + { + "bits_per_byte": 1.5369119525458341, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.0653061866760254, + "step": 5330, + "unclipped_grad_norm": 0.776537299156189 + }, + { + "bits_per_byte": 1.498436853808061, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.0386372804641724, + "step": 5340, + "unclipped_grad_norm": 0.7446993589401245 + }, + { + "bits_per_byte": 1.591613098444561, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.103222131729126, + "step": 5350, + "unclipped_grad_norm": 0.7884904146194458 + }, + { + "bits_per_byte": 1.507752294092045, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.0450942516326904, + "step": 5360, + "unclipped_grad_norm": 0.7611383199691772 + }, + { + "bits_per_byte": 1.4795394001312556, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.0255385637283325, + "step": 5370, + "unclipped_grad_norm": 0.7561848163604736 + }, + { + "bits_per_byte": 1.5937561742569624, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.1047075986862183, + "step": 5380, + "unclipped_grad_norm": 0.7885301113128662 + }, + { + "bits_per_byte": 1.5351353717624214, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.0640747547149658, + "step": 5390, + "unclipped_grad_norm": 0.7585124373435974 + }, + { + "bits_per_byte": 1.5593752945056292, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.0808765888214111, + "step": 5400, + "unclipped_grad_norm": 0.7710508108139038 + }, + { + "bits_per_byte": 1.5683073854608398, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.0870678424835205, + "step": 5410, + "unclipped_grad_norm": 0.7589167952537537 + }, + { + "bits_per_byte": 1.5314236421915002, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.0615019798278809, + "step": 5420, + "unclipped_grad_norm": 0.767048180103302 + }, + { + "bits_per_byte": 1.584175364742697, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.0980666875839233, + "step": 5430, + "unclipped_grad_norm": 0.8128401041030884 + }, + { + "bits_per_byte": 1.5152878859222498, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.0503175258636475, + "step": 5440, + "unclipped_grad_norm": 0.7724525332450867 + }, + { + "bits_per_byte": 1.5222868918817627, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.055168867111206, + "step": 5450, + "unclipped_grad_norm": 0.7851795554161072 + }, + { + "bits_per_byte": 1.5033344037566811, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.04203200340271, + "step": 5460, + "unclipped_grad_norm": 0.7367083430290222 + }, + { + "bits_per_byte": 1.5478779103301485, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0729072093963623, + "step": 5470, + "unclipped_grad_norm": 0.7894169092178345 + }, + { + "bits_per_byte": 1.5656561009151388, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.0852301120758057, + "step": 5480, + "unclipped_grad_norm": 0.7735373973846436 + }, + { + "bits_per_byte": 1.5637094292900227, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.0838807821273804, + "step": 5490, + "unclipped_grad_norm": 0.7929646372795105 + }, + { + "bits_per_byte": 1.5582739176095037, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.080113172531128, + "step": 5500, + "unclipped_grad_norm": 0.7717483043670654 + }, + { + "bits_per_byte": 1.637272428456826, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.1348707675933838, + "step": 5510, + "unclipped_grad_norm": 0.8310741782188416 + }, + { + "bits_per_byte": 1.556972352907774, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.0792109966278076, + "step": 5520, + "unclipped_grad_norm": 0.7800202965736389 + }, + { + "bits_per_byte": 1.524772385152032, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.056891679763794, + "step": 5530, + "unclipped_grad_norm": 0.7846898436546326 + }, + { + "bits_per_byte": 1.5967227030016948, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.1067638397216797, + "step": 5540, + "unclipped_grad_norm": 0.8770669102668762 + }, + { + "bits_per_byte": 1.5173437665306637, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.0517425537109375, + "step": 5550, + "unclipped_grad_norm": 0.7648112773895264 + }, + { + "bits_per_byte": 1.535894331200679, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.0646008253097534, + "step": 5560, + "unclipped_grad_norm": 0.7926616072654724 + }, + { + "bits_per_byte": 1.485117313446689, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.029404878616333, + "step": 5570, + "unclipped_grad_norm": 0.7731112241744995 + }, + { + "bits_per_byte": 1.5585174450431254, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.0802819728851318, + "step": 5580, + "unclipped_grad_norm": 0.7589283585548401 + }, + { + "bits_per_byte": 1.4983826792730393, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.0385997295379639, + "step": 5590, + "unclipped_grad_norm": 0.7527924180030823 + }, + { + "bits_per_byte": 1.5603170715017538, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.0815293788909912, + "step": 5600, + "unclipped_grad_norm": 0.7817327976226807 + }, + { + "bits_per_byte": 1.5487868386399586, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.0735372304916382, + "step": 5610, + "unclipped_grad_norm": 0.7518688440322876 + }, + { + "bits_per_byte": 1.5629566612271961, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.0833590030670166, + "step": 5620, + "unclipped_grad_norm": 0.7753328680992126 + }, + { + "bits_per_byte": 1.5381488517708395, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0661635398864746, + "step": 5630, + "unclipped_grad_norm": 0.7587193846702576 + }, + { + "bits_per_byte": 1.5073486508104699, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.0448144674301147, + "step": 5640, + "unclipped_grad_norm": 0.7750795483589172 + }, + { + "bits_per_byte": 1.544124216992417, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.070305347442627, + "step": 5650, + "unclipped_grad_norm": 0.810060441493988 + }, + { + "bits_per_byte": 1.6034221151834047, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.1114075183868408, + "step": 5660, + "unclipped_grad_norm": 0.8257125616073608 + }, + { + "bits_per_byte": 1.5092800159796593, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.04615318775177, + "step": 5670, + "unclipped_grad_norm": 0.7718081474304199 + }, + { + "bits_per_byte": 1.5244084698628062, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0566394329071045, + "step": 5680, + "unclipped_grad_norm": 0.7504932284355164 + }, + { + "bits_per_byte": 1.6046094834049616, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.1122305393218994, + "step": 5690, + "unclipped_grad_norm": 0.8047118782997131 + }, + { + "bits_per_byte": 1.5260552037448176, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.0577808618545532, + "step": 5700, + "unclipped_grad_norm": 0.7779625654220581 + }, + { + "bits_per_byte": 1.5308217029134803, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.0610847473144531, + "step": 5710, + "unclipped_grad_norm": 0.7768195867538452 + }, + { + "bits_per_byte": 1.550781837389967, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0749200582504272, + "step": 5720, + "unclipped_grad_norm": 0.7495343089103699 + }, + { + "bits_per_byte": 1.5469536755644118, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.0722665786743164, + "step": 5730, + "unclipped_grad_norm": 0.7540338039398193 + }, + { + "bits_per_byte": 1.5444633667799184, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.070540428161621, + "step": 5740, + "unclipped_grad_norm": 0.82101970911026 + }, + { + "bits_per_byte": 1.5463558638700128, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.071852207183838, + "step": 5750, + "unclipped_grad_norm": 0.8126563429832458 + }, + { + "bits_per_byte": 1.5604214649708275, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.0816017389297485, + "step": 5760, + "unclipped_grad_norm": 0.7811734080314636 + }, + { + "bits_per_byte": 1.5220113756750804, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.0549778938293457, + "step": 5770, + "unclipped_grad_norm": 0.743891179561615 + }, + { + "bits_per_byte": 1.4910395360291497, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.0335098505020142, + "step": 5780, + "unclipped_grad_norm": 0.7729052901268005 + }, + { + "bits_per_byte": 1.5158927489053342, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 1.0507367849349976, + "step": 5790, + "unclipped_grad_norm": 0.7885496616363525 + }, + { + "bits_per_byte": 1.4937343321855192, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.0353777408599854, + "step": 5800, + "unclipped_grad_norm": 0.815751314163208 + }, + { + "bits_per_byte": 1.572335047161396, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.0898596048355103, + "step": 5810, + "unclipped_grad_norm": 0.7941069006919861 + }, + { + "bits_per_byte": 1.5041929411497883, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.0426270961761475, + "step": 5820, + "unclipped_grad_norm": 0.7538033723831177 + }, + { + "bits_per_byte": 1.5036577311403032, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.0422561168670654, + "step": 5830, + "unclipped_grad_norm": 0.7654010653495789 + }, + { + "bits_per_byte": 1.5142389637346376, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.0495904684066772, + "step": 5840, + "unclipped_grad_norm": 0.782671332359314 + }, + { + "bits_per_byte": 1.4626641184632951, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.0138415098190308, + "step": 5850, + "unclipped_grad_norm": 0.7731701731681824 + }, + { + "bits_per_byte": 1.5333125276459265, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.062811255455017, + "step": 5860, + "unclipped_grad_norm": 0.8036836981773376 + }, + { + "bits_per_byte": 1.5458509228070796, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0715022087097168, + "step": 5870, + "unclipped_grad_norm": 0.7855542898178101 + }, + { + "bits_per_byte": 1.5542240701469863, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.0773060321807861, + "step": 5880, + "unclipped_grad_norm": 0.8068346977233887 + }, + { + "bits_per_byte": 1.5580857685894598, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.0799827575683594, + "step": 5890, + "unclipped_grad_norm": 0.7776660919189453 + }, + { + "bits_per_byte": 1.5568578124622994, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.0791316032409668, + "step": 5900, + "unclipped_grad_norm": 0.8147860765457153 + }, + { + "bits_per_byte": 1.5487622451208851, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.0735201835632324, + "step": 5910, + "unclipped_grad_norm": 0.8097375631332397 + }, + { + "bits_per_byte": 1.4962420112177497, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.0371159315109253, + "step": 5920, + "unclipped_grad_norm": 0.7938137054443359 + }, + { + "bits_per_byte": 1.5859806666288039, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.099318027496338, + "step": 5930, + "unclipped_grad_norm": 0.8615660071372986 + }, + { + "bits_per_byte": 1.4582749492306253, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.0107991695404053, + "step": 5940, + "unclipped_grad_norm": 0.7383407354354858 + }, + { + "bits_per_byte": 1.5144308963730004, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.049723505973816, + "step": 5950, + "unclipped_grad_norm": 0.7872124314308167 + }, + { + "bits_per_byte": 1.5454942307891901, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.0712549686431885, + "step": 5960, + "unclipped_grad_norm": 0.8394684195518494 + }, + { + "bits_per_byte": 1.5163290688905733, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.051039218902588, + "step": 5970, + "unclipped_grad_norm": 0.8709865808486938 + }, + { + "bits_per_byte": 1.5581524978579946, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.080029010772705, + "step": 5980, + "unclipped_grad_norm": 0.7908971905708313 + }, + { + "bits_per_byte": 1.5353004751072497, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0641891956329346, + "step": 5990, + "unclipped_grad_norm": 0.809161901473999 + }, + { + "bits_per_byte": 1.4757800313660454, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.022932767868042, + "step": 6000, + "unclipped_grad_norm": 0.7770777940750122 + }, + { + "bits_per_byte": 1.5240531537061237, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0563931465148926, + "step": 6010, + "unclipped_grad_norm": 0.8071803450584412 + }, + { + "bits_per_byte": 1.4806007050697298, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0262742042541504, + "step": 6020, + "unclipped_grad_norm": 0.794861912727356 + }, + { + "bits_per_byte": 1.5097744661008898, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.0464959144592285, + "step": 6030, + "unclipped_grad_norm": 0.7855466604232788 + }, + { + "bits_per_byte": 1.581006240435248, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.095870018005371, + "step": 6040, + "unclipped_grad_norm": 0.8642356991767883 + }, + { + "bits_per_byte": 1.4856858880904416, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.029798984527588, + "step": 6050, + "unclipped_grad_norm": 0.7963036298751831 + }, + { + "bits_per_byte": 1.472290331397388, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.020513892173767, + "step": 6060, + "unclipped_grad_norm": 0.7694856524467468 + }, + { + "bits_per_byte": 1.5551403937107833, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 1.0779411792755127, + "step": 6070, + "unclipped_grad_norm": 0.8520480394363403 + }, + { + "bits_per_byte": 1.4845119345156519, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.0289852619171143, + "step": 6080, + "unclipped_grad_norm": 0.8321053981781006 + }, + { + "bits_per_byte": 1.5000108390287574, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.0397282838821411, + "step": 6090, + "unclipped_grad_norm": 0.7783229947090149 + }, + { + "bits_per_byte": 1.4838635599218992, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.0285358428955078, + "step": 6100, + "unclipped_grad_norm": 0.7777309417724609 + }, + { + "bits_per_byte": 1.5525548065377117, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.0761489868164062, + "step": 6110, + "unclipped_grad_norm": 0.8001437783241272 + }, + { + "bits_per_byte": 1.5519358409772566, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.075719952583313, + "step": 6120, + "unclipped_grad_norm": 0.8187872171401978 + }, + { + "bits_per_byte": 1.481371531310897, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.026808500289917, + "step": 6130, + "unclipped_grad_norm": 0.8310397863388062 + }, + { + "bits_per_byte": 1.500678303696756, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.0401909351348877, + "step": 6140, + "unclipped_grad_norm": 0.8289735913276672 + }, + { + "bits_per_byte": 1.5486944839564452, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.0734732151031494, + "step": 6150, + "unclipped_grad_norm": 0.8152301907539368 + }, + { + "bits_per_byte": 1.4119691364337668, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 0.9787024259567261, + "step": 6160, + "unclipped_grad_norm": 0.7890213131904602 + }, + { + "bits_per_byte": 1.505312720189556, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 1.0434032678604126, + "step": 6170, + "unclipped_grad_norm": 0.7962583899497986 + }, + { + "bits_per_byte": 1.4973528471596727, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0378859043121338, + "step": 6180, + "unclipped_grad_norm": 0.817657470703125 + }, + { + "bits_per_byte": 1.5128529555513341, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.0486297607421875, + "step": 6190, + "unclipped_grad_norm": 0.8283278942108154 + }, + { + "bits_per_byte": 1.5046406119899842, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.0429373979568481, + "step": 6200, + "unclipped_grad_norm": 0.8058339953422546 + }, + { + "bits_per_byte": 1.5145284105360397, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.0497910976409912, + "step": 6210, + "unclipped_grad_norm": 0.8322389125823975 + }, + { + "bits_per_byte": 1.5196104978690357, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0533137321472168, + "step": 6220, + "unclipped_grad_norm": 0.8018003106117249 + }, + { + "bits_per_byte": 1.4532162515381466, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.0072927474975586, + "step": 6230, + "unclipped_grad_norm": 0.8009246587753296 + }, + { + "bits_per_byte": 1.493980267376253, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.035548210144043, + "step": 6240, + "unclipped_grad_norm": 0.858398973941803 + }, + { + "bits_per_byte": 1.5440934320979125, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.0702840089797974, + "step": 6250, + "unclipped_grad_norm": 0.8211870193481445 + }, + { + "bits_per_byte": 1.420789696639893, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 0.9848163723945618, + "step": 6260, + "unclipped_grad_norm": 0.7960385680198669 + }, + { + "bits_per_byte": 1.5192355756901548, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.053053855895996, + "step": 6270, + "unclipped_grad_norm": 0.853398859500885 + }, + { + "bits_per_byte": 1.515179708834857, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.05024254322052, + "step": 6280, + "unclipped_grad_norm": 0.8216928839683533 + }, + { + "bits_per_byte": 1.4594521704757812, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 1.0116151571273804, + "step": 6290, + "unclipped_grad_norm": 0.82486891746521 + }, + { + "bits_per_byte": 1.5286770792572213, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.0595982074737549, + "step": 6300, + "unclipped_grad_norm": 0.8303583860397339 + }, + { + "bits_per_byte": 1.4787826764674592, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.025014042854309, + "step": 6310, + "unclipped_grad_norm": 0.8127238154411316 + }, + { + "bits_per_byte": 1.4303636268481237, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 0.9914525151252747, + "step": 6320, + "unclipped_grad_norm": 0.8019699454307556 + }, + { + "bits_per_byte": 1.5193810730127846, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 1.053154706954956, + "step": 6330, + "unclipped_grad_norm": 0.8504864573478699 + }, + { + "bits_per_byte": 1.4801012674516243, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 1.025928020477295, + "step": 6340, + "unclipped_grad_norm": 0.8109097480773926 + }, + { + "bits_per_byte": 1.5798897290658467, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.0950961112976074, + "step": 6350, + "unclipped_grad_norm": 0.8805332183837891 + }, + { + "bits_per_byte": 1.4671606048701031, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.016958236694336, + "step": 6360, + "unclipped_grad_norm": 0.8624815940856934 + }, + { + "bits_per_byte": 1.4366161421115666, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 0.9957864284515381, + "step": 6370, + "unclipped_grad_norm": 0.7994072437286377 + }, + { + "bits_per_byte": 1.4747529509750927, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.0222208499908447, + "step": 6380, + "unclipped_grad_norm": 0.8374411463737488 + }, + { + "bits_per_byte": 1.5195124677580438, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0532457828521729, + "step": 6390, + "unclipped_grad_norm": 0.8183005452156067 + }, + { + "bits_per_byte": 1.5244810465414704, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.056689739227295, + "step": 6400, + "unclipped_grad_norm": 0.868681013584137 + }, + { + "bits_per_byte": 1.5082558592937714, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.0454432964324951, + "step": 6410, + "unclipped_grad_norm": 0.8705461621284485 + }, + { + "bits_per_byte": 1.4777107086046313, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.024271011352539, + "step": 6420, + "unclipped_grad_norm": 0.83303302526474 + }, + { + "bits_per_byte": 1.5220701937416756, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.055018663406372, + "step": 6430, + "unclipped_grad_norm": 0.8086349368095398 + }, + { + "bits_per_byte": 1.4606352391310664, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.0124351978302002, + "step": 6440, + "unclipped_grad_norm": 0.8565614819526672 + }, + { + "bits_per_byte": 1.5349305404252438, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0639327764511108, + "step": 6450, + "unclipped_grad_norm": 0.8753380179405212 + }, + { + "bits_per_byte": 1.5846364502296602, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.098386287689209, + "step": 6460, + "unclipped_grad_norm": 0.8585286140441895 + }, + { + "bits_per_byte": 1.5606606928381777, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.0817675590515137, + "step": 6470, + "unclipped_grad_norm": 0.8935820460319519 + }, + { + "bits_per_byte": 1.461889336621158, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 1.0133044719696045, + "step": 6480, + "unclipped_grad_norm": 0.8306283354759216 + }, + { + "bits_per_byte": 1.5556855787140185, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0783190727233887, + "step": 6490, + "unclipped_grad_norm": 0.8523006439208984 + }, + { + "bits_per_byte": 1.5229404259550414, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.055621862411499, + "step": 6500, + "unclipped_grad_norm": 0.870681881904602 + }, + { + "bits_per_byte": 1.4682962063137504, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 1.0177453756332397, + "step": 6510, + "unclipped_grad_norm": 0.8530683517456055 + }, + { + "bits_per_byte": 1.4349458466063871, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 0.9946286678314209, + "step": 6520, + "unclipped_grad_norm": 0.7847518920898438 + }, + { + "bits_per_byte": 1.4647672943006964, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.0152993202209473, + "step": 6530, + "unclipped_grad_norm": 0.813064455986023 + }, + { + "bits_per_byte": 1.5062825303577714, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.0440754890441895, + "step": 6540, + "unclipped_grad_norm": 0.8500922322273254 + }, + { + "bits_per_byte": 1.4396140576900818, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 0.9978644251823425, + "step": 6550, + "unclipped_grad_norm": 0.812800943851471 + }, + { + "bits_per_byte": 1.5300410736612142, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.0605436563491821, + "step": 6560, + "unclipped_grad_norm": 0.8465895056724548 + }, + { + "bits_per_byte": 1.4745451959328504, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 1.0220768451690674, + "step": 6570, + "unclipped_grad_norm": 0.8436051607131958 + }, + { + "bits_per_byte": 1.512577267362001, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.0484386682510376, + "step": 6580, + "unclipped_grad_norm": 0.8864383101463318 + }, + { + "bits_per_byte": 1.4839879033784729, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.028622031211853, + "step": 6590, + "unclipped_grad_norm": 0.8163285851478577 + }, + { + "bits_per_byte": 1.422441332027454, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9859611988067627, + "step": 6600, + "unclipped_grad_norm": 0.8087835311889648 + }, + { + "bits_per_byte": 1.4474950766745445, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.0033271312713623, + "step": 6610, + "unclipped_grad_norm": 0.7959599494934082 + }, + { + "bits_per_byte": 1.4354506156866693, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 0.9949785470962524, + "step": 6620, + "unclipped_grad_norm": 0.822913408279419 + }, + { + "bits_per_byte": 1.4764734654143243, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.0234134197235107, + "step": 6630, + "unclipped_grad_norm": 0.8453969955444336 + }, + { + "bits_per_byte": 1.5233788097320906, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.055925726890564, + "step": 6640, + "unclipped_grad_norm": 0.9219359159469604 + }, + { + "bits_per_byte": 1.5062144252280298, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.0440282821655273, + "step": 6650, + "unclipped_grad_norm": 0.809925377368927 + }, + { + "bits_per_byte": 1.5099806732992744, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.0466388463974, + "step": 6660, + "unclipped_grad_norm": 0.8199595212936401 + }, + { + "bits_per_byte": 1.4888895809107139, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.0320196151733398, + "step": 6670, + "unclipped_grad_norm": 0.8765659332275391 + }, + { + "bits_per_byte": 1.4764005447703583, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.0233628749847412, + "step": 6680, + "unclipped_grad_norm": 0.8244244456291199 + }, + { + "bits_per_byte": 1.5037727875337306, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.042335867881775, + "step": 6690, + "unclipped_grad_norm": 0.8424786329269409 + }, + { + "bits_per_byte": 1.5564452260828794, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.0788456201553345, + "step": 6700, + "unclipped_grad_norm": 0.904712975025177 + }, + { + "bits_per_byte": 1.5152017226141676, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.0502578020095825, + "step": 6710, + "unclipped_grad_norm": 0.836388885974884 + }, + { + "bits_per_byte": 1.5406873156975747, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.0679230690002441, + "step": 6720, + "unclipped_grad_norm": 0.8633847832679749 + }, + { + "bits_per_byte": 1.4816277854606825, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 1.0269861221313477, + "step": 6730, + "unclipped_grad_norm": 0.8614170551300049 + }, + { + "bits_per_byte": 1.4234319520964238, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 0.9866478443145752, + "step": 6740, + "unclipped_grad_norm": 0.8195486068725586 + }, + { + "bits_per_byte": 1.514777441414489, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 1.0499637126922607, + "step": 6750, + "unclipped_grad_norm": 0.8303126692771912 + }, + { + "bits_per_byte": 1.4734512142907121, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 1.0213185548782349, + "step": 6760, + "unclipped_grad_norm": 0.8410579562187195 + }, + { + "bits_per_byte": 1.3806758612133734, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 0.9570115804672241, + "step": 6770, + "unclipped_grad_norm": 0.8137068152427673 + }, + { + "bits_per_byte": 1.4863191282109183, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 1.0302379131317139, + "step": 6780, + "unclipped_grad_norm": 0.8781068921089172 + }, + { + "bits_per_byte": 1.4456942463373599, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.002078890800476, + "step": 6790, + "unclipped_grad_norm": 0.8598523139953613 + }, + { + "bits_per_byte": 1.5178521472466142, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0520949363708496, + "step": 6800, + "unclipped_grad_norm": 0.8937218189239502 + }, + { + "bits_per_byte": 1.4967939035443685, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0374984741210938, + "step": 6810, + "unclipped_grad_norm": 0.881040632724762 + }, + { + "bits_per_byte": 1.5062997286228577, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 1.0440874099731445, + "step": 6820, + "unclipped_grad_norm": 0.9157562255859375 + }, + { + "bits_per_byte": 1.4672610427382071, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.0170278549194336, + "step": 6830, + "unclipped_grad_norm": 0.8176568746566772 + }, + { + "bits_per_byte": 1.4378814184739643, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 0.9966634511947632, + "step": 6840, + "unclipped_grad_norm": 0.8490193486213684 + }, + { + "bits_per_byte": 1.4989227047967484, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.0389740467071533, + "step": 6850, + "unclipped_grad_norm": 0.8620986938476562 + }, + { + "bits_per_byte": 1.4607226063177048, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.012495756149292, + "step": 6860, + "unclipped_grad_norm": 0.8512988686561584 + }, + { + "bits_per_byte": 1.4084268957652708, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 0.9762471318244934, + "step": 6870, + "unclipped_grad_norm": 0.8148202896118164 + }, + { + "bits_per_byte": 1.4263828304199277, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 0.9886932373046875, + "step": 6880, + "unclipped_grad_norm": 0.7912194132804871 + }, + { + "bits_per_byte": 1.4300409014037798, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 0.9912288188934326, + "step": 6890, + "unclipped_grad_norm": 0.8738032579421997 + }, + { + "bits_per_byte": 1.5441197454434945, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 1.0703022480010986, + "step": 6900, + "unclipped_grad_norm": 0.8831967711448669 + }, + { + "bits_per_byte": 1.4946933074467301, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.0360424518585205, + "step": 6910, + "unclipped_grad_norm": 0.8877848386764526 + }, + { + "bits_per_byte": 1.4471836160938318, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 1.0031112432479858, + "step": 6920, + "unclipped_grad_norm": 0.8452301621437073 + }, + { + "bits_per_byte": 1.439531420026342, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.9978071451187134, + "step": 6930, + "unclipped_grad_norm": 0.9102557301521301 + }, + { + "bits_per_byte": 1.4125162992374867, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 0.9790816903114319, + "step": 6940, + "unclipped_grad_norm": 0.834708034992218 + }, + { + "bits_per_byte": 1.428224076680065, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 0.9899694919586182, + "step": 6950, + "unclipped_grad_norm": 0.8557949066162109 + }, + { + "bits_per_byte": 1.4928450098979076, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 1.0347613096237183, + "step": 6960, + "unclipped_grad_norm": 0.8528392314910889 + }, + { + "bits_per_byte": 1.4881342331081244, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 1.0314960479736328, + "step": 6970, + "unclipped_grad_norm": 0.8449586629867554 + }, + { + "bits_per_byte": 1.4533909859114233, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.0074138641357422, + "step": 6980, + "unclipped_grad_norm": 0.8697038888931274 + }, + { + "bits_per_byte": 1.4942865684774396, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.035760521888733, + "step": 6990, + "unclipped_grad_norm": 0.9199418425559998 + }, + { + "bits_per_byte": 1.5390417856941192, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 1.0667824745178223, + "step": 7000, + "unclipped_grad_norm": 0.8614180684089661 + }, + { + "bits_per_byte": 1.5199293537037353, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 1.053534746170044, + "step": 7010, + "unclipped_grad_norm": 0.8796895146369934 + }, + { + "bits_per_byte": 1.4606459020554199, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 1.0124425888061523, + "step": 7020, + "unclipped_grad_norm": 0.882849395275116 + }, + { + "bits_per_byte": 1.5166278027551219, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 1.0512462854385376, + "step": 7030, + "unclipped_grad_norm": 0.9236263036727905 + }, + { + "bits_per_byte": 1.5110320032439988, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.0473675727844238, + "step": 7040, + "unclipped_grad_norm": 0.8798070549964905 + }, + { + "bits_per_byte": 1.4849828230137143, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.0293116569519043, + "step": 7050, + "unclipped_grad_norm": 0.8692458868026733 + }, + { + "bits_per_byte": 1.5017370288954675, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 1.0409247875213623, + "step": 7060, + "unclipped_grad_norm": 0.9077903032302856 + }, + { + "bits_per_byte": 1.4637164803039246, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 1.014570951461792, + "step": 7070, + "unclipped_grad_norm": 0.9024561643600464 + }, + { + "bits_per_byte": 1.4319170601420421, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 0.9925292730331421, + "step": 7080, + "unclipped_grad_norm": 0.8655173778533936 + }, + { + "bits_per_byte": 1.470153102995116, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 1.0190324783325195, + "step": 7090, + "unclipped_grad_norm": 0.8878218531608582 + }, + { + "bits_per_byte": 1.466990170063098, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 1.0168401002883911, + "step": 7100, + "unclipped_grad_norm": 0.878326952457428 + }, + { + "bits_per_byte": 1.483959010293128, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 1.0286020040512085, + "step": 7110, + "unclipped_grad_norm": 0.8648343682289124 + }, + { + "bits_per_byte": 1.516946142641869, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.051466941833496, + "step": 7120, + "unclipped_grad_norm": 0.9018513560295105 + }, + { + "bits_per_byte": 1.4770895072697148, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 1.0238404273986816, + "step": 7130, + "unclipped_grad_norm": 0.9407969117164612 + }, + { + "bits_per_byte": 1.4577559055903213, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.010439395904541, + "step": 7140, + "unclipped_grad_norm": 0.8684452772140503 + }, + { + "bits_per_byte": 1.4413475568194534, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 0.9990659952163696, + "step": 7150, + "unclipped_grad_norm": 0.8560382723808289 + }, + { + "bits_per_byte": 1.5182693771576075, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0523841381072998, + "step": 7160, + "unclipped_grad_norm": 0.9146431088447571 + }, + { + "bits_per_byte": 1.4531813390600214, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.0072685480117798, + "step": 7170, + "unclipped_grad_norm": 0.8626469373703003 + }, + { + "bits_per_byte": 1.4719135174093478, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 1.0202527046203613, + "step": 7180, + "unclipped_grad_norm": 0.892516016960144 + }, + { + "bits_per_byte": 1.44853866739998, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 1.0040504932403564, + "step": 7190, + "unclipped_grad_norm": 0.8912770748138428 + }, + { + "bits_per_byte": 1.475482329397402, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 1.0227264165878296, + "step": 7200, + "unclipped_grad_norm": 0.8843866586685181 + }, + { + "bits_per_byte": 1.548766200721855, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 1.073522925376892, + "step": 7210, + "unclipped_grad_norm": 0.915386974811554 + }, + { + "bits_per_byte": 1.5142310525326979, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 1.049584984779358, + "step": 7220, + "unclipped_grad_norm": 0.9140926003456116 + }, + { + "bits_per_byte": 1.4494560228596822, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 1.0046863555908203, + "step": 7230, + "unclipped_grad_norm": 0.8712339997291565 + }, + { + "bits_per_byte": 1.4662377459655733, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 1.0163185596466064, + "step": 7240, + "unclipped_grad_norm": 0.8568669557571411 + }, + { + "bits_per_byte": 1.494899342662464, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 1.0361852645874023, + "step": 7250, + "unclipped_grad_norm": 0.915006160736084 + }, + { + "bits_per_byte": 1.394093775651039, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 0.9663121700286865, + "step": 7260, + "unclipped_grad_norm": 0.862156867980957 + }, + { + "bits_per_byte": 1.4258056566436321, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 0.9882931709289551, + "step": 7270, + "unclipped_grad_norm": 0.9044613242149353 + }, + { + "bits_per_byte": 1.497159366677452, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.0377517938613892, + "step": 7280, + "unclipped_grad_norm": 0.9371742606163025 + }, + { + "bits_per_byte": 1.4271982861589938, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 0.9892584681510925, + "step": 7290, + "unclipped_grad_norm": 0.857363224029541 + }, + { + "bits_per_byte": 1.4553361096926816, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 1.0087621212005615, + "step": 7300, + "unclipped_grad_norm": 0.9214392304420471 + }, + { + "bits_per_byte": 1.4604574090700744, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 1.0123119354248047, + "step": 7310, + "unclipped_grad_norm": 0.9040766954421997 + }, + { + "bits_per_byte": 1.4019453856022042, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 0.9717544913291931, + "step": 7320, + "unclipped_grad_norm": 0.8734325170516968 + }, + { + "bits_per_byte": 1.435404266362262, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 0.9949464201927185, + "step": 7330, + "unclipped_grad_norm": 0.8989059329032898 + }, + { + "bits_per_byte": 1.3964726396777734, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.9679610729217529, + "step": 7340, + "unclipped_grad_norm": 0.863217294216156 + }, + { + "bits_per_byte": 1.4921366133590037, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 1.0342702865600586, + "step": 7350, + "unclipped_grad_norm": 0.9100724458694458 + }, + { + "bits_per_byte": 1.4459979676987837, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 1.0022894144058228, + "step": 7360, + "unclipped_grad_norm": 0.9171856641769409 + }, + { + "bits_per_byte": 1.4742903176442717, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 1.0219001770019531, + "step": 7370, + "unclipped_grad_norm": 0.8950089812278748 + }, + { + "bits_per_byte": 1.3994789824061806, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 0.9700449109077454, + "step": 7380, + "unclipped_grad_norm": 0.8583411574363708 + }, + { + "bits_per_byte": 1.4862434558445388, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 1.0301854610443115, + "step": 7390, + "unclipped_grad_norm": 0.8824332356452942 + }, + { + "bits_per_byte": 1.4193422046589061, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.9838130474090576, + "step": 7400, + "unclipped_grad_norm": 0.8839223980903625 + }, + { + "bits_per_byte": 1.4124227406754173, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 0.9790168404579163, + "step": 7410, + "unclipped_grad_norm": 0.8857054710388184 + }, + { + "bits_per_byte": 1.481928067169089, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0271942615509033, + "step": 7420, + "unclipped_grad_norm": 0.8876377940177917 + }, + { + "bits_per_byte": 1.48142794162038, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 1.0268476009368896, + "step": 7430, + "unclipped_grad_norm": 0.9130218625068665 + }, + { + "bits_per_byte": 1.5177393266276482, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 1.0520167350769043, + "step": 7440, + "unclipped_grad_norm": 0.9722045660018921 + }, + { + "bits_per_byte": 1.4565547787566955, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 1.0096068382263184, + "step": 7450, + "unclipped_grad_norm": 0.8777660131454468 + }, + { + "bits_per_byte": 1.4360160946427063, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 0.9953705072402954, + "step": 7460, + "unclipped_grad_norm": 0.8731237649917603 + }, + { + "bits_per_byte": 1.4290065977414907, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 0.9905118942260742, + "step": 7470, + "unclipped_grad_norm": 0.8881433606147766 + }, + { + "bits_per_byte": 1.4786669321434285, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 1.0249338150024414, + "step": 7480, + "unclipped_grad_norm": 0.9415614008903503 + }, + { + "bits_per_byte": 1.52533012488878, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 1.0572782754898071, + "step": 7490, + "unclipped_grad_norm": 0.9333840012550354 + }, + { + "bits_per_byte": 1.476641664446868, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.0235300064086914, + "step": 7500, + "unclipped_grad_norm": 0.9058641195297241 + }, + { + "bits_per_byte": 1.4478546923974986, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 1.003576397895813, + "step": 7510, + "unclipped_grad_norm": 0.8721203207969666 + }, + { + "bits_per_byte": 1.4558838744356797, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 1.0091418027877808, + "step": 7520, + "unclipped_grad_norm": 0.9465364217758179 + }, + { + "bits_per_byte": 1.5068653795615454, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.044479489326477, + "step": 7530, + "unclipped_grad_norm": 0.9264300465583801 + }, + { + "bits_per_byte": 1.4528602474508607, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.0070459842681885, + "step": 7540, + "unclipped_grad_norm": 0.8836243152618408 + }, + { + "bits_per_byte": 1.4716030887245404, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 1.0200375318527222, + "step": 7550, + "unclipped_grad_norm": 0.9015586376190186 + }, + { + "bits_per_byte": 1.445749280785636, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 1.0021170377731323, + "step": 7560, + "unclipped_grad_norm": 0.9012975692749023 + }, + { + "bits_per_byte": 1.4049243830891243, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 0.973819375038147, + "step": 7570, + "unclipped_grad_norm": 0.9037790894508362 + }, + { + "bits_per_byte": 1.5829785374753427, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 1.0972371101379395, + "step": 7580, + "unclipped_grad_norm": 0.948422372341156 + }, + { + "bits_per_byte": 1.4422967290695652, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 0.9997239112854004, + "step": 7590, + "unclipped_grad_norm": 0.9019100069999695 + }, + { + "bits_per_byte": 1.4206405016902695, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 0.9847129583358765, + "step": 7600, + "unclipped_grad_norm": 0.9176172614097595 + }, + { + "bits_per_byte": 1.4385910188914242, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 0.9971553087234497, + "step": 7610, + "unclipped_grad_norm": 0.9086384177207947 + }, + { + "bits_per_byte": 1.4116053071358663, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 0.9784502387046814, + "step": 7620, + "unclipped_grad_norm": 0.868698239326477 + }, + { + "bits_per_byte": 1.4130682775554309, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 0.9794642925262451, + "step": 7630, + "unclipped_grad_norm": 0.9313812255859375 + }, + { + "bits_per_byte": 1.412768941751604, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.979256808757782, + "step": 7640, + "unclipped_grad_norm": 0.8901358246803284 + }, + { + "bits_per_byte": 1.5564364549676855, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 1.0788395404815674, + "step": 7650, + "unclipped_grad_norm": 0.959291934967041 + }, + { + "bits_per_byte": 1.4663772238954231, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 1.0164152383804321, + "step": 7660, + "unclipped_grad_norm": 0.9459472298622131 + }, + { + "bits_per_byte": 1.4918406312168688, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 1.0340651273727417, + "step": 7670, + "unclipped_grad_norm": 0.9111846685409546 + }, + { + "bits_per_byte": 1.4452396961911296, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 1.0017638206481934, + "step": 7680, + "unclipped_grad_norm": 0.9520997405052185 + }, + { + "bits_per_byte": 1.4557892839777051, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 1.0090762376785278, + "step": 7690, + "unclipped_grad_norm": 0.8787317872047424 + }, + { + "bits_per_byte": 1.39387071415287, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 0.9661575555801392, + "step": 7700, + "unclipped_grad_norm": 0.8992334008216858 + }, + { + "bits_per_byte": 1.4749290612095762, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 1.0223429203033447, + "step": 7710, + "unclipped_grad_norm": 0.8973020315170288 + }, + { + "bits_per_byte": 1.4882167847805385, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 1.0315532684326172, + "step": 7720, + "unclipped_grad_norm": 0.9374682307243347 + }, + { + "bits_per_byte": 1.4924389588592204, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 1.0344798564910889, + "step": 7730, + "unclipped_grad_norm": 0.8935887813568115 + }, + { + "bits_per_byte": 1.428355213451348, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 0.9900603890419006, + "step": 7740, + "unclipped_grad_norm": 0.9144420027732849 + }, + { + "bits_per_byte": 1.4327148876593947, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9930822849273682, + "step": 7750, + "unclipped_grad_norm": 0.931445837020874 + }, + { + "bits_per_byte": 1.462334427721591, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 1.013612985610962, + "step": 7760, + "unclipped_grad_norm": 0.9179406762123108 + }, + { + "bits_per_byte": 1.3952964503285226, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 0.9671458005905151, + "step": 7770, + "unclipped_grad_norm": 0.8928722739219666 + }, + { + "bits_per_byte": 1.465962401741542, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 1.0161277055740356, + "step": 7780, + "unclipped_grad_norm": 0.95033198595047 + }, + { + "bits_per_byte": 1.4099031088489518, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 0.9772703647613525, + "step": 7790, + "unclipped_grad_norm": 0.8963057994842529 + }, + { + "bits_per_byte": 1.4445094578555662, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 1.0012576580047607, + "step": 7800, + "unclipped_grad_norm": 0.900338888168335 + }, + { + "bits_per_byte": 1.4162000816276425, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9816350936889648, + "step": 7810, + "unclipped_grad_norm": 0.9288457036018372 + }, + { + "bits_per_byte": 1.4082734012493758, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 0.9761407375335693, + "step": 7820, + "unclipped_grad_norm": 0.9182540774345398 + }, + { + "bits_per_byte": 1.442771917133899, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 1.0000532865524292, + "step": 7830, + "unclipped_grad_norm": 0.9559081792831421 + }, + { + "bits_per_byte": 1.4321396056922586, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.9926835298538208, + "step": 7840, + "unclipped_grad_norm": 0.9008753895759583 + }, + { + "bits_per_byte": 1.3863931664673312, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.9609745144844055, + "step": 7850, + "unclipped_grad_norm": 0.9209492802619934 + }, + { + "bits_per_byte": 1.4512835105077504, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 1.005953073501587, + "step": 7860, + "unclipped_grad_norm": 0.9152867794036865 + }, + { + "bits_per_byte": 1.4641908084550042, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.014899730682373, + "step": 7870, + "unclipped_grad_norm": 0.951532244682312 + }, + { + "bits_per_byte": 1.3924494495261397, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9651724100112915, + "step": 7880, + "unclipped_grad_norm": 0.9254504442214966 + }, + { + "bits_per_byte": 1.4617833953082267, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 1.0132310390472412, + "step": 7890, + "unclipped_grad_norm": 0.9539278745651245 + }, + { + "bits_per_byte": 1.4563287935534617, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 1.0094501972198486, + "step": 7900, + "unclipped_grad_norm": 0.9324126839637756 + }, + { + "bits_per_byte": 1.4567504950133774, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.0097424983978271, + "step": 7910, + "unclipped_grad_norm": 0.9240184426307678 + }, + { + "bits_per_byte": 1.4753062191629183, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 1.0226043462753296, + "step": 7920, + "unclipped_grad_norm": 0.9371556639671326 + }, + { + "bits_per_byte": 1.4228014636983606, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.986210823059082, + "step": 7930, + "unclipped_grad_norm": 0.9046318531036377 + }, + { + "bits_per_byte": 1.4268002323135718, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 0.9889825582504272, + "step": 7940, + "unclipped_grad_norm": 0.9520049095153809 + }, + { + "bits_per_byte": 1.491558407686803, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 1.0338695049285889, + "step": 7950, + "unclipped_grad_norm": 0.9350817203521729 + }, + { + "bits_per_byte": 1.4239238224778914, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 0.9869887828826904, + "step": 7960, + "unclipped_grad_norm": 0.9114158153533936 + }, + { + "bits_per_byte": 1.45589986882221, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 1.009152889251709, + "step": 7970, + "unclipped_grad_norm": 0.9354726672172546 + }, + { + "bits_per_byte": 1.4961933401275556, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 1.0370821952819824, + "step": 7980, + "unclipped_grad_norm": 0.9774372577667236 + }, + { + "bits_per_byte": 1.5217867663330535, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 1.0548222064971924, + "step": 7990, + "unclipped_grad_norm": 0.9857766032218933 + }, + { + "bits_per_byte": 1.4296398378619677, + "learning_rate": 3e-05, + "loss_nats": 0.9909508228302002, + "step": 8000, + "unclipped_grad_norm": 0.9315535426139832 + } + ], + "variant": "uniform_group_6_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073002.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073002.json new file mode 100644 index 0000000..a5ce6ea --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073002.json @@ -0,0 +1,14045 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "4f67f8f7b22f5a1b89a0dd630263ac8852eb9bc493eb2288a04ce2712ef18c5f", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.00040311424527317286, + 0.00035970969474874437, + 0.00032544726855121553, + 0.00029308689408935606, + 0.00026978139067068696, + 0.00025443214690312743, + 0.00023969457834027708, + 0.0002233884297311306, + 0.0002095247182296589, + 0.0001984660339076072, + 0.0001901928335428238, + 0.00018121300672646612, + 0.0001734284160193056, + 0.0001677417167229578, + 0.00016109299031086266, + 0.00015624657680746168, + 0.0001518603676231578, + 0.00014820833166595548, + 0.00014440594532061368, + 0.0001414585713064298, + 0.00013923499500378966, + 0.00013687640603166074, + 0.00013471563579514623, + 0.00013265138841234148, + 0.00013098257477395236, + 0.0001293915556743741, + 0.00012844328011851758, + 0.0001269879430765286, + 0.0001256384130101651, + 0.00012456141121219844, + 0.0001231025526067242, + 0.00012163266364950687 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00029608183103846386, + "first_to_last_ratio": 2.343484699021421, + "imbalance_abs_log_ratio": 0.8516390092453431, + "last_quartile_mean": 0.0001263425492652459, + "mean": 0.0001858347804954974, + "normalized": [ + 2.16920774571012, + 1.9356424765570717, + 1.751272112160408, + 1.5771369240348272, + 1.451727119925344, + 1.369130935687747, + 1.289826251583108, + 1.2020808437231323, + 1.127478492836466, + 1.067970341065492, + 1.0234512239081746, + 0.9751296621832141, + 0.9332398141881068, + 0.9026389800429313, + 0.8668613586828855, + 0.8407822065969367, + 0.8171794710239251, + 0.7975274126338607, + 0.777066300159633, + 0.7612061150730458, + 0.7492407752334779, + 0.7365489154759011, + 0.7249215428669998, + 0.7138135717041164, + 0.7048334785593375, + 0.6962720074755282, + 0.6911692191098191, + 0.683337870004401, + 0.6760758813563923, + 0.6702804011180051, + 0.662430102042749, + 0.6545204472768429 + ], + "population_cv": 0.39438035124793563 + }, + "activation_output_rms_by_block": [ + 0.0036361459642648697, + 0.005152894649654627, + 0.006386642809957266, + 0.007284422870725393, + 0.0037505170330405235, + 0.0054016136564314365, + 0.006585344672203064, + 0.007922882214188576, + 0.0044892593286931515, + 0.006135277915745974, + 0.007316684816032648, + 0.0082859443500638, + 0.0044554853811860085, + 0.006464940495789051, + 0.007893112488090992, + 0.009196000173687935, + 0.004961193539202213, + 0.006907128728926182, + 0.008217990398406982, + 0.009676996618509293, + 0.005381465889513493, + 0.007502692751586437, + 0.0091153085231781, + 0.010543142445385456, + 0.006136673036962748, + 0.008469858206808567, + 0.010564134456217289, + 0.012301022186875343, + 0.005773921962827444, + 0.008156762458384037, + 0.010457510128617287, + 0.012218860909342766 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.005765057983808219, + "first_to_last_ratio": 0.622587017371887, + "imbalance_abs_log_ratio": 0.4738718734697682, + "last_quartile_mean": 0.009259842918254435, + "mean": 0.007398182220640592, + "normalized": [ + 0.49149180917985336, + 0.6965082091758006, + 0.8632718983507628, + 0.9846233376628902, + 0.5069511565390686, + 0.730127144119427, + 0.8901300989627224, + 1.0709228264321597, + 0.6068057253534959, + 0.8292953231982888, + 0.9889841312125875, + 1.119997332175246, + 0.6022405569783624, + 0.8738552664669654, + 1.0668989020126547, + 1.2430080659586231, + 0.6705962885532487, + 0.9336251153230055, + 1.1108121094231989, + 1.3080235563150786, + 0.7274038039370601, + 1.01412651484229, + 1.232101109614035, + 1.4250990487866828, + 0.8294838994154143, + 1.1448566626512724, + 1.427936504016329, + 1.6627087330393202, + 0.7804514393709395, + 1.1025360304896303, + 1.4135242707919937, + 1.6516031296515918 + ], + "population_cv": 0.30590027610070303 + }, + "bits_per_byte": 8.038491115106986, + "branch_output_rms_by_sublayer": [ + 0.0023513585329055786, + 0.0027412420604377985, + 0.0023208872880786657, + 0.002683233004063368, + 0.002778682392090559, + 0.002711885841563344, + 0.002497476991266012, + 0.0027249811682850122, + 0.002628475893288851, + 0.0027012599166482687, + 0.002920303260907531, + 0.002700963756069541, + 0.002932395786046982, + 0.0026683746837079525, + 0.0032228652853518724, + 0.002738022478297353, + 0.003561305580660701, + 0.0027370047755539417, + 0.003354643704369664, + 0.002654933836311102, + 0.0034162327647209167, + 0.002663377905264497, + 0.0036349669098854065, + 0.002684312639757991, + 0.0035210708156228065, + 0.0027356327045708895, + 0.0038089773152023554, + 0.002652956871315837, + 0.003915763925760984, + 0.002679797587916255, + 0.0037244565319269896, + 0.002698024269193411, + 0.004162859171628952, + 0.0026484597474336624, + 0.004167158622294664, + 0.0026680538430809975, + 0.004440344404429197, + 0.002654345240443945, + 0.0041700867004692554, + 0.002591830212622881, + 0.004737259820103645, + 0.002620960585772991, + 0.004552462603896856, + 0.002651234855875373, + 0.004706811159849167, + 0.0026270023081451654, + 0.005186605267226696, + 0.002615687670186162, + 0.005580419674515724, + 0.00258000404573977, + 0.004905648995190859, + 0.002626793459057808, + 0.005227713380008936, + 0.002573782578110695, + 0.005349088925868273, + 0.002592637902125716, + 0.005235649179667234, + 0.002581534907221794, + 0.004925078712403774, + 0.002551236655563116, + 0.005810525268316269, + 0.002563327783718705, + 0.005564783234149218, + 0.002589041367173195 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.010012756197717748, + 0.008904953530422711, + 0.008980942528705073, + 0.007787912846174446, + 0.007907551845183929, + 0.008289745083748344, + 0.007663833460373458, + 0.007350080301903749, + 0.00709564946584838, + 0.006310668475383808, + 0.006285078227083486, + 0.006362421438950662, + 0.005604026016423943, + 0.005959901601525332, + 0.006167718108848569, + 0.005523196426059091, + 0.0054641843609823, + 0.00557989785603791, + 0.005260407000687673, + 0.004763357766419076, + 0.004970214176820314, + 0.004845025235106639, + 0.004921304265254701, + 0.004820106373346748, + 0.0054804096177630225, + 0.0042302744374162005, + 0.004588429583035324, + 0.004469681656036737, + 0.004349111730454971, + 0.003928502929307311, + 0.004468266803351117, + 0.003904642007649288 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.008362221974278682, + "first_to_last_ratio": 1.8887369414995314, + "imbalance_abs_log_ratio": 0.6359083207487641, + "last_quartile_mean": 0.004427414845626747, + "mean": 0.00600782035481319, + "normalized": [ + 1.6666204390908572, + 1.4822269982304765, + 1.4948753455169401, + 1.2962958920592769, + 1.3162097696294732, + 1.3798257261648947, + 1.2756429133626719, + 1.2234187888150154, + 1.1810688480662828, + 1.0504089840715678, + 1.0461494944748422, + 1.0590232502297414, + 0.9327885465041001, + 0.9920239370590588, + 1.0266149359654666, + 0.9193344840336578, + 0.9095119424808776, + 0.9287724210274617, + 0.875593258455752, + 0.7928595538984271, + 0.8272907449435313, + 0.8064530809788657, + 0.8191497039874, + 0.8023053434820334, + 0.9122126318860999, + 0.7041279844573081, + 0.7637428072161453, + 0.7439772483303089, + 0.7239084182952745, + 0.6538982022256998, + 0.7437417465006834, + 0.6499265585598059 + ], + "population_cv": 0.2648157248292805 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.027978340163826942, + 0.014031016267836094, + 0.0140878576785326, + 0.014151837676763535, + 0.014187770895659924, + 0.014265215955674648, + 0.014337530359625816, + 0.014401593245565891, + 0.014477464370429516, + 0.009722241200506687, + 0.009757626801729202, + 0.009786579757928848, + 0.009825476445257664, + 0.009839320555329323, + 0.00987118761986494, + 0.00996283907443285, + 0.009999874979257584, + 0.007562503684312105, + 0.007595349568873644, + 0.007635751739144325, + 0.007658981718122959, + 0.007660485804080963, + 0.007689184043556452, + 0.007726665120571852, + 0.007755559403449297, + 0.006238248664885759, + 0.00625597033649683, + 0.006290202960371971, + 0.006315375678241253, + 0.006327399518340826, + 0.00635211868211627, + 0.006382675375789404, + 0.006408242508769035, + 0.005390914157032967, + 0.005408597644418478, + 0.0054517327807843685, + 0.005475745070725679, + 0.005509276874363422, + 0.005517482291907072, + 0.0055727302096784115, + 0.005585856735706329, + 0.0048178439028561115, + 0.004830927588045597, + 0.004884926602244377, + 0.004899685736745596, + 0.004938251338899136, + 0.004948681220412254, + 0.004975184798240662, + 0.004989070352166891, + 0.004412420094013214, + 0.004419817589223385, + 0.004455919377505779, + 0.004463871940970421, + 0.004530393984168768, + 0.004542885348200798, + 0.0045819394290447235, + 0.004592607729136944, + 0.004127359017729759, + 0.004137600306421518, + 0.004185030702501535, + 0.004193917382508516, + 0.004221425391733646, + 0.004230021964758635, + 0.004281661007553339 + ], + "loss_nats": 5.571857452392578, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0023513585329055786, + 0.0036361459642648697, + 0.004408807028084993, + 0.005152894649654627, + 0.005783648230135441, + 0.006386642809957266, + 0.006751363631337881, + 0.007284422870725393, + 0.002628475893288851, + 0.0037505170330405235, + 0.004723479505628347, + 0.0054016136564314365, + 0.006024772301316261, + 0.006585344672203064, + 0.007445912808179855, + 0.007922882214188576, + 0.003561305580660701, + 0.0044892593286931515, + 0.005543973762542009, + 0.006135277915745974, + 0.006855612155050039, + 0.007316684816032648, + 0.007876631803810596, + 0.0082859443500638, + 0.0035210708156228065, + 0.0044554853811860085, + 0.0058924350887537, + 0.006464940495789051, + 0.007390674669295549, + 0.007893112488090992, + 0.008790239691734314, + 0.009196000173687935, + 0.004162859171628952, + 0.004961193539202213, + 0.006355749908834696, + 0.006907128728926182, + 0.007784162648022175, + 0.008217990398406982, + 0.00937180407345295, + 0.009676996618509293, + 0.004737259820103645, + 0.005381465889513493, + 0.006999304983764887, + 0.007502692751586437, + 0.008728929795324802, + 0.0091153085231781, + 0.010217384435236454, + 0.010543142445385456, + 0.005580419674515724, + 0.006136673036962748, + 0.008008211851119995, + 0.008469858206808567, + 0.010209310799837112, + 0.010564134456217289, + 0.012040933594107628, + 0.012301022186875343, + 0.005235649179667234, + 0.005773921962827444, + 0.007745372597128153, + 0.008156762458384037, + 0.010180563665926456, + 0.010457510128617287, + 0.011906592175364494, + 0.012218860909342766 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.918300641700625e-05, + 6.569862307514995e-05, + 6.211578147485852e-05, + 5.870820314157754e-05, + 5.3743147873319685e-05, + 5.017181683797389e-05, + 4.677380638895556e-05, + 4.3350388295948505e-05, + 3.90466047974769e-05, + 3.6370460293255746e-05, + 3.3763986721169204e-05, + 3.127431045868434e-05, + 2.8748780096066184e-05, + 2.706669874896761e-05, + 2.5477476810920052e-05, + 2.406512612651568e-05, + 2.296643469890114e-05, + 2.192013016610872e-05, + 2.1079362340969965e-05, + 2.029096140176989e-05, + 1.9626824723673053e-05, + 1.8995608115801588e-05, + 1.8462749721948057e-05, + 1.7994259906117804e-05, + 1.7608077541808598e-05, + 1.7285083231399767e-05, + 1.7000989828375168e-05, + 1.6760361177148297e-05, + 1.6355328625650145e-05, + 1.6166051864274777e-05, + 1.599646384420339e-05, + 1.5857614926062524e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.621809668809874e-05, + "first_to_last_ratio": 3.3807778051248394, + "imbalance_abs_log_ratio": 1.2181058028808338, + "last_quartile_mean": 1.6628746379865333e-05, + "mean": 3.093514123975183e-05, + "normalized": [ + 2.2363888976884874, + 2.1237537778145605, + 2.0079359261188503, + 1.8977835816743314, + 1.7372847098645032, + 1.6218389452026432, + 1.5119958892849974, + 1.4013315135682334, + 1.2622087125725419, + 1.1757004764057615, + 1.0914444016755382, + 1.010963881376972, + 0.9293243523040341, + 0.8749499004771555, + 0.8235771937637496, + 0.777921973590082, + 0.7424060074886337, + 0.7085834842719655, + 0.681405078373551, + 0.6559194685588147, + 0.6344507875869166, + 0.6140462708278224, + 0.5968212518850026, + 0.581676992086756, + 0.5691933780209194, + 0.5587523618346484, + 0.5495688445905268, + 0.541790355739871, + 0.5286973962360145, + 0.5225788930131442, + 0.5170968420744705, + 0.5126084540285014 + ], + "population_cv": 0.5268224121184485 + }, + "activation_output_rms_by_block": [ + 0.005867472849786282, + 0.009899183176457882, + 0.014523761346936226, + 0.019194046035408974, + 0.0068074981682002544, + 0.012258335016667843, + 0.01779257506132126, + 0.02363724075257778, + 0.008278015069663525, + 0.014763166196644306, + 0.021510249003767967, + 0.0282937902957201, + 0.009244468063116074, + 0.01668422669172287, + 0.025010215118527412, + 0.03293043002486229, + 0.010345770977437496, + 0.019711239263415337, + 0.02933182567358017, + 0.039894845336675644, + 0.013393603265285492, + 0.024709276854991913, + 0.03661581501364708, + 0.05052796006202698, + 0.015496755950152874, + 0.030097337439656258, + 0.04464881867170334, + 0.06112745404243469, + 0.019440028816461563, + 0.03749694302678108, + 0.05591440200805664, + 0.07976537942886353 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013747514050919563, + "first_to_last_ratio": 0.3197216006351338, + "imbalance_abs_log_ratio": 1.1403046598714879, + "last_quartile_mean": 0.04299838992301375, + "mean": 0.026100379021954723, + "normalized": [ + 0.22480412429454644, + 0.3792735411286954, + 0.5564578711565586, + 0.7353933833398977, + 0.2608198970012668, + 0.46966118792208195, + 0.681697957196513, + 0.9056282566891065, + 0.3171607225588697, + 0.5656303375604641, + 0.8241355033838513, + 1.0840375257355594, + 0.35418903516075195, + 0.6392331190933543, + 0.9582318746210428, + 1.2616839777369657, + 0.3963839363687017, + 0.755208928070927, + 1.1238084186021693, + 1.5285159385278466, + 0.5131574240366112, + 0.9467018403911811, + 1.4028844172280848, + 1.9359090540227264, + 0.5937368165082026, + 1.1531379454045259, + 1.7106578657017326, + 2.3420140370764893, + 0.7448178741047896, + 1.4366436209696407, + 2.1422831431307343, + 3.0561004252761115 + ], + "population_cv": 0.6562424505820315 + }, + "bits_per_byte": 5.118470606751324, + "branch_output_rms_by_sublayer": [ + 0.0030840476974844933, + 0.004966099280864, + 0.002655037911608815, + 0.004918529186397791, + 0.003271719440817833, + 0.0055613163858652115, + 0.0027445550076663494, + 0.005832584574818611, + 0.003107215277850628, + 0.006150941364467144, + 0.0030507307965308428, + 0.006149527616798878, + 0.003512246999889612, + 0.00637229485437274, + 0.0038306997157633305, + 0.006466181017458439, + 0.004710398614406586, + 0.006538694724440575, + 0.004033357836306095, + 0.007035540416836739, + 0.004843658767640591, + 0.006693827919661999, + 0.005020692013204098, + 0.006725885905325413, + 0.0051618111319839954, + 0.006978998892009258, + 0.005093796644359827, + 0.006672017741948366, + 0.006545273587107658, + 0.00692571559920907, + 0.005800293292850256, + 0.007155226543545723, + 0.006114646792411804, + 0.007234795019030571, + 0.007061939220875502, + 0.007538367062807083, + 0.0074903410859405994, + 0.007269904948771, + 0.008021182380616665, + 0.007699424400925636, + 0.008909563533961773, + 0.007777373306453228, + 0.008841089904308319, + 0.007820341736078262, + 0.009142287075519562, + 0.00803503580391407, + 0.011494839563965797, + 0.008086730726063251, + 0.010605492629110813, + 0.007917716167867184, + 0.011382341384887695, + 0.008475739508867264, + 0.011174763552844524, + 0.008375676348805428, + 0.013001051731407642, + 0.008532542735338211, + 0.013455186039209366, + 0.00911042746156454, + 0.013846703805029392, + 0.00917432364076376, + 0.01532104704529047, + 0.008162973448634148, + 0.02034594677388668, + 0.008917862549424171 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0002896418808574006, + 0.00022661045816944742, + 0.0002314713770153176, + 0.00020091647105038477, + 0.00020794290188369758, + 0.00019081077480906144, + 0.00021022076180580377, + 0.00020394051786795197, + 0.00020942208376948285, + 0.00017983848700119638, + 0.00018958120678526298, + 0.00018922310550907825, + 0.00018568247476386356, + 0.00017154313102846677, + 0.00020642320446521382, + 0.00017889122600891086, + 0.00017637798795408973, + 0.00018343752269128468, + 0.00017388789725190538, + 0.00017233038347103785, + 0.00017736576877746426, + 0.00016634634241705708, + 0.0001606641802499389, + 0.0001658655655482842, + 0.0001604642275406402, + 0.00015364811835042724, + 0.00014433547076812394, + 0.00015780503957837236, + 0.0001502864744307819, + 0.00014656770050506659, + 0.00014671070034449045, + 0.0001716411833726761 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00022019439293238314, + "first_to_last_ratio": 1.430461968449502, + "imbalance_abs_log_ratio": 0.357997446960622, + "last_quartile_mean": 0.00015393236436132235, + "mean": 0.00018374670706381818, + "normalized": [ + 1.5763105934562591, + 1.233276295344667, + 1.2597307495416714, + 1.0934425670039531, + 1.1316823316538438, + 1.0384445950521977, + 1.1440790703954804, + 1.1099002595846257, + 1.139732445364298, + 0.9787303940023022, + 1.0317529484728039, + 1.029804063064695, + 1.01053497899896, + 0.933584790550217, + 1.1234117212969685, + 0.9735751397535471, + 0.9598974086258364, + 0.998317333804376, + 0.9463456517428164, + 0.9378692343650259, + 0.9652731829140333, + 0.9053024409263689, + 0.8743785552256871, + 0.9026859212811711, + 0.8732903577145924, + 0.8361952211587813, + 0.7855132376222335, + 0.8588183271418526, + 0.8179002325118403, + 0.7976616443752719, + 0.7984398887406209, + 0.9341184183130007 + ], + "population_cv": 0.16089493367899624 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6923023462295532, + "mean_weights": [ + 0.48015159368515015, + 0.5198484063148499 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927933692932129, + "mean_weights": [ + 0.5129696726799011, + 0.4870303273200989 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928650140762329, + "mean_weights": [ + 0.5081887245178223, + 0.49181127548217773 + ], + "sources": 2 + }, + { + "entropy_mean": 0.692642092704773, + "mean_weights": [ + 0.515070915222168, + 0.48492908477783203 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6925288438796997, + "mean_weights": [ + 0.5162286758422852, + 0.48377129435539246 + ], + "sources": 2 + }, + { + "entropy_mean": 0.692801296710968, + "mean_weights": [ + 0.5107811689376831, + 0.4892188608646393 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6924335956573486, + "mean_weights": [ + 0.5171298384666443, + 0.4828701615333557 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929628849029541, + "mean_weights": [ + 0.5073151588439941, + 0.49268481135368347 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0978691577911377, + "mean_weights": [ + 0.3493620753288269, + 0.32304954528808594, + 0.32758840918540955 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0983185768127441, + "mean_weights": [ + 0.3433151841163635, + 0.32686397433280945, + 0.32982081174850464 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098139762878418, + "mean_weights": [ + 0.3443644642829895, + 0.3302045464515686, + 0.3254309594631195 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984498262405396, + "mean_weights": [ + 0.3400895893573761, + 0.3297577500343323, + 0.330152690410614 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982575416564941, + "mean_weights": [ + 0.34352922439575195, + 0.33093199133872986, + 0.3255387544631958 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984960794448853, + "mean_weights": [ + 0.3382323384284973, + 0.3337849974632263, + 0.32798269391059875 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0980844497680664, + "mean_weights": [ + 0.34468135237693787, + 0.33182331919670105, + 0.3234953284263611 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985528230667114, + "mean_weights": [ + 0.33591216802597046, + 0.3344636559486389, + 0.3296241760253906 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3860273361206055, + "mean_weights": [ + 0.25755617022514343, + 0.2500244975090027, + 0.2457255870103836, + 0.24669376015663147 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862013816833496, + "mean_weights": [ + 0.2531728744506836, + 0.25236332416534424, + 0.2467680126428604, + 0.24769577383995056 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3858840465545654, + "mean_weights": [ + 0.2582244575023651, + 0.25243258476257324, + 0.2448945790529251, + 0.24444839358329773 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861620426177979, + "mean_weights": [ + 0.25381484627723694, + 0.2533356547355652, + 0.24765396118164062, + 0.24519556760787964 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3855750560760498, + "mean_weights": [ + 0.2592918872833252, + 0.25440728664398193, + 0.2462831735610962, + 0.2400176227092743 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861032724380493, + "mean_weights": [ + 0.25141841173171997, + 0.2558484971523285, + 0.24897170066833496, + 0.24376140534877777 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3856598138809204, + "mean_weights": [ + 0.2575046420097351, + 0.2521578371524811, + 0.24600350856781006, + 0.24433401226997375 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862515687942505, + "mean_weights": [ + 0.25004899501800537, + 0.25259798765182495, + 0.25026780366897583, + 0.24708521366119385 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6089379787445068, + "mean_weights": [ + 0.20501357316970825, + 0.20669279992580414, + 0.19997380673885345, + 0.1957656890153885, + 0.19255411624908447 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093051433563232, + "mean_weights": [ + 0.20157386362552643, + 0.20416845381259918, + 0.20096652209758759, + 0.19674228131771088, + 0.19654889404773712 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085665225982666, + "mean_weights": [ + 0.20556388795375824, + 0.20921266078948975, + 0.20156028866767883, + 0.1941545158624649, + 0.18950864672660828 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093413829803467, + "mean_weights": [ + 0.20107820630073547, + 0.20404160022735596, + 0.2001822292804718, + 0.19849099218845367, + 0.1962069571018219 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6080409288406372, + "mean_weights": [ + 0.2054605782032013, + 0.21280795335769653, + 0.2033655345439911, + 0.1930479109287262, + 0.1853180229663849 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093814373016357, + "mean_weights": [ + 0.2011115550994873, + 0.20251870155334473, + 0.19948235154151917, + 0.1983872652053833, + 0.1985001266002655 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6076922416687012, + "mean_weights": [ + 0.20588652789592743, + 0.2154477834701538, + 0.20285755395889282, + 0.19265122711658478, + 0.18315690755844116 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093541383743286, + "mean_weights": [ + 0.2006198763847351, + 0.2033575475215912, + 0.20118525624275208, + 0.19840578734874725, + 0.19643151760101318 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7901891469955444, + "mean_weights": [ + 0.17122846841812134, + 0.1810305416584015, + 0.17142155766487122, + 0.1620669662952423, + 0.15437543392181396, + 0.1598770171403885 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7916991710662842, + "mean_weights": [ + 0.16690796613693237, + 0.16926559805870056, + 0.16834613680839539, + 0.16567304730415344, + 0.16515053808689117, + 0.16465671360492706 + ], + "sources": 6 + }, + { + "entropy_mean": 1.789719581604004, + "mean_weights": [ + 0.17202284932136536, + 0.18247804045677185, + 0.17337769269943237, + 0.16244632005691528, + 0.15673619508743286, + 0.15293890237808228 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917168140411377, + "mean_weights": [ + 0.1664523184299469, + 0.16856399178504944, + 0.1666605919599533, + 0.16678282618522644, + 0.16458262503147125, + 0.16695764660835266 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7894618511199951, + "mean_weights": [ + 0.17301465570926666, + 0.18332527577877045, + 0.17253267765045166, + 0.1639079749584198, + 0.1556718945503235, + 0.15154753625392914 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917406558990479, + "mean_weights": [ + 0.16621100902557373, + 0.16746005415916443, + 0.16678889095783234, + 0.1668531894683838, + 0.16529077291488647, + 0.16739608347415924 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7883973121643066, + "mean_weights": [ + 0.17285701632499695, + 0.18654102087020874, + 0.17581810057163239, + 0.16295000910758972, + 0.15416306257247925, + 0.14767077565193176 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.945841670036316, + "mean_weights": [ + 0.1422179937362671, + 0.14141613245010376, + 0.1419726312160492, + 0.14163202047348022, + 0.1428576409816742, + 0.14370085299015045, + 0.14620274305343628 + ], + "sources": 7 + }, + { + "entropy_mean": 2.074218988418579, + "mean_weights": [ + 0.13300631940364838, + 0.14625868201255798, + 0.1377277672290802, + 0.12726855278015137, + 0.11886662989854813, + 0.1138627678155899, + 0.10946840047836304, + 0.113540880382061 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0794034004211426, + "mean_weights": [ + 0.12426760792732239, + 0.12337200343608856, + 0.12434753775596619, + 0.12425623089075089, + 0.12553738057613373, + 0.12575145065784454, + 0.12636850774288177, + 0.12609928846359253 + ], + "sources": 8 + }, + { + "entropy_mean": 2.071866989135742, + "mean_weights": [ + 0.13419705629348755, + 0.14923259615898132, + 0.1412179321050644, + 0.12855416536331177, + 0.11907775700092316, + 0.11311224102973938, + 0.10782244801521301, + 0.10678580403327942 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079402208328247, + "mean_weights": [ + 0.12453928589820862, + 0.12370485067367554, + 0.12388431280851364, + 0.12455643713474274, + 0.12515997886657715, + 0.1251543164253235, + 0.12670117616653442, + 0.126299649477005 + ], + "sources": 8 + }, + { + "entropy_mean": 2.073277473449707, + "mean_weights": [ + 0.13280916213989258, + 0.147087961435318, + 0.13949216902256012, + 0.12860605120658875, + 0.12044619768857956, + 0.11539597809314728, + 0.10893120616674423, + 0.10723128914833069 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079406261444092, + "mean_weights": [ + 0.12449289858341217, + 0.12421604245901108, + 0.12407072633504868, + 0.12465694546699524, + 0.1250179409980774, + 0.12552028894424438, + 0.12690746784210205, + 0.12511767446994781 + ], + "sources": 8 + }, + { + "entropy_mean": 2.072551965713501, + "mean_weights": [ + 0.13300633430480957, + 0.1484868824481964, + 0.14011326432228088, + 0.12896595895290375, + 0.11925266683101654, + 0.11479288339614868, + 0.10988710075616837, + 0.1054949015378952 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079289197921753, + "mean_weights": [ + 0.12425784021615982, + 0.12249879539012909, + 0.12262538075447083, + 0.1231345534324646, + 0.12554849684238434, + 0.12622186541557312, + 0.12740382552146912, + 0.1283092498779297 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1905746459960938, + "mean_weights": [ + 0.11894530802965164, + 0.13222399353981018, + 0.1255521923303604, + 0.11697500199079514, + 0.1087704747915268, + 0.10384097695350647, + 0.09815281629562378, + 0.0949372723698616, + 0.10060198605060577 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1971940994262695, + "mean_weights": [ + 0.11043104529380798, + 0.11006320267915726, + 0.1108100414276123, + 0.1104082316160202, + 0.11129562556743622, + 0.11133190989494324, + 0.11100216209888458, + 0.1123153492808342, + 0.11234242469072342 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1903302669525146, + "mean_weights": [ + 0.11926354467868805, + 0.13217687606811523, + 0.12629367411136627, + 0.11622180044651031, + 0.11075383424758911, + 0.10492376983165741, + 0.0998297780752182, + 0.09527494013309479, + 0.09526178240776062 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1971096992492676, + "mean_weights": [ + 0.11036054790019989, + 0.10916533321142197, + 0.10960874706506729, + 0.10944295674562454, + 0.11106343567371368, + 0.11172300577163696, + 0.11184054613113403, + 0.1131286472082138, + 0.11366678774356842 + ], + "sources": 9 + }, + { + "entropy_mean": 2.189469337463379, + "mean_weights": [ + 0.1198464184999466, + 0.1340266764163971, + 0.12665340304374695, + 0.11636817455291748, + 0.10994137823581696, + 0.10405561327934265, + 0.09988697618246078, + 0.09452254325151443, + 0.09469881653785706 + ], + "sources": 9 + }, + { + "entropy_mean": 2.196969509124756, + "mean_weights": [ + 0.10976777970790863, + 0.10751202702522278, + 0.10838233679533005, + 0.10999103635549545, + 0.1104883924126625, + 0.11148744821548462, + 0.1139722689986229, + 0.11430276185274124, + 0.11409594118595123 + ], + "sources": 9 + }, + { + "entropy_mean": 2.190587282180786, + "mean_weights": [ + 0.11944106221199036, + 0.13183563947677612, + 0.12491786479949951, + 0.1163654774427414, + 0.11068646609783173, + 0.10502398014068604, + 0.10101237148046494, + 0.0961514264345169, + 0.09456571191549301 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028569862246513367, + 0.01370697095990181, + 0.015033197589218616, + 0.014952084980905056, + 0.015676941722631454, + 0.015785424038767815, + 0.016541553661227226, + 0.016757339239120483, + 0.01775228977203369, + 0.012117800302803516, + 0.012944038026034832, + 0.01309970673173666, + 0.014040122739970684, + 0.014189697802066803, + 0.015270783565938473, + 0.015521221794188023, + 0.016725214198231697, + 0.012799641117453575, + 0.013756495900452137, + 0.013930512592196465, + 0.01502426527440548, + 0.015204393304884434, + 0.016265366226434708, + 0.016438564285635948, + 0.017614256590604782, + 0.014356615021824837, + 0.015256335958838463, + 0.015486872754991055, + 0.016392884775996208, + 0.016713088378310204, + 0.017736028879880905, + 0.017855215817689896, + 0.019016770645976067, + 0.016060268506407738, + 0.01702474057674408, + 0.017304476350545883, + 0.01832628808915615, + 0.01854640245437622, + 0.019656823948025703, + 0.019735300913453102, + 0.021159222349524498, + 0.018558871001005173, + 0.019339462742209435, + 0.019886428490281105, + 0.020691141486167908, + 0.021280329674482346, + 0.02208683453500271, + 0.022898191586136818, + 0.02391328290104866, + 0.02066885121166706, + 0.022248582914471626, + 0.021940546110272408, + 0.02382468245923519, + 0.023327330127358437, + 0.025332899764180183, + 0.02472243271768093, + 0.027272451668977737, + 0.023586342111229897, + 0.025675876066088676, + 0.025116410106420517, + 0.027488600462675095, + 0.026563847437500954, + 0.02940056100487709, + 0.028492337092757225 + ], + "loss_nats": 3.547853469848633, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.193621873855591, + "mean_weights": [ + 0.10075727105140686, + 0.09505389630794525, + 0.10224547982215881, + 0.11049330979585648, + 0.11544385552406311, + 0.11876599490642548, + 0.1197839081287384, + 0.119516521692276, + 0.11793976277112961 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.0030840476974844933, + 0.005867472849786282, + 0.0066835456527769566, + 0.009899183176457882, + 0.01063398364931345, + 0.014523761346936226, + 0.014861593954265118, + 0.019194046035408974, + 0.003107215277850628, + 0.0068074981682002544, + 0.007714417297393084, + 0.012258335016667843, + 0.012939994223415852, + 0.01779257506132126, + 0.018522104248404503, + 0.02363724075257778, + 0.004710398614406586, + 0.008278015069663525, + 0.009563729166984558, + 0.014763166196644306, + 0.01629871316254139, + 0.021510249003767967, + 0.023043211549520493, + 0.0282937902957201, + 0.0051618111319839954, + 0.009244468063116074, + 0.011889521032571793, + 0.01668422669172287, + 0.019782964140176773, + 0.025010215118527412, + 0.02735232748091221, + 0.03293043002486229, + 0.006114646792411804, + 0.010345770977437496, + 0.014176057651638985, + 0.019711239263415337, + 0.02387424372136593, + 0.02933182567358017, + 0.033970363438129425, + 0.039894845336675644, + 0.008909563533961773, + 0.013393603265285492, + 0.019036034122109413, + 0.024709276854991913, + 0.030538424849510193, + 0.03661581501364708, + 0.0440770722925663, + 0.05052796006202698, + 0.010605492629110813, + 0.015496755950152874, + 0.023913949728012085, + 0.030097337439656258, + 0.038264647126197815, + 0.04464881867170334, + 0.05440589785575867, + 0.06112745404243469, + 0.013455186039209366, + 0.019440028816461563, + 0.030407289043068886, + 0.03749694302678108, + 0.049828581511974335, + 0.05591440200805664, + 0.07261542230844498, + 0.07976537942886353 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00015765291755087674, + 0.00016633255290798843, + 0.0001665915478952229, + 0.00015825126320123672, + 0.00015264090325217694, + 0.00013679999392479658, + 0.00012187181710032746, + 0.00011186554183950648, + 0.00010154431947739795, + 9.19416343094781e-05, + 8.304396033054218e-05, + 7.604345591971651e-05, + 7.339986768784001e-05, + 6.749473686795682e-05, + 6.202288204804063e-05, + 5.7042274420382455e-05, + 5.401543603511527e-05, + 4.9763970309868455e-05, + 4.582337351166643e-05, + 4.208743848721497e-05, + 4.056218676851131e-05, + 3.7538517062785104e-05, + 3.4402542951283976e-05, + 3.2338921300834045e-05, + 3.0097604394541122e-05, + 2.904982102336362e-05, + 2.8162437956780195e-05, + 2.7535101253306493e-05, + 2.297476567036938e-05, + 2.2671043552691117e-05, + 2.2578438802156597e-05, + 2.2612201064475812e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00014650081720901653, + "first_to_last_ratio": 5.698164537515351, + "imbalance_abs_log_ratio": 1.7401441120216228, + "last_quartile_mean": 2.5710176714710542e-05, + "mean": 7.271104590245159e-05, + "normalized": [ + 2.16821138513648, + 2.2875830053543518, + 2.291144981172743, + 2.1764404738936785, + 2.0992808088190404, + 1.8814196966486507, + 1.6761114571757016, + 1.5384944673961116, + 1.3965459885369933, + 1.26447960097873, + 1.1421092806442796, + 1.0458308634665447, + 1.0094734132460774, + 0.9282597441729429, + 0.8530049496365367, + 0.7845063114194479, + 0.7428779955604248, + 0.6844072959235286, + 0.6302119979561649, + 0.5788314273965891, + 0.5578545359246948, + 0.516269799132671, + 0.47314053214745505, + 0.44475940208891535, + 0.4139344169924301, + 0.39952418044345794, + 0.38731994028201217, + 0.37869213558346154, + 0.3159735276149389, + 0.31179641650467194, + 0.3105228170207812, + 0.3109871517294926 + ], + "population_cv": 0.6609918675171289 + }, + "activation_output_rms_by_block": [ + 0.021120406687259674, + 0.038012173026800156, + 0.04921139031648636, + 0.05904547870159149, + 0.02449391968548298, + 0.026677677407860756, + 0.03334493562579155, + 0.039425674825906754, + 0.03169336915016174, + 0.03166627138853073, + 0.036849819123744965, + 0.04246170446276665, + 0.023929333314299583, + 0.03132571280002594, + 0.03933583199977875, + 0.04560377076268196, + 0.029249923303723335, + 0.04593157768249512, + 0.0467093288898468, + 0.06614020466804504, + 0.03474031388759613, + 0.043857965618371964, + 0.050220053642988205, + 0.065526083111763, + 0.02748512476682663, + 0.054549459367990494, + 0.08253413438796997, + 0.11049347370862961, + 0.02669628895819187, + 0.05261136591434479, + 0.08427217602729797, + 0.11668161302804947 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.036416457034647465, + "first_to_last_ratio": 0.5246159848193603, + "imbalance_abs_log_ratio": 0.6450887415229958, + "last_quartile_mean": 0.0694154545199126, + "mean": 0.04724676738260314, + "normalized": [ + 0.44702331730395756, + 0.8045454775557136, + 1.0415821662035363, + 1.2497252610626814, + 0.5184253027753588, + 0.5646455595961856, + 0.7057612080793824, + 0.8344629063537539, + 0.6708050287019561, + 0.670231491862672, + 0.7799437118170658, + 0.8987218981335343, + 0.5064755673233777, + 0.6630234095456966, + 0.8325613407841888, + 0.965225205639653, + 0.6190883508888172, + 0.9721633929056427, + 0.9886248621328064, + 1.3998884650127135, + 0.7352950437067988, + 0.9282744206225001, + 1.0629309987772808, + 1.3868902941260388, + 0.581735561805801, + 1.1545649023191435, + 1.7468736796236368, + 2.338646214964428, + 0.5650394817915476, + 1.1135442450125586, + 1.7836601464151822, + 2.4696210871563906 + ], + "population_cv": 0.4894210708186825 + }, + "bits_per_byte": 3.078995767368415, + "branch_output_rms_by_sublayer": [ + 0.01104653999209404, + 0.01683947630226612, + 0.01422838307917118, + 0.014211748726665974, + 0.016164524480700493, + 0.011905338615179062, + 0.018276382237672806, + 0.010613416321575642, + 0.022278686985373497, + 0.008512545377016068, + 0.01990687847137451, + 0.008251024410128593, + 0.014404041692614555, + 0.008535181172192097, + 0.014417328871786594, + 0.008772693574428558, + 0.029408056288957596, + 0.00884132832288742, + 0.027921510860323906, + 0.008686050772666931, + 0.015500270761549473, + 0.00909200869500637, + 0.015328317880630493, + 0.009221813641488552, + 0.021008778363466263, + 0.009694541804492474, + 0.018537800759077072, + 0.009053537622094154, + 0.021732496097683907, + 0.009831193834543228, + 0.024701334536075592, + 0.010189603082835674, + 0.02535322494804859, + 0.012025527656078339, + 0.042191602289676666, + 0.012406121008098125, + 0.026748627424240112, + 0.013192851096391678, + 0.037852395325899124, + 0.015181704424321651, + 0.031291887164115906, + 0.014347006566822529, + 0.014219598844647408, + 0.01539402361959219, + 0.022828778252005577, + 0.017770104110240936, + 0.010967823676764965, + 0.018301088362932205, + 0.012939315289258957, + 0.021906793117523193, + 0.01274110283702612, + 0.02497999183833599, + 0.013105918653309345, + 0.025628667324781418, + 0.012536930851638317, + 0.026799935847520828, + 0.006649280898272991, + 0.02590365707874298, + 0.005377242341637611, + 0.026600118726491928, + 0.009160320274531841, + 0.02971830777823925, + 0.006215699482709169, + 0.03135161101818085 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0005356545959707064, + 0.0005845033376345634, + 0.0005432581639947152, + 0.0005433358955984542, + 0.0005717638625891249, + 0.00045042492938920865, + 0.00036885726160131873, + 0.0003273341802234718, + 0.0003849458245536528, + 0.0002985897391080209, + 0.00023870049037499802, + 0.00022016573334702983, + 0.0002341392297148409, + 0.0001939450024928393, + 0.00019565091702592304, + 0.00019954327464054465, + 0.00021073146798841852, + 0.00021544756758684998, + 0.00017157267551424354, + 0.00020562784353885224, + 0.00017153028226678455, + 0.00013823344918438045, + 0.00014444779351893358, + 0.00011198207677485065, + 0.00011418116185736458, + 0.00011525029156024048, + 0.0001121035660944868, + 0.00011003117119016207, + 8.205945174933982e-05, + 8.081753174604079e-05, + 8.76155900923687e-05, + 8.505927729227282e-05 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0004906415283751954, + "first_to_last_ratio": 4.986713579974873, + "imbalance_abs_log_ratio": 1.606777091582941, + "last_quartile_mean": 9.838975519778451e-05, + "mean": 0.0002514844886317188, + "normalized": [ + 2.1299707146357427, + 2.324212283686916, + 2.160205454222977, + 2.1605145452693546, + 2.2735551830651173, + 1.7910644582490494, + 1.4667197313369256, + 1.301607832771068, + 1.530694106217337, + 1.1873087709408767, + 0.9491658577979255, + 0.8754644652038438, + 0.9310285138807156, + 0.7712006555476191, + 0.7779840342854701, + 0.793461559900665, + 0.8379501619959544, + 0.8567032056691085, + 0.6822395943691762, + 0.8176561690052369, + 0.6820710223523109, + 0.5496698819735699, + 0.5743805286156918, + 0.4452842295925474, + 0.4540286459757556, + 0.45827992091001823, + 0.44576731831224203, + 0.437526671282279, + 0.3263002509451391, + 0.3213618946669603, + 0.34839361492658705, + 0.33822872239582175 + ], + "population_cv": 0.6307097628496828 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6556750535964966, + "mean_weights": [ + 0.38573914766311646, + 0.6142609119415283 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6756486892700195, + "mean_weights": [ + 0.44063055515289307, + 0.5593694448471069 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6893900632858276, + "mean_weights": [ + 0.5004085898399353, + 0.4995914101600647 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6682843565940857, + "mean_weights": [ + 0.4320886433124542, + 0.5679113864898682 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6887181401252747, + "mean_weights": [ + 0.49143603444099426, + 0.5085639953613281 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6821175813674927, + "mean_weights": [ + 0.48318418860435486, + 0.5168157815933228 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6806358098983765, + "mean_weights": [ + 0.44999855756759644, + 0.5500014424324036 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6727093458175659, + "mean_weights": [ + 0.4386836290359497, + 0.5613163709640503 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0913090705871582, + "mean_weights": [ + 0.3023547828197479, + 0.3372229337692261, + 0.3604222536087036 + ], + "sources": 3 + }, + { + "entropy_mean": 1.086233377456665, + "mean_weights": [ + 0.29878678917884827, + 0.32076382637023926, + 0.38044941425323486 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0796632766723633, + "mean_weights": [ + 0.2683682143688202, + 0.3598070740699768, + 0.3718246817588806 + ], + "sources": 3 + }, + { + "entropy_mean": 1.077853798866272, + "mean_weights": [ + 0.2858407497406006, + 0.3118148446083069, + 0.40234440565109253 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0844547748565674, + "mean_weights": [ + 0.28308427333831787, + 0.3325739800930023, + 0.3843417167663574 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0643466711044312, + "mean_weights": [ + 0.2924281358718872, + 0.2851218283176422, + 0.4224500060081482 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0809199810028076, + "mean_weights": [ + 0.27812594175338745, + 0.342665433883667, + 0.37920862436294556 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0571953058242798, + "mean_weights": [ + 0.28698650002479553, + 0.2881914973258972, + 0.42482197284698486 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3821542263031006, + "mean_weights": [ + 0.23639552295207977, + 0.23288217186927795, + 0.267272412776947, + 0.26344990730285645 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3573906421661377, + "mean_weights": [ + 0.24117499589920044, + 0.21879665553569794, + 0.25600990653038025, + 0.28401845693588257 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3826407194137573, + "mean_weights": [ + 0.2427489310503006, + 0.2344237118959427, + 0.24531877040863037, + 0.27750858664512634 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3598005771636963, + "mean_weights": [ + 0.2422315627336502, + 0.22225120663642883, + 0.24271328747272491, + 0.29280394315719604 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3785480260849, + "mean_weights": [ + 0.24458681046962738, + 0.230476975440979, + 0.2454908788204193, + 0.2794453501701355 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3521864414215088, + "mean_weights": [ + 0.24506612122058868, + 0.2232067883014679, + 0.2267303764820099, + 0.3049967288970947 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3819146156311035, + "mean_weights": [ + 0.2574343681335449, + 0.2250547707080841, + 0.2448711097240448, + 0.27263975143432617 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3559112548828125, + "mean_weights": [ + 0.25089216232299805, + 0.22848385572433472, + 0.22462190687656403, + 0.2960020899772644 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6070215702056885, + "mean_weights": [ + 0.20163314044475555, + 0.19226586818695068, + 0.18959230184555054, + 0.20772667229175568, + 0.20878201723098755 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5803847312927246, + "mean_weights": [ + 0.19484558701515198, + 0.1812257170677185, + 0.17531642317771912, + 0.20867419242858887, + 0.23993808031082153 + ], + "sources": 5 + }, + { + "entropy_mean": 1.603074312210083, + "mean_weights": [ + 0.18645665049552917, + 0.195221409201622, + 0.19015321135520935, + 0.19906622171401978, + 0.2291024923324585 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5820807218551636, + "mean_weights": [ + 0.19427818059921265, + 0.1825372576713562, + 0.17709994316101074, + 0.19292914867401123, + 0.2531554698944092 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6022968292236328, + "mean_weights": [ + 0.18803977966308594, + 0.19512943923473358, + 0.1910586953163147, + 0.20167870819568634, + 0.22409336268901825 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5825834274291992, + "mean_weights": [ + 0.20298117399215698, + 0.18301546573638916, + 0.17448818683624268, + 0.19272486865520477, + 0.24679034948349 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5987768173217773, + "mean_weights": [ + 0.17917495965957642, + 0.19803766906261444, + 0.19053050875663757, + 0.19715584814548492, + 0.23510101437568665 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5845110416412354, + "mean_weights": [ + 0.197947159409523, + 0.18003496527671814, + 0.18084511160850525, + 0.18948791921138763, + 0.25168484449386597 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7801141738891602, + "mean_weights": [ + 0.14672595262527466, + 0.17037373781204224, + 0.15641994774341583, + 0.15368908643722534, + 0.1737832874059677, + 0.1990080177783966 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7745113372802734, + "mean_weights": [ + 0.1546814739704132, + 0.15577378869056702, + 0.14963877201080322, + 0.15061523020267487, + 0.17969340085983276, + 0.20959731936454773 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7815184593200684, + "mean_weights": [ + 0.14583583176136017, + 0.1699339747428894, + 0.16043305397033691, + 0.15603342652320862, + 0.1730409860610962, + 0.1947227269411087 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7792775630950928, + "mean_weights": [ + 0.16502341628074646, + 0.1597382128238678, + 0.1500682830810547, + 0.15197905898094177, + 0.1727825105190277, + 0.20040853321552277 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7788512706756592, + "mean_weights": [ + 0.13877840340137482, + 0.18352721631526947, + 0.16435548663139343, + 0.1535210758447647, + 0.16609123349189758, + 0.19372659921646118 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7788422107696533, + "mean_weights": [ + 0.17329949140548706, + 0.16182762384414673, + 0.1533004641532898, + 0.14409664273262024, + 0.16285094618797302, + 0.20462486147880554 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7780876159667969, + "mean_weights": [ + 0.13669076561927795, + 0.19325950741767883, + 0.1745108962059021, + 0.155942901968956, + 0.16215042769908905, + 0.17744547128677368 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.926490306854248, + "mean_weights": [ + 0.14336994290351868, + 0.15386521816253662, + 0.14485357701778412, + 0.13281306624412537, + 0.13362693786621094, + 0.14526863396167755, + 0.14620263874530792 + ], + "sources": 7 + }, + { + "entropy_mean": 2.050598621368408, + "mean_weights": [ + 0.10214991122484207, + 0.17109990119934082, + 0.14453881978988647, + 0.12801259756088257, + 0.1215713620185852, + 0.1180717870593071, + 0.11994239687919617, + 0.094613216817379 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0560684204101562, + "mean_weights": [ + 0.11530975252389908, + 0.15603813529014587, + 0.14282989501953125, + 0.12499736994504929, + 0.11362733691930771, + 0.11264583468437195, + 0.1284077763557434, + 0.10614388436079025 + ], + "sources": 8 + }, + { + "entropy_mean": 2.021805763244629, + "mean_weights": [ + 0.09832236915826797, + 0.19011005759239197, + 0.15889745950698853, + 0.13304349780082703, + 0.1182064563035965, + 0.10651260614395142, + 0.118741974234581, + 0.0761655867099762 + ], + "sources": 8 + }, + { + "entropy_mean": 2.054328203201294, + "mean_weights": [ + 0.11400935053825378, + 0.1576521098613739, + 0.14239449799060822, + 0.12372182309627533, + 0.11994212865829468, + 0.1126628965139389, + 0.12732404470443726, + 0.10229314863681793 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0332984924316406, + "mean_weights": [ + 0.09896409511566162, + 0.1816747784614563, + 0.1569770723581314, + 0.13333658874034882, + 0.11546047031879425, + 0.11191503703594208, + 0.11938562989234924, + 0.08228632807731628 + ], + "sources": 8 + }, + { + "entropy_mean": 2.057521343231201, + "mean_weights": [ + 0.11754059791564941, + 0.15274786949157715, + 0.13966582715511322, + 0.11639910936355591, + 0.11484728753566742, + 0.11592372506856918, + 0.13062642514705658, + 0.11224918067455292 + ], + "sources": 8 + }, + { + "entropy_mean": 2.014674425125122, + "mean_weights": [ + 0.09421488642692566, + 0.19402191042900085, + 0.16799814999103546, + 0.1342812329530716, + 0.11433981359004974, + 0.10257101058959961, + 0.11725098639726639, + 0.07532202452421188 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0641510486602783, + "mean_weights": [ + 0.12189871072769165, + 0.13979385793209076, + 0.13118034601211548, + 0.11444050818681717, + 0.11027108132839203, + 0.11781850457191467, + 0.1353064924478531, + 0.12929052114486694 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1438536643981934, + "mean_weights": [ + 0.0832926407456398, + 0.16492733359336853, + 0.14394685626029968, + 0.1154983788728714, + 0.09942111372947693, + 0.08847770094871521, + 0.10524503886699677, + 0.07023467123508453, + 0.12895625829696655 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1927287578582764, + "mean_weights": [ + 0.10766296833753586, + 0.12428262084722519, + 0.11294365674257278, + 0.10774405300617218, + 0.10894092917442322, + 0.10671180486679077, + 0.11455762386322021, + 0.10718376189470291, + 0.10997258871793747 + ], + "sources": 9 + }, + { + "entropy_mean": 2.132080554962158, + "mean_weights": [ + 0.08590839803218842, + 0.17444255948066711, + 0.15452122688293457, + 0.12361177802085876, + 0.10692349821329117, + 0.09489229321479797, + 0.10832498222589493, + 0.06877826154232025, + 0.08259701728820801 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1687204837799072, + "mean_weights": [ + 0.10786207020282745, + 0.1327856481075287, + 0.12112715095281601, + 0.10758252441883087, + 0.10492686927318573, + 0.10216525197029114, + 0.1163531020283699, + 0.10325929522514343, + 0.10393808782100677 + ], + "sources": 9 + }, + { + "entropy_mean": 2.131237030029297, + "mean_weights": [ + 0.08818842470645905, + 0.17723610997200012, + 0.1545344889163971, + 0.12202068418264389, + 0.10704749822616577, + 0.09184901416301727, + 0.11134066432714462, + 0.07371222972869873, + 0.07407087087631226 + ], + "sources": 9 + }, + { + "entropy_mean": 2.178445339202881, + "mean_weights": [ + 0.10010097920894623, + 0.13788053393363953, + 0.12790179252624512, + 0.12066912651062012, + 0.11775651574134827, + 0.10218970477581024, + 0.11021019518375397, + 0.08918462693691254, + 0.0941065102815628 + ], + "sources": 9 + }, + { + "entropy_mean": 2.134946823120117, + "mean_weights": [ + 0.08934780955314636, + 0.17393918335437775, + 0.1521010547876358, + 0.122739776968956, + 0.10991882532835007, + 0.09505578875541687, + 0.11176487803459167, + 0.07412641495466232, + 0.07100629061460495 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.04395066946744919, + 0.01782958209514618, + 0.02398398518562317, + 0.027281105518341064, + 0.030504858121275902, + 0.03158063441514969, + 0.0345294289290905, + 0.036064065992832184, + 0.03914797306060791, + 0.02573774941265583, + 0.02693457528948784, + 0.0278943944722414, + 0.028038110584020615, + 0.028648704290390015, + 0.029745543375611305, + 0.03064391389489174, + 0.0316680446267128, + 0.023377127945423126, + 0.025463232770562172, + 0.02357678860425949, + 0.025237303227186203, + 0.024294763803482056, + 0.026284173130989075, + 0.02487398311495781, + 0.027305712923407555, + 0.02059704251587391, + 0.021993694826960564, + 0.02087780088186264, + 0.022182976827025414, + 0.021769104525446892, + 0.023165101185441017, + 0.02258509211242199, + 0.02381843514740467, + 0.01899546943604946, + 0.019910024479031563, + 0.020049091428518295, + 0.020845653489232063, + 0.020806532353162766, + 0.02142876759171486, + 0.021993650123476982, + 0.022452883422374725, + 0.019026977941393852, + 0.019900770857930183, + 0.019746651872992516, + 0.020814836025238037, + 0.020726781338453293, + 0.022063175216317177, + 0.022047264501452446, + 0.023606114089488983, + 0.021263254806399345, + 0.02233981527388096, + 0.02242393232882023, + 0.024128029122948647, + 0.023844225332140923, + 0.02686251513659954, + 0.02480592206120491, + 0.030858300626277924, + 0.022977767512202263, + 0.028576472774147987, + 0.025326652452349663, + 0.030559083446860313, + 0.027281664311885834, + 0.030932744964957237, + 0.02894538640975952 + ], + "loss_nats": 2.134197235107422, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.173508644104004, + "mean_weights": [ + 0.12063480168581009, + 0.08908510953187943, + 0.09895257651805878, + 0.09448663890361786, + 0.09707897901535034, + 0.10958927124738693, + 0.12360340356826782, + 0.1459284871816635, + 0.12064072489738464 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.01104653999209404, + 0.021120406687259674, + 0.028624296188354492, + 0.038012173026800156, + 0.04175331071019173, + 0.04921139031648636, + 0.05247887223958969, + 0.05904547870159149, + 0.022278686985373497, + 0.02449391968548298, + 0.0233079232275486, + 0.026677677407860756, + 0.02912965975701809, + 0.03334493562579155, + 0.03473967686295509, + 0.039425674825906754, + 0.029408056288957596, + 0.03169336915016174, + 0.028289956972002983, + 0.03166627138853073, + 0.032567042857408524, + 0.036849819123744965, + 0.03772272914648056, + 0.04246170446276665, + 0.021008778363466263, + 0.023929333314299583, + 0.027481559664011, + 0.03132571280002594, + 0.034497153013944626, + 0.03933583199977875, + 0.040259890258312225, + 0.04560377076268196, + 0.02535322494804859, + 0.029249923303723335, + 0.04067646712064743, + 0.04593157768249512, + 0.03940827399492264, + 0.0467093288898468, + 0.05872092396020889, + 0.06614020466804504, + 0.031291887164115906, + 0.03474031388759613, + 0.03670363873243332, + 0.043857965618371964, + 0.03770771250128746, + 0.050220053642988205, + 0.05158250778913498, + 0.065526083111763, + 0.012939315289258957, + 0.02748512476682663, + 0.03386048600077629, + 0.054549459367990494, + 0.06028575077652931, + 0.08253413438796997, + 0.08655966073274612, + 0.11049347370862961, + 0.006649280898272991, + 0.02669628895819187, + 0.02823398821055889, + 0.05261136591434479, + 0.05658188462257385, + 0.08427217602729797, + 0.08723503351211548, + 0.11668161302804947 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.705685129621997e-05, + 8.402342791669071e-05, + 7.824215572327375e-05, + 7.493723387597129e-05, + 8.14162558526732e-05, + 7.776702841510996e-05, + 7.247507892316207e-05, + 6.979653699090704e-05, + 7.473726873286068e-05, + 7.097745401551947e-05, + 6.635783211095259e-05, + 6.161812780192122e-05, + 5.7675006246427074e-05, + 5.520996273844503e-05, + 5.349946513888426e-05, + 5.363513992051594e-05, + 6.706128624500707e-05, + 7.242875290103257e-05, + 6.669403956038877e-05, + 5.7040786487050354e-05, + 6.76524214213714e-05, + 6.425228639272973e-05, + 6.026056507835165e-05, + 5.773886732640676e-05, + 8.325560338562354e-05, + 6.863444286864251e-05, + 5.239458187134005e-05, + 4.488468402996659e-05, + 2.9393931981758215e-05, + 2.6196124963462353e-05, + 2.3977429009391926e-05, + 2.3381806386169046e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 7.6964321124251e-05, + "first_to_last_ratio": 1.7485999351686712, + "imbalance_abs_log_ratio": 0.5588154306885184, + "last_quartile_mean": 4.401482556204428e-05, + "mean": 6.170851361275709e-05, + "normalized": [ + 1.2487231791026303, + 1.3616180814849579, + 1.2679312973615142, + 1.214374313830165, + 1.3193682862560783, + 1.2602317551047457, + 1.1744745527006049, + 1.1310681931006341, + 1.2111338348200003, + 1.1502052125404978, + 1.0753432261773737, + 0.9985352781076032, + 0.9346361283040828, + 0.8946895575042889, + 0.8669705686740807, + 0.869169208273198, + 1.0867428547354185, + 1.1737238293495253, + 1.0807915416488174, + 0.9243584579755334, + 1.096322329944851, + 1.0412223959231253, + 0.9765356763656335, + 0.9356710111142642, + 1.3491753165224838, + 1.1122362029224702, + 0.8490656929469201, + 0.7273661509922922, + 0.4763351158677328, + 0.42451395163805716, + 0.38855949699029924, + 0.37890730172011944 + ], + "population_cv": 0.26885057128159573 + }, + "activation_output_rms_by_block": [ + 0.08078505098819733, + 0.16996513307094574, + 0.196508526802063, + 0.2064388245344162, + 0.07466654479503632, + 0.08950652927160263, + 0.09770005196332932, + 0.10525599867105484, + 0.03102109581232071, + 0.047966018319129944, + 0.06952836364507675, + 0.09958801418542862, + 0.04605082795023918, + 0.07880882918834686, + 0.10951601713895798, + 0.1483204960823059, + 0.041445259004831314, + 0.06898269802331924, + 0.09753179550170898, + 0.14137907326221466, + 0.060624755918979645, + 0.08929698914289474, + 0.13302429020404816, + 0.1656738966703415, + 0.050240688025951385, + 0.07620147615671158, + 0.10883574187755585, + 0.15144884586334229, + 0.06723883002996445, + 0.12474417686462402, + 0.19924305379390717, + 0.30584004521369934 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.12760333251208067, + "first_to_last_ratio": 0.9419019997461231, + "imbalance_abs_log_ratio": 0.05985404405749527, + "last_quartile_mean": 0.1354741072282195, + "mean": 0.11041806056164205, + "normalized": [ + 0.7316289615782394, + 1.5392874336536722, + 1.7796773988106775, + 1.8696110354081934, + 0.6762167748214788, + 0.8106149375955998, + 0.8848194893695605, + 0.9532498409741091, + 0.2809422267927162, + 0.4344037386198466, + 0.6296828914710181, + 0.9019177993063241, + 0.4170588372590625, + 0.7137313297071632, + 0.9918306532636434, + 1.3432630072279201, + 0.3753485507173354, + 0.6247410765271404, + 0.8832956765008639, + 1.2803980847253542, + 0.5490474620783186, + 0.8087172396316792, + 1.204733080144855, + 1.5004238964861394, + 0.45500426081025025, + 0.6901178644970973, + 0.9856697475391459, + 1.3715948739997508, + 0.6089477544520686, + 1.1297443210839975, + 1.8044426136490381, + 2.76983714129774 + ], + "population_cv": 0.5315829537491391 + }, + "bits_per_byte": 2.074907221061618, + "branch_output_rms_by_sublayer": [ + 0.04378553107380867, + 0.06483227014541626, + 0.1036754623055458, + 0.026424840092658997, + 0.038812410086393356, + 0.019707385450601578, + 0.024522552266716957, + 0.014634309336543083, + 0.074164479970932, + 0.005159751977771521, + 0.026272347196936607, + 0.009668966755270958, + 0.040062956511974335, + 0.013387647457420826, + 0.0229177363216877, + 0.015108250081539154, + 0.026560308411717415, + 0.01570763997733593, + 0.028497489169239998, + 0.017616942524909973, + 0.030835391953587532, + 0.020325668156147003, + 0.03228187561035156, + 0.02830914594233036, + 0.03479894623160362, + 0.02550320513546467, + 0.02672903798520565, + 0.027497688308358192, + 0.022892287001013756, + 0.030179399996995926, + 0.0405505895614624, + 0.03085489757359028, + 0.030135396867990494, + 0.02526100166141987, + 0.03402213752269745, + 0.021919436752796173, + 0.05164136737585068, + 0.023420462384819984, + 0.07183743268251419, + 0.042585037648677826, + 0.052755579352378845, + 0.028536641970276833, + 0.04333524778485298, + 0.03185921162366867, + 0.05283308029174805, + 0.04655485972762108, + 0.03495188429951668, + 0.05507093667984009, + 0.021918848156929016, + 0.04290303587913513, + 0.013743500225245953, + 0.040707845240831375, + 0.018669594079256058, + 0.04862925782799721, + 0.019438164308667183, + 0.05885249376296997, + 0.022322971373796463, + 0.06281275302171707, + 0.012408392503857613, + 0.06911066919565201, + 0.01825278252363205, + 0.08749783784151077, + 0.011333034373819828, + 0.12079072743654251 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00027970557238011725, + 0.0002142346638570605, + 0.00015020321260852002, + 0.0001256556783039675, + 0.00016651488453778068, + 0.00010840748528467976, + 0.00012786362768092588, + 0.00012393150655672962, + 0.0001276624792167679, + 0.0001309456680611586, + 0.00015441355006647235, + 0.00017555840097308522, + 0.00013421440339207993, + 0.0001389660706597323, + 0.0001394254013785331, + 0.00014686826488177038, + 0.00017959890592673047, + 0.0002036206151602009, + 0.0002673797557556375, + 0.0003240353660220287, + 0.00031365049972526644, + 0.00028713925115907797, + 0.0003256481078663223, + 0.0003032381636723614, + 0.00036731340763422387, + 0.00029242469794656, + 0.0002552966777958181, + 0.00022296799786489577, + 0.00014127114709573763, + 0.00013035709887149207, + 0.00012244822157113814, + 0.0001226218325015198 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00016206457890122265, + "first_to_last_ratio": 0.7835352535128403, + "imbalance_abs_log_ratio": 0.2439392232872106, + "last_quartile_mean": 0.00020683763516017318, + "mean": 0.00019698695676276224, + "normalized": [ + 1.4199192524049993, + 1.0875576097917499, + 0.7625033407131346, + 0.6378883169168338, + 0.845309188355653, + 0.5503282403374473, + 0.6490969238888044, + 0.629135596556194, + 0.6480757980870578, + 0.6647428348205852, + 0.7838770272106776, + 0.8912184027723007, + 0.6813364986074619, + 0.7054582341057922, + 0.7077900165057508, + 0.7455735511394725, + 0.9117299382569133, + 1.0336756225206716, + 1.3573475124936905, + 1.6449584853086234, + 1.5922399375051308, + 1.4576561609857706, + 1.6531455341914383, + 1.539381940082261, + 1.8646585219172216, + 1.4844876166026388, + 1.2960080301320667, + 1.1318921898642422, + 0.717159904479742, + 0.6617549761352237, + 0.6216057325998822, + 0.6224870647105698 + ], + "population_cv": 0.39088664041570675 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.44606006145477295, + "mean_weights": [ + 0.826358437538147, + 0.17364157736301422 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6494801640510559, + "mean_weights": [ + 0.5386382937431335, + 0.46136170625686646 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5072237849235535, + "mean_weights": [ + 0.7891030311584473, + 0.21089698374271393 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6888259053230286, + "mean_weights": [ + 0.48377636075019836, + 0.516223669052124 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5385948419570923, + "mean_weights": [ + 0.7636419534683228, + 0.23635801672935486 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6874731779098511, + "mean_weights": [ + 0.5392146110534668, + 0.4607853889465332 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6250219941139221, + "mean_weights": [ + 0.658421516418457, + 0.34157851338386536 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6778561472892761, + "mean_weights": [ + 0.5009177923202515, + 0.49908214807510376 + ], + "sources": 2 + }, + { + "entropy_mean": 0.9856091737747192, + "mean_weights": [ + 0.2565188407897949, + 0.21247120201587677, + 0.5310099124908447 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0935600996017456, + "mean_weights": [ + 0.362551212310791, + 0.3228307366371155, + 0.3146181106567383 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0267367362976074, + "mean_weights": [ + 0.2532237768173218, + 0.28825676441192627, + 0.45851945877075195 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0889358520507812, + "mean_weights": [ + 0.37500596046447754, + 0.27442610263824463, + 0.35056793689727783 + ], + "sources": 3 + }, + { + "entropy_mean": 1.048783302307129, + "mean_weights": [ + 0.36981725692749023, + 0.23526868224143982, + 0.39491406083106995 + ], + "sources": 3 + }, + { + "entropy_mean": 1.083635687828064, + "mean_weights": [ + 0.3503548502922058, + 0.2639753818511963, + 0.3856697678565979 + ], + "sources": 3 + }, + { + "entropy_mean": 1.03965425491333, + "mean_weights": [ + 0.3518397808074951, + 0.24306148290634155, + 0.40509873628616333 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0872478485107422, + "mean_weights": [ + 0.38870689272880554, + 0.2820753753185272, + 0.32921773195266724 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3598366975784302, + "mean_weights": [ + 0.2696279287338257, + 0.17194676399230957, + 0.25822848081588745, + 0.30019688606262207 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3792282342910767, + "mean_weights": [ + 0.2700187563896179, + 0.23244406282901764, + 0.22507816553115845, + 0.2724589705467224 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3468334674835205, + "mean_weights": [ + 0.2877344489097595, + 0.17893925309181213, + 0.20925769209861755, + 0.3240686357021332 + ], + "sources": 4 + }, + { + "entropy_mean": 1.369699239730835, + "mean_weights": [ + 0.29393064975738525, + 0.2159734070301056, + 0.20193371176719666, + 0.2881622314453125 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3446769714355469, + "mean_weights": [ + 0.30399465560913086, + 0.18959194421768188, + 0.1933445930480957, + 0.31306880712509155 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3735146522521973, + "mean_weights": [ + 0.27098315954208374, + 0.21050837635993958, + 0.22181057929992676, + 0.2966978847980499 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3240225315093994, + "mean_weights": [ + 0.35080486536026, + 0.172488272190094, + 0.17881089448928833, + 0.2978959381580353 + ], + "sources": 4 + }, + { + "entropy_mean": 1.377153992652893, + "mean_weights": [ + 0.2878970503807068, + 0.2405650019645691, + 0.21319051086902618, + 0.25834745168685913 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5993653535842896, + "mean_weights": [ + 0.22905559837818146, + 0.16991935670375824, + 0.17330366373062134, + 0.21185429394245148, + 0.2158670723438263 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6007554531097412, + "mean_weights": [ + 0.21910503506660461, + 0.19737505912780762, + 0.16681483387947083, + 0.19314531981945038, + 0.22355973720550537 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5868866443634033, + "mean_weights": [ + 0.19438114762306213, + 0.1601680964231491, + 0.16636969149112701, + 0.23071441054344177, + 0.24836665391921997 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5958073139190674, + "mean_weights": [ + 0.20357678830623627, + 0.1843593567609787, + 0.16133233904838562, + 0.19679602980613708, + 0.25393548607826233 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5782508850097656, + "mean_weights": [ + 0.1877414584159851, + 0.15772804617881775, + 0.15999537706375122, + 0.23602920770645142, + 0.2585059106349945 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5815508365631104, + "mean_weights": [ + 0.19985726475715637, + 0.18821944296360016, + 0.1518726646900177, + 0.18143925070762634, + 0.27861136198043823 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5813524723052979, + "mean_weights": [ + 0.1692967414855957, + 0.17348387837409973, + 0.18078652024269104, + 0.22814308106899261, + 0.2482897937297821 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5857596397399902, + "mean_weights": [ + 0.20381811261177063, + 0.1866980940103531, + 0.1528576761484146, + 0.18745407462120056, + 0.2691720426082611 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7821550369262695, + "mean_weights": [ + 0.15334151685237885, + 0.1545167863368988, + 0.15932831168174744, + 0.18871493637561798, + 0.182026669383049, + 0.1620718240737915 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7694556713104248, + "mean_weights": [ + 0.15681086480617523, + 0.1612417846918106, + 0.12987953424453735, + 0.13358694314956665, + 0.18665696680545807, + 0.23182392120361328 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7623435258865356, + "mean_weights": [ + 0.13764861226081848, + 0.1307733952999115, + 0.14122238755226135, + 0.1800851821899414, + 0.18067581951618195, + 0.22959458827972412 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7788951396942139, + "mean_weights": [ + 0.1631966084241867, + 0.1648799479007721, + 0.136900395154953, + 0.14072662591934204, + 0.18161770701408386, + 0.2126787155866623 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6925528049468994, + "mean_weights": [ + 0.12795597314834595, + 0.1244763657450676, + 0.12078876048326492, + 0.15743941068649292, + 0.13497920334339142, + 0.334360271692276 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7834964990615845, + "mean_weights": [ + 0.16874437034130096, + 0.16465449333190918, + 0.14263096451759338, + 0.14435216784477234, + 0.1813679188489914, + 0.19825008511543274 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6858867406845093, + "mean_weights": [ + 0.13065475225448608, + 0.12962345778942108, + 0.1387796700000763, + 0.1617896407842636, + 0.10867037624120712, + 0.3304820954799652 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9304890632629395, + "mean_weights": [ + 0.15168382227420807, + 0.17430326342582703, + 0.1433834284543991, + 0.12557822465896606, + 0.14954906702041626, + 0.11833585053682327, + 0.1371663212776184 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9888954162597656, + "mean_weights": [ + 0.0894118994474411, + 0.1001531183719635, + 0.09781105816364288, + 0.09313399344682693, + 0.07770239561796188, + 0.12109808623790741, + 0.2211158126592636, + 0.19957363605499268 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0028316974639893, + "mean_weights": [ + 0.10818593204021454, + 0.1385313868522644, + 0.1135009154677391, + 0.10438016057014465, + 0.10318554937839508, + 0.0847199410200119, + 0.11071670055389404, + 0.23677942156791687 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9430785179138184, + "mean_weights": [ + 0.08076326549053192, + 0.09917690604925156, + 0.09752663969993591, + 0.0953429639339447, + 0.07893246412277222, + 0.109538733959198, + 0.1611182540655136, + 0.2776007652282715 + ], + "sources": 8 + }, + { + "entropy_mean": 2.017845392227173, + "mean_weights": [ + 0.12543556094169617, + 0.13131409883499146, + 0.11206898093223572, + 0.10096686333417892, + 0.12375514209270477, + 0.08785608410835266, + 0.09669481217861176, + 0.22190845012664795 + ], + "sources": 8 + }, + { + "entropy_mean": 1.939591884613037, + "mean_weights": [ + 0.0938539057970047, + 0.09975366294384003, + 0.09483259916305542, + 0.0926026776432991, + 0.07918749004602432, + 0.08985430002212524, + 0.15825043618679047, + 0.2916649580001831 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0119400024414062, + "mean_weights": [ + 0.1197943240404129, + 0.11812202632427216, + 0.11788229644298553, + 0.11404071748256683, + 0.10226859152317047, + 0.08651213347911835, + 0.11990596354007721, + 0.22147393226623535 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9896682500839233, + "mean_weights": [ + 0.08792920410633087, + 0.11592824012041092, + 0.1117393895983696, + 0.1149701252579689, + 0.08694544434547424, + 0.10450676083564758, + 0.16563302278518677, + 0.21234780550003052 + ], + "sources": 8 + }, + { + "entropy_mean": 2.037464141845703, + "mean_weights": [ + 0.11910928785800934, + 0.12488348037004471, + 0.11528573930263519, + 0.11451076716184616, + 0.10971179604530334, + 0.09581712633371353, + 0.1339237242937088, + 0.18675807118415833 + ], + "sources": 8 + }, + { + "entropy_mean": 2.118051052093506, + "mean_weights": [ + 0.08720456063747406, + 0.11005577445030212, + 0.09554139524698257, + 0.10255646705627441, + 0.0674656555056572, + 0.08457769453525543, + 0.14587485790252686, + 0.11972570419311523, + 0.1869978904724121 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1537227630615234, + "mean_weights": [ + 0.09310418367385864, + 0.09753461182117462, + 0.09363073110580444, + 0.11028033494949341, + 0.09481626003980637, + 0.09292517602443695, + 0.13477998971939087, + 0.13720324635505676, + 0.14572547376155853 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1486706733703613, + "mean_weights": [ + 0.08586862683296204, + 0.10532578080892563, + 0.09901279211044312, + 0.10676511377096176, + 0.08152840286493301, + 0.09973809123039246, + 0.14624802768230438, + 0.13867199420928955, + 0.13684119284152985 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1462960243225098, + "mean_weights": [ + 0.11163505911827087, + 0.12262783944606781, + 0.12182606011629105, + 0.10256005823612213, + 0.10385803878307343, + 0.08958365023136139, + 0.10037393867969513, + 0.12028154730796814, + 0.12725383043289185 + ], + "sources": 9 + }, + { + "entropy_mean": 2.122485399246216, + "mean_weights": [ + 0.09664355963468552, + 0.12085417658090591, + 0.1058068498969078, + 0.10329756140708923, + 0.08439455181360245, + 0.08982720971107483, + 0.1242167055606842, + 0.1537831425666809, + 0.12117624282836914 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1450929641723633, + "mean_weights": [ + 0.08327501267194748, + 0.10482308268547058, + 0.11693352460861206, + 0.12533506751060486, + 0.08294328302145004, + 0.10945956408977509, + 0.1787133514881134, + 0.10134769976139069, + 0.0971694141626358 + ], + "sources": 9 + }, + { + "entropy_mean": 2.165238857269287, + "mean_weights": [ + 0.096982441842556, + 0.12501254677772522, + 0.11098039150238037, + 0.12363876402378082, + 0.08682079613208771, + 0.09174230694770813, + 0.11498035490512848, + 0.1423938274383545, + 0.10744856297969818 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05602883920073509, + 0.047651853412389755, + 0.05483235791325569, + 0.061409324407577515, + 0.09753062576055527, + 0.06899483501911163, + 0.10207932442426682, + 0.08647006005048752, + 0.11204452067613602, + 0.0756918340921402, + 0.08783775568008423, + 0.08894960582256317, + 0.08321566134691238, + 0.07988443970680237, + 0.08486919850111008, + 0.08313087373971939, + 0.08570585399866104, + 0.056473005563020706, + 0.06609638035297394, + 0.05575483664870262, + 0.06234678998589516, + 0.05763322114944458, + 0.0631181001663208, + 0.05498155578970909, + 0.0678502693772316, + 0.04911094903945923, + 0.05352577939629555, + 0.047900278121232986, + 0.05176490917801857, + 0.05012411251664162, + 0.05616219341754913, + 0.057099416851997375, + 0.06198548525571823, + 0.050075892359018326, + 0.050101637840270996, + 0.04961330443620682, + 0.05178922787308693, + 0.05030789226293564, + 0.05230242758989334, + 0.05474415421485901, + 0.0530126616358757, + 0.045334622263908386, + 0.04584610089659691, + 0.045406509190797806, + 0.046224597841501236, + 0.045296136289834976, + 0.04668199643492699, + 0.04638773202896118, + 0.049726344645023346, + 0.046698544174432755, + 0.04086939990520477, + 0.04283207282423973, + 0.04238206148147583, + 0.04585997387766838, + 0.04673394188284874, + 0.050025053322315216, + 0.05139697343111038, + 0.041993267834186554, + 0.04778415337204933, + 0.04815749078989029, + 0.05211605131626129, + 0.054591212421655655, + 0.05582152307033539, + 0.05712570250034332 + ], + "loss_nats": 1.4382160902023315, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.893858790397644, + "mean_weights": [ + 0.11031289398670197, + 0.07768478244543076, + 0.07875259220600128, + 0.05730495601892471, + 0.06675289571285248, + 0.04819699749350548, + 0.056015171110630035, + 0.3010379672050476, + 0.2039417177438736 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.04378553107380867, + 0.08078505098819733, + 0.15753045678138733, + 0.16996513307094574, + 0.18771789968013763, + 0.196508526802063, + 0.20114651322364807, + 0.2064388245344162, + 0.074164479970932, + 0.07466654479503632, + 0.08760276436805725, + 0.08950652927160263, + 0.09584233164787292, + 0.09770005196332932, + 0.10250397771596909, + 0.10525599867105484, + 0.026560308411717415, + 0.03102109581232071, + 0.039734840393066406, + 0.047966018319129944, + 0.05911440774798393, + 0.06952836364507675, + 0.08398861438035965, + 0.09958801418542862, + 0.03479894623160362, + 0.04605082795023918, + 0.06355229765176773, + 0.07880882918834686, + 0.09107054024934769, + 0.10951601713895798, + 0.12963709235191345, + 0.1483204960823059, + 0.030135396867990494, + 0.041445259004831314, + 0.06172371283173561, + 0.06898269802331924, + 0.09058881551027298, + 0.09753179550170898, + 0.12748143076896667, + 0.14137907326221466, + 0.052755579352378845, + 0.060624755918979645, + 0.07781422883272171, + 0.08929698914289474, + 0.11390363425016403, + 0.13302429020404816, + 0.14357247948646545, + 0.1656738966703415, + 0.021918848156929016, + 0.050240688025951385, + 0.05278511345386505, + 0.07620147615671158, + 0.07971729338169098, + 0.10883574187755585, + 0.11238589137792587, + 0.15144884586334229, + 0.022322971373796463, + 0.06723883002996445, + 0.06908708810806274, + 0.12474417686462402, + 0.12758055329322815, + 0.19924305379390717, + 0.20096708834171295, + 0.30584004521369934 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.482440221589059e-05, + 6.893377576489002e-05, + 6.453022069763392e-05, + 6.28324196441099e-05, + 7.08676889189519e-05, + 6.678568752249703e-05, + 6.212758307810873e-05, + 5.9668491303455085e-05, + 6.326405127765611e-05, + 6.0879101511090994e-05, + 5.78742146899458e-05, + 5.375179898692295e-05, + 5.117018008604646e-05, + 4.859026739723049e-05, + 4.697149051935412e-05, + 4.7182635171338916e-05, + 5.89179435337428e-05, + 6.238038622541353e-05, + 5.742459688917734e-05, + 4.92809122079052e-05, + 5.9098329074913636e-05, + 5.6166292779380456e-05, + 5.261234036879614e-05, + 5.0296643166802824e-05, + 9.122773917624727e-05, + 7.43377095204778e-05, + 5.317839895724319e-05, + 4.298858038964681e-05, + 3.403230584808625e-05, + 2.872925870178733e-05, + 2.510720514692366e-05, + 2.4274773750221357e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 6.507128364319215e-05, + "first_to_last_ratio": 1.3923608598595871, + "imbalance_abs_log_ratio": 0.33100076672063466, + "last_quartile_mean": 4.673449643632921e-05, + "mean": 5.532210701630902e-05, + "normalized": [ + 1.17176307469237, + 1.246043932212637, + 1.1664454623648073, + 1.1357560843730488, + 1.2810012622632039, + 1.2072151825816853, + 1.123015489265321, + 1.0785650533134024, + 1.143558239005716, + 1.1004479907669418, + 1.046131787295889, + 0.9716151803668082, + 0.9249499494110272, + 0.8783155598702346, + 0.8490546194400527, + 0.8528712609848613, + 1.06499818447576, + 1.1275851479594536, + 1.038004515486883, + 0.8907996254260008, + 1.0682588256714696, + 1.0152594651324933, + 0.9510183759502501, + 0.9091599340562954, + 1.649028645083117, + 1.3437252037158123, + 0.9612504263723386, + 0.777059709185945, + 0.6151664801569017, + 0.5193088306148194, + 0.4538367481108778, + 0.4387897543935759 + ], + "population_cv": 0.2506428547669031 + }, + "activation_output_rms_by_block": [ + 0.1168021708726883, + 0.2426510751247406, + 0.26324620842933655, + 0.27180376648902893, + 0.08633904159069061, + 0.09592776000499725, + 0.11017252504825592, + 0.11965741962194443, + 0.031808726489543915, + 0.052126478403806686, + 0.08115695416927338, + 0.1256735473871231, + 0.05155307427048683, + 0.0906737670302391, + 0.1273737996816635, + 0.17883411049842834, + 0.05321500822901726, + 0.0791405439376831, + 0.1188904270529747, + 0.18642042577266693, + 0.07936744391918182, + 0.1209215447306633, + 0.17601744830608368, + 0.21971817314624786, + 0.06200244650244713, + 0.09177663922309875, + 0.12988021969795227, + 0.17691734433174133, + 0.0796835869550705, + 0.14378617703914642, + 0.22992798686027527, + 0.36901578307151794 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.16332499589771032, + "first_to_last_ratio": 1.0184021544363575, + "imbalance_abs_log_ratio": 0.018234883769777943, + "last_quartile_mean": 0.1603737729601562, + "mean": 0.1363275507465005, + "normalized": [ + 0.8567759798595705, + 1.7799122319445722, + 1.930983187095027, + 1.993755224095861, + 0.633320565930485, + 0.7036564471357211, + 0.8081457082224007, + 0.877720013062114, + 0.23332573874734883, + 0.38236202527202495, + 0.5953085324637263, + 0.9218499613538251, + 0.37815594858261026, + 0.6651169667006336, + 0.9343217785707426, + 1.3117972817612509, + 0.3903466902883772, + 0.580517610008589, + 0.8720939120666085, + 1.3674449863719298, + 0.5821819836458785, + 0.8869927176753547, + 1.291136291543784, + 1.6116931019674172, + 0.4548049617479006, + 0.6732068369199734, + 0.9527070572804689, + 1.297737275869623, + 0.5845009795799911, + 1.0547110708862888, + 1.6865848876565215, + 2.7068320456933797 + ], + "population_cv": 0.5561339234804242 + }, + "bits_per_byte": 1.9208319197545975, + "branch_output_rms_by_sublayer": [ + 0.06720057874917984, + 0.09194618463516235, + 0.1542929708957672, + 0.02290475182235241, + 0.04378741979598999, + 0.024343235418200493, + 0.02535730041563511, + 0.01759958639740944, + 0.0859028622508049, + 0.0033764971885830164, + 0.026840992271900177, + 0.009192747063934803, + 0.03328750282526016, + 0.015104862861335278, + 0.02842075191438198, + 0.020453952252864838, + 0.02650054544210434, + 0.01834537461400032, + 0.02435290813446045, + 0.019151484593749046, + 0.03560688719153404, + 0.025080570951104164, + 0.0463782399892807, + 0.0412943996489048, + 0.03340631350874901, + 0.033797912299633026, + 0.029824024066329002, + 0.037299152463674545, + 0.024991154670715332, + 0.04334021359682083, + 0.05468008667230606, + 0.04339433088898659, + 0.03627083823084831, + 0.034207191318273544, + 0.03836885467171669, + 0.0268046073615551, + 0.06414666771888733, + 0.03589542955160141, + 0.08977804332971573, + 0.06778597831726074, + 0.06535173952579498, + 0.04178106412291527, + 0.058602262288331985, + 0.046724773943424225, + 0.06658618152141571, + 0.06704804301261902, + 0.04809500277042389, + 0.07868821918964386, + 0.02468123845756054, + 0.05430305004119873, + 0.016670553013682365, + 0.0522756464779377, + 0.027080193161964417, + 0.06416259706020355, + 0.024483874440193176, + 0.0750751867890358, + 0.031223785132169724, + 0.07228703796863556, + 0.017983488738536835, + 0.08541928231716156, + 0.022625867277383804, + 0.10780027508735657, + 0.01881556585431099, + 0.1644713133573532 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00025086625102172574, + 0.00017275118172797245, + 0.00012514895187085083, + 0.00010290640329296357, + 0.00015019642557553023, + 8.577832334804324e-05, + 0.00011943735053133124, + 0.00012936143606774142, + 0.00011310561698072889, + 0.00011448947514152496, + 0.00014707945466462663, + 0.0001879151917541964, + 0.00013945460994582343, + 0.00014870581484238427, + 0.00015256427365011396, + 0.00016232429675411335, + 0.00017350765572746708, + 0.00019590653199164516, + 0.00026947758478017655, + 0.0003445478052122783, + 0.00033388159232033325, + 0.0003224206708504143, + 0.00034751320015048785, + 0.000333831894033299, + 0.000461635572616515, + 0.0003720801351889239, + 0.0003279167816253298, + 0.00026831033154320815, + 0.00019924703275310594, + 0.00016528720948054884, + 0.00014886231192741455, + 0.00015071164622625237 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00014205579042951983, + "first_to_last_ratio": 0.5427023085127022, + "imbalance_abs_log_ratio": 0.6111943441670763, + "last_quartile_mean": 0.0002617563776701623, + "mean": 0.00020991321917490847, + "normalized": [ + 1.195095058842833, + 0.8229647585177994, + 0.5961937621783194, + 0.4902330767802558, + 0.715516755761724, + 0.4086370724302463, + 0.5689844165164798, + 0.6162615035690157, + 0.538820839513135, + 0.5454133645872371, + 0.7006679009675607, + 0.8952041824370175, + 0.6643441060737771, + 0.7084156749484003, + 0.7267968842066637, + 0.7732923986023861, + 0.8265685048776906, + 0.9332739155813129, + 1.283757096572544, + 1.6413821224150094, + 1.5905696345980438, + 1.5359712557300162, + 1.6555088884655964, + 1.5903328782506847, + 2.199173422383947, + 1.7725426566818128, + 1.5621540316243536, + 1.2781964499322016, + 0.9491876382834422, + 0.787407339710343, + 0.7091612072481068, + 0.7179712017120424 + ], + "population_cv": 0.456906460492845 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.27054542303085327, + "mean_weights": [ + 0.9172357320785522, + 0.08276425302028656 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6030205488204956, + "mean_weights": [ + 0.6185705661773682, + 0.38142940402030945 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5557048320770264, + "mean_weights": [ + 0.7457994222640991, + 0.25420060753822327 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6846973299980164, + "mean_weights": [ + 0.44420135021209717, + 0.5557986497879028 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5171459317207336, + "mean_weights": [ + 0.7784675359725952, + 0.2215324342250824 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6861977577209473, + "mean_weights": [ + 0.5459616184234619, + 0.45403844118118286 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6331400275230408, + "mean_weights": [ + 0.6326359510421753, + 0.3673640489578247 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6767870187759399, + "mean_weights": [ + 0.4901413917541504, + 0.5098586082458496 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5945888757705688, + "mean_weights": [ + 0.0988544300198555, + 0.09581693261861801, + 0.8053286075592041 + ], + "sources": 3 + }, + { + "entropy_mean": 1.090721845626831, + "mean_weights": [ + 0.3291929364204407, + 0.29565638303756714, + 0.3751507103443146 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9767088294029236, + "mean_weights": [ + 0.19007860124111176, + 0.2831641137599945, + 0.5267572402954102 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0831801891326904, + "mean_weights": [ + 0.3917744755744934, + 0.262482225894928, + 0.3457432985305786 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0387191772460938, + "mean_weights": [ + 0.3427180051803589, + 0.22980591654777527, + 0.42747607827186584 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0842926502227783, + "mean_weights": [ + 0.3716634213924408, + 0.26596155762672424, + 0.36237502098083496 + ], + "sources": 3 + }, + { + "entropy_mean": 1.029632329940796, + "mean_weights": [ + 0.35569411516189575, + 0.23412483930587769, + 0.41018107533454895 + ], + "sources": 3 + }, + { + "entropy_mean": 1.082836389541626, + "mean_weights": [ + 0.40107351541519165, + 0.2790895402431488, + 0.31983691453933716 + ], + "sources": 3 + }, + { + "entropy_mean": 1.358649492263794, + "mean_weights": [ + 0.27061086893081665, + 0.17173956334590912, + 0.28016793727874756, + 0.27748164534568787 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3756213188171387, + "mean_weights": [ + 0.2776360511779785, + 0.23655575513839722, + 0.21141205728054047, + 0.2743961215019226 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3476225137710571, + "mean_weights": [ + 0.29357391595840454, + 0.18873462080955505, + 0.2354579120874405, + 0.2822335362434387 + ], + "sources": 4 + }, + { + "entropy_mean": 1.362433671951294, + "mean_weights": [ + 0.30741798877716064, + 0.2113640010356903, + 0.19012174010276794, + 0.2910962700843811 + ], + "sources": 4 + }, + { + "entropy_mean": 1.339874029159546, + "mean_weights": [ + 0.31812360882759094, + 0.19406723976135254, + 0.20874395966529846, + 0.27906519174575806 + ], + "sources": 4 + }, + { + "entropy_mean": 1.368829607963562, + "mean_weights": [ + 0.2827906608581543, + 0.20493531227111816, + 0.21238988637924194, + 0.2998841404914856 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3017525672912598, + "mean_weights": [ + 0.3803398907184601, + 0.1567097008228302, + 0.18186230957508087, + 0.28108808398246765 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3721790313720703, + "mean_weights": [ + 0.29204899072647095, + 0.22929143905639648, + 0.20132920145988464, + 0.27733033895492554 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5936287641525269, + "mean_weights": [ + 0.23810744285583496, + 0.16175906360149384, + 0.17396828532218933, + 0.2102920413017273, + 0.21587316691875458 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5892211198806763, + "mean_weights": [ + 0.21221551299095154, + 0.18061739206314087, + 0.15159276127815247, + 0.19318453967571259, + 0.2623898386955261 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5729010105133057, + "mean_weights": [ + 0.20277059078216553, + 0.1468176692724228, + 0.15990334749221802, + 0.23365569114685059, + 0.2568526864051819 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5752346515655518, + "mean_weights": [ + 0.19923943281173706, + 0.16521164774894714, + 0.1496223509311676, + 0.1875215470790863, + 0.2984050214290619 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5683115720748901, + "mean_weights": [ + 0.19119979441165924, + 0.14639705419540405, + 0.17197296023368835, + 0.21879951655864716, + 0.2716306746006012 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5807743072509766, + "mean_weights": [ + 0.20149533450603485, + 0.19269785284996033, + 0.15572935342788696, + 0.1653299629688263, + 0.28474748134613037 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5668588876724243, + "mean_weights": [ + 0.1679404079914093, + 0.16119730472564697, + 0.1804666668176651, + 0.22590681910514832, + 0.26448875665664673 + ], + "sources": 5 + }, + { + "entropy_mean": 1.586836576461792, + "mean_weights": [ + 0.20100411772727966, + 0.19404375553131104, + 0.15109606087207794, + 0.18302085995674133, + 0.2708352208137512 + ], + "sources": 5 + }, + { + "entropy_mean": 1.777756690979004, + "mean_weights": [ + 0.1529928296804428, + 0.13548150658607483, + 0.15264040231704712, + 0.18369901180267334, + 0.18887785077095032, + 0.1863083839416504 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7673547267913818, + "mean_weights": [ + 0.1572723239660263, + 0.1643807291984558, + 0.13138844072818756, + 0.13080847263336182, + 0.17658346891403198, + 0.23956657946109772 + ], + "sources": 6 + }, + { + "entropy_mean": 1.71624755859375, + "mean_weights": [ + 0.12761017680168152, + 0.10813925415277481, + 0.12779518961906433, + 0.1639901101589203, + 0.169490247964859, + 0.302975058555603 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7770512104034424, + "mean_weights": [ + 0.1621619164943695, + 0.16865593194961548, + 0.1381720006465912, + 0.13563960790634155, + 0.17634665966033936, + 0.2190239131450653 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6464762687683105, + "mean_weights": [ + 0.12666356563568115, + 0.10769195109605789, + 0.11430014669895172, + 0.14935913681983948, + 0.12380701303482056, + 0.378178209066391 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7848572731018066, + "mean_weights": [ + 0.17050954699516296, + 0.17488402128219604, + 0.15024812519550323, + 0.13998055458068848, + 0.1692512035369873, + 0.19512654840946198 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6740212440490723, + "mean_weights": [ + 0.13242365419864655, + 0.12629662454128265, + 0.13919296860694885, + 0.16006915271282196, + 0.10172457993030548, + 0.3402930498123169 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9326330423355103, + "mean_weights": [ + 0.14874926209449768, + 0.17107561230659485, + 0.14548304677009583, + 0.12476755678653717, + 0.1583840548992157, + 0.11667339503765106, + 0.13486707210540771 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9666157960891724, + "mean_weights": [ + 0.08988402783870697, + 0.0995347797870636, + 0.09809467196464539, + 0.08753523230552673, + 0.07272347062826157, + 0.11169559508562088, + 0.2209419310092926, + 0.21959027647972107 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9565823078155518, + "mean_weights": [ + 0.10550355166196823, + 0.12934984266757965, + 0.10976016521453857, + 0.09540660679340363, + 0.1064731627702713, + 0.07497953623533249, + 0.09354954212903976, + 0.28497761487960815 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8834141492843628, + "mean_weights": [ + 0.07920653373003006, + 0.0927790105342865, + 0.08975619077682495, + 0.08865852653980255, + 0.0728786289691925, + 0.09942600131034851, + 0.14772385358810425, + 0.3295712471008301 + ], + "sources": 8 + }, + { + "entropy_mean": 1.989919662475586, + "mean_weights": [ + 0.1199994683265686, + 0.1271483153104782, + 0.11034627258777618, + 0.09819234907627106, + 0.1280786395072937, + 0.08099433779716492, + 0.0841262936592102, + 0.2511143386363983 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8668277263641357, + "mean_weights": [ + 0.09269082546234131, + 0.09107277542352676, + 0.0890658050775528, + 0.08591768890619278, + 0.06984538584947586, + 0.08319880068302155, + 0.13828742504119873, + 0.3499212861061096 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9981560707092285, + "mean_weights": [ + 0.11538191884756088, + 0.1227874755859375, + 0.11879344284534454, + 0.10834046453237534, + 0.11927321553230286, + 0.08193410187959671, + 0.09606333822011948, + 0.2374260425567627 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9545683860778809, + "mean_weights": [ + 0.08704488724470139, + 0.11071415990591049, + 0.10622664541006088, + 0.11126819998025894, + 0.08313152194023132, + 0.10047920048236847, + 0.1495528221130371, + 0.2515825629234314 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0306220054626465, + "mean_weights": [ + 0.11977323889732361, + 0.1289086788892746, + 0.12189700454473495, + 0.11558981239795685, + 0.11734835803508759, + 0.08786976337432861, + 0.11333989351987839, + 0.1952732652425766 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0011324882507324, + "mean_weights": [ + 0.08124207705259323, + 0.09177684038877487, + 0.08694697916507721, + 0.082175113260746, + 0.062464237213134766, + 0.06832721084356308, + 0.12702247500419617, + 0.11632965505123138, + 0.2837154269218445 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1438820362091064, + "mean_weights": [ + 0.08689041435718536, + 0.10568472743034363, + 0.10176857560873032, + 0.11979985237121582, + 0.0954645574092865, + 0.08530118316411972, + 0.1181618869304657, + 0.13306701183319092, + 0.15386179089546204 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1027603149414062, + "mean_weights": [ + 0.08224186301231384, + 0.09548088908195496, + 0.10087728500366211, + 0.08843713998794556, + 0.0712515264749527, + 0.08890259265899658, + 0.1294780671596527, + 0.1532236933708191, + 0.19010695815086365 + ], + "sources": 9 + }, + { + "entropy_mean": 2.140306234359741, + "mean_weights": [ + 0.10960794985294342, + 0.12152446061372757, + 0.1214505136013031, + 0.10309655964374542, + 0.11317203938961029, + 0.08485642075538635, + 0.09329693764448166, + 0.122958704829216, + 0.13003641366958618 + ], + "sources": 9 + }, + { + "entropy_mean": 2.063279628753662, + "mean_weights": [ + 0.09549129009246826, + 0.12276661396026611, + 0.09499089419841766, + 0.095876544713974, + 0.07354907691478729, + 0.07892966270446777, + 0.10574527084827423, + 0.16992294788360596, + 0.1627276986837387 + ], + "sources": 9 + }, + { + "entropy_mean": 2.132303237915039, + "mean_weights": [ + 0.08130102604627609, + 0.10504058003425598, + 0.12868456542491913, + 0.14128273725509644, + 0.09074105322360992, + 0.11087878048419952, + 0.15957562625408173, + 0.09397614747285843, + 0.08851947635412216 + ], + "sources": 9 + }, + { + "entropy_mean": 2.150486946105957, + "mean_weights": [ + 0.09787449240684509, + 0.11536414176225662, + 0.10489250719547272, + 0.1239461898803711, + 0.08205638080835342, + 0.08530600368976593, + 0.10530193150043488, + 0.15596537292003632, + 0.12929296493530273 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.059640128165483475, + 0.055493082851171494, + 0.06647300720214844, + 0.08116324245929718, + 0.1425611823797226, + 0.0804748386144638, + 0.13150103390216827, + 0.11318554729223251, + 0.14635778963565826, + 0.08794279396533966, + 0.10851149260997772, + 0.1140768751502037, + 0.101070836186409, + 0.09991079568862915, + 0.10510540753602982, + 0.10183984041213989, + 0.10598240047693253, + 0.07249010354280472, + 0.0838986486196518, + 0.0745912417769432, + 0.07693319767713547, + 0.07456963509321213, + 0.07731878757476807, + 0.06445791572332382, + 0.08107778429985046, + 0.05914963036775589, + 0.061363689601421356, + 0.0565083771944046, + 0.060109417885541916, + 0.05957059562206268, + 0.0694725513458252, + 0.0689009502530098, + 0.07612684369087219, + 0.05829247832298279, + 0.060142867267131805, + 0.05563490837812424, + 0.062123168259859085, + 0.05913867801427841, + 0.06380540877580643, + 0.0680575743317604, + 0.06542093306779861, + 0.05585707351565361, + 0.056685853749513626, + 0.05669882521033287, + 0.05797213315963745, + 0.05780034884810448, + 0.05993710830807686, + 0.05976834520697594, + 0.06388194859027863, + 0.061331938952207565, + 0.05007411539554596, + 0.05387401208281517, + 0.05314013361930847, + 0.05847715586423874, + 0.058448728173971176, + 0.06415793299674988, + 0.06349384039640427, + 0.0486287921667099, + 0.057339318096637726, + 0.05916948989033699, + 0.06191723421216011, + 0.06987231969833374, + 0.06776049733161926, + 0.07122725248336792 + ], + "loss_nats": 1.3314192295074463, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.8463083505630493, + "mean_weights": [ + 0.1012595072388649, + 0.07702568173408508, + 0.07309570908546448, + 0.05961478874087334, + 0.05607825890183449, + 0.04819445312023163, + 0.04543677717447281, + 0.28805360198020935, + 0.25124120712280273 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.06720057874917984, + 0.1168021708726883, + 0.23284447193145752, + 0.2426510751247406, + 0.25351840257644653, + 0.26324620842933655, + 0.26641416549682617, + 0.27180376648902893, + 0.0859028622508049, + 0.08633904159069061, + 0.09406589716672897, + 0.09592776000499725, + 0.10818329453468323, + 0.11017252504825592, + 0.1166386753320694, + 0.11965741962194443, + 0.02650054544210434, + 0.031808726489543915, + 0.04396911710500717, + 0.052126478403806686, + 0.06932095438241959, + 0.08115695416927338, + 0.10344681888818741, + 0.1256735473871231, + 0.03340631350874901, + 0.05155307427048683, + 0.06845457851886749, + 0.0906737670302391, + 0.09912405163049698, + 0.1273737996816635, + 0.1522112637758255, + 0.17883411049842834, + 0.03627083823084831, + 0.05321500822901726, + 0.06814355403184891, + 0.0791405439376831, + 0.10699275881052017, + 0.1188904270529747, + 0.15982621908187866, + 0.18642042577266693, + 0.06535173952579498, + 0.07936744391918182, + 0.10392377525568008, + 0.1209215447306633, + 0.1486607789993286, + 0.17601744830608368, + 0.18852856755256653, + 0.21971817314624786, + 0.02468123845756054, + 0.06200244650244713, + 0.06476796418428421, + 0.09177663922309875, + 0.09692107141017914, + 0.12988021969795227, + 0.1334778219461441, + 0.17691734433174133, + 0.031223785132169724, + 0.0796835869550705, + 0.08222448080778122, + 0.14378617703914642, + 0.1467829793691635, + 0.22992798686027527, + 0.23182450234889984, + 0.36901578307151794 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00011124008597107604, + 0.00011688668018905446, + 0.00011025072308257222, + 0.00010790223313961178, + 0.00012647708354052156, + 0.00011840949446195737, + 0.00011017030919902027, + 0.000105609382444527, + 0.00011779580381698906, + 0.0001132004035753198, + 0.00010671230120351538, + 9.833346848608926e-05, + 9.582890197634697e-05, + 8.986716420622543e-05, + 8.526558667654172e-05, + 8.496129157720134e-05, + 0.00010948973067570478, + 0.00011349687702022493, + 0.00010318395652575418, + 8.565785537939519e-05, + 0.00010152422328246757, + 9.607809624867514e-05, + 9.015767864184454e-05, + 8.61775697558187e-05, + 0.00017974359798245132, + 0.0001403082860633731, + 9.42990300245583e-05, + 6.95259150234051e-05, + 7.184838614193723e-05, + 5.5086766224121675e-05, + 4.244847150403075e-05, + 3.74928604287561e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00011336824900354259, + "first_to_last_ratio": 1.31298102295576, + "imbalance_abs_log_ratio": 0.2723001420218161, + "last_quartile_mean": 8.63441641740792e-05, + "mean": 9.923219420215901e-05, + "normalized": [ + 1.1210080243157192, + 1.1779108698425953, + 1.1110378438066775, + 1.0873712307498702, + 1.2745569576226279, + 1.1932568398188368, + 1.1102274829736989, + 1.0642653152400943, + 1.187072449259881, + 1.1407628792799271, + 1.075379840801642, + 0.9909432042363304, + 0.9657037491393293, + 0.9056250839636294, + 0.859253263137928, + 0.8561867674125543, + 1.1033690382039603, + 1.1437505532630414, + 1.0398233895296574, + 0.8632063018267061, + 1.0230976357898443, + 0.9682149731864415, + 0.9085527067775245, + 0.868443658317733, + 1.8113435808571552, + 1.4139391675400481, + 0.9502866562886768, + 0.700638694754292, + 0.724043106368941, + 0.5551299824318824, + 0.4277691513860246, + 0.3778296018767301 + ], + "population_cv": 0.2678383955560582 + }, + "activation_output_rms_by_block": [ + 0.10511645674705505, + 0.24062439799308777, + 0.2566715478897095, + 0.2626059055328369, + 0.06572642922401428, + 0.07418278604745865, + 0.08785579353570938, + 0.09792577475309372, + 0.026660021394491196, + 0.04438937455415726, + 0.06966898590326309, + 0.11229602992534637, + 0.046715885400772095, + 0.07927730679512024, + 0.11152955144643784, + 0.15471883118152618, + 0.043632324784994125, + 0.06648805737495422, + 0.10486011952161789, + 0.16671040654182434, + 0.07499032467603683, + 0.11423121392726898, + 0.1617974191904068, + 0.20152884721755981, + 0.05626026913523674, + 0.08138494193553925, + 0.11230085045099258, + 0.15036241710186005, + 0.06580562144517899, + 0.11371545493602753, + 0.1807933896780014, + 0.2874876856803894 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.14883863646537066, + "first_to_last_ratio": 1.1360528719285305, + "imbalance_abs_log_ratio": 0.12755986140651335, + "last_quartile_mean": 0.13101382879540324, + "mean": 0.11932232568506151, + "normalized": [ + 0.8809454236124987, + 2.0165915859550885, + 2.151077314459714, + 2.2008111560443187, + 0.5508309433851645, + 0.6217008059602874, + 0.7362896510045851, + 0.8206827531300245, + 0.2234286102070912, + 0.372012314538004, + 0.5838721756659933, + 0.941114994873128, + 0.3915100140109041, + 0.6643962589563012, + 0.9346913983289944, + 1.2966461246312777, + 0.3656677368169723, + 0.5572138909732812, + 0.878797148141287, + 1.3971434564713279, + 0.6284685137127293, + 0.9573331165935369, + 1.355969373388294, + 1.6889450156165544, + 0.471498261639064, + 0.6820596352634466, + 0.9411553940659741, + 1.2601364936409765, + 0.5514946265702687, + 0.9530107151629551, + 1.5151681685721259, + 2.4093369286078308 + ], + "population_cv": 0.5708800395199106 + }, + "bits_per_byte": 1.8577768885728607, + "branch_output_rms_by_sublayer": [ + 0.06238187104463577, + 0.08053887635469437, + 0.16399894654750824, + 0.017000190913677216, + 0.03862038999795914, + 0.019791575148701668, + 0.021512897685170174, + 0.014420416206121445, + 0.06530912220478058, + 0.0033905019517987967, + 0.022078748792409897, + 0.008273337967693806, + 0.030224867165088654, + 0.013179181143641472, + 0.02611314132809639, + 0.01788962632417679, + 0.021946560591459274, + 0.015773503109812737, + 0.01972159370779991, + 0.016600634902715683, + 0.03097161091864109, + 0.022710414603352547, + 0.045145582407712936, + 0.037076350301504135, + 0.030420396476984024, + 0.030718937516212463, + 0.024537762627005577, + 0.0359642468392849, + 0.023272594437003136, + 0.041099995374679565, + 0.043957170099020004, + 0.04132891073822975, + 0.026337940245866776, + 0.03245834633708, + 0.030893171206116676, + 0.02636813186109066, + 0.05636321008205414, + 0.038928091526031494, + 0.07654957473278046, + 0.06857583671808243, + 0.06051572784781456, + 0.04119264334440231, + 0.05507543310523033, + 0.04661190137267113, + 0.05821720138192177, + 0.06451209634542465, + 0.04355492815375328, + 0.07666601985692978, + 0.02224583551287651, + 0.04911409318447113, + 0.015111318789422512, + 0.048929363489151, + 0.022675910964608192, + 0.06037810072302818, + 0.022706424817442894, + 0.06889855116605759, + 0.024911684915423393, + 0.05986437201499939, + 0.015384807251393795, + 0.07041782140731812, + 0.019863350316882133, + 0.09056799858808517, + 0.01973430998623371, + 0.13252165913581848 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00039191822003706517, + 0.00026462320551387335, + 0.00017778635958335228, + 0.00015849989789238825, + 0.00020547069871121068, + 0.00015590483188384908, + 0.00019141420411307775, + 0.00021198189606938124, + 0.00018748318786559928, + 0.00017306148569263557, + 0.0002503442914174138, + 0.0003249768393815801, + 0.0002484548480205063, + 0.00028792292531092464, + 0.00028808665289140875, + 0.00031128748157802885, + 0.00031647562615831783, + 0.00034860364231012946, + 0.00048771446649442993, + 0.0005942059172746123, + 0.0005465704215023327, + 0.0005516667910170206, + 0.000571905249483349, + 0.0005665802942068898, + 0.0008802203173288816, + 0.0007037795562637934, + 0.0006030101992944859, + 0.0004527085851154531, + 0.000392272427207564, + 0.00030193073255886215, + 0.0002518269486390726, + 0.00022468296634653017 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00021969991422552472, + "first_to_last_ratio": 0.461259887874593, + "imbalance_abs_log_ratio": 0.7737936467306832, + "last_quartile_mean": 0.00047630396659433037, + "mean": 0.0003632303489738756, + "normalized": [ + 1.0789798295881188, + 0.728527245207987, + 0.4894589895519416, + 0.4363619348992998, + 0.5656760215429814, + 0.42921752635560234, + 0.5269774528858124, + 0.583601829164962, + 0.5161550745834939, + 0.47645106247738833, + 0.6892163392311119, + 0.8946852604680157, + 0.6840145618954758, + 0.7926730960788543, + 0.7931238502103486, + 0.8569974465443618, + 0.8712807920713682, + 0.9597315953772406, + 1.3427139771558778, + 1.6358928127928787, + 1.5047487718093826, + 1.5187794537969561, + 1.5744974259419104, + 1.559837430455569, + 2.4233115977657174, + 1.9375571403985599, + 1.6601316519888483, + 1.2463401981534672, + 1.0799549881108013, + 0.8312376248620617, + 0.6932982041574521, + 0.6185688144761546 + ], + "population_cv": 0.4929153085177118 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.26523321866989136, + "mean_weights": [ + 0.9203108549118042, + 0.079689159989357 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5873921513557434, + "mean_weights": [ + 0.652948796749115, + 0.347051203250885 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6240328550338745, + "mean_weights": [ + 0.6679002046585083, + 0.3320997655391693 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6765515208244324, + "mean_weights": [ + 0.4129548668861389, + 0.5870451331138611 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5575729608535767, + "mean_weights": [ + 0.7447558641433716, + 0.2552441358566284 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6891655325889587, + "mean_weights": [ + 0.5198555588722229, + 0.4801444411277771 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6571643948554993, + "mean_weights": [ + 0.5838730335235596, + 0.4161270260810852 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6814906597137451, + "mean_weights": [ + 0.48431262373924255, + 0.5156873464584351 + ], + "sources": 2 + }, + { + "entropy_mean": 0.4672169089317322, + "mean_weights": [ + 0.06870044022798538, + 0.07141369581222534, + 0.8598858714103699 + ], + "sources": 3 + }, + { + "entropy_mean": 1.08608078956604, + "mean_weights": [ + 0.30299142003059387, + 0.29511722922325134, + 0.4018913805484772 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9544397592544556, + "mean_weights": [ + 0.17331764101982117, + 0.27222588658332825, + 0.5544564723968506 + ], + "sources": 3 + }, + { + "entropy_mean": 1.08473539352417, + "mean_weights": [ + 0.36410826444625854, + 0.26349887251853943, + 0.372392863035202 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0342564582824707, + "mean_weights": [ + 0.3111056685447693, + 0.22618326544761658, + 0.4627110958099365 + ], + "sources": 3 + }, + { + "entropy_mean": 1.08628249168396, + "mean_weights": [ + 0.34201672673225403, + 0.2747601270675659, + 0.38322314620018005 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0200296640396118, + "mean_weights": [ + 0.32127073407173157, + 0.22071510553359985, + 0.4580141305923462 + ], + "sources": 3 + }, + { + "entropy_mean": 1.08608078956604, + "mean_weights": [ + 0.38335341215133667, + 0.28152504563331604, + 0.3351215124130249 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3548029661178589, + "mean_weights": [ + 0.24374310672283173, + 0.17125959694385529, + 0.30325156450271606, + 0.2817457318305969 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3733057975769043, + "mean_weights": [ + 0.2756107449531555, + 0.22786866128444672, + 0.20998063683509827, + 0.2865399718284607 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3419569730758667, + "mean_weights": [ + 0.2657378017902374, + 0.1853940784931183, + 0.24564385414123535, + 0.30322426557540894 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3651163578033447, + "mean_weights": [ + 0.2952176630496979, + 0.21084429323673248, + 0.19521303474903107, + 0.2987250089645386 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3278114795684814, + "mean_weights": [ + 0.2845562994480133, + 0.1759188175201416, + 0.2037607729434967, + 0.3357641100883484 + ], + "sources": 4 + }, + { + "entropy_mean": 1.36915922164917, + "mean_weights": [ + 0.2759510278701782, + 0.20335039496421814, + 0.21458977460861206, + 0.3061088025569916 + ], + "sources": 4 + }, + { + "entropy_mean": 1.311241865158081, + "mean_weights": [ + 0.34542542695999146, + 0.1547655463218689, + 0.18925689160823822, + 0.31055212020874023 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3734006881713867, + "mean_weights": [ + 0.28805550932884216, + 0.2271580696105957, + 0.2063707411289215, + 0.27841565012931824 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5907577276229858, + "mean_weights": [ + 0.22183816134929657, + 0.15149948000907898, + 0.17484480142593384, + 0.22298912703990936, + 0.22882843017578125 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5877845287322998, + "mean_weights": [ + 0.20816664397716522, + 0.1766117662191391, + 0.15435215830802917, + 0.19166965782642365, + 0.26919978857040405 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5564230680465698, + "mean_weights": [ + 0.19287627935409546, + 0.12910859286785126, + 0.15882132947444916, + 0.2405325174331665, + 0.27866131067276 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5712422132492065, + "mean_weights": [ + 0.1944999098777771, + 0.16216421127319336, + 0.15100529789924622, + 0.1858535259962082, + 0.30647706985473633 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5436700582504272, + "mean_weights": [ + 0.1839108169078827, + 0.12745870649814606, + 0.15784117579460144, + 0.21722730994224548, + 0.3135620057582855 + ], + "sources": 5 + }, + { + "entropy_mean": 1.579209566116333, + "mean_weights": [ + 0.19799913465976715, + 0.18985037505626678, + 0.15529179573059082, + 0.16504307091236115, + 0.2918156385421753 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5596954822540283, + "mean_weights": [ + 0.168576180934906, + 0.14996618032455444, + 0.17477169632911682, + 0.22731834650039673, + 0.279367595911026 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5889356136322021, + "mean_weights": [ + 0.19976919889450073, + 0.193306565284729, + 0.15291622281074524, + 0.1847071349620819, + 0.2693008780479431 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7739142179489136, + "mean_weights": [ + 0.15349149703979492, + 0.12625131011009216, + 0.15205146372318268, + 0.19054609537124634, + 0.19579070806503296, + 0.18186889588832855 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7696181535720825, + "mean_weights": [ + 0.15602925419807434, + 0.16335827112197876, + 0.13179393112659454, + 0.1345069408416748, + 0.17888715863227844, + 0.2354244589805603 + ], + "sources": 6 + }, + { + "entropy_mean": 1.684680461883545, + "mean_weights": [ + 0.12515553832054138, + 0.09176654368638992, + 0.1207391768693924, + 0.16392305493354797, + 0.16349825263023376, + 0.33491742610931396 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7786550521850586, + "mean_weights": [ + 0.16337358951568604, + 0.1725746989250183, + 0.13922110199928284, + 0.13472947478294373, + 0.17843866348266602, + 0.21166247129440308 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6269428730010986, + "mean_weights": [ + 0.12816154956817627, + 0.09998750686645508, + 0.10617299377918243, + 0.1530253142118454, + 0.12042219936847687, + 0.39223045110702515 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7851910591125488, + "mean_weights": [ + 0.16887614130973816, + 0.17742523550987244, + 0.152394637465477, + 0.13861840963363647, + 0.16898959875106812, + 0.19369596242904663 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6659839153289795, + "mean_weights": [ + 0.13287794589996338, + 0.12202534079551697, + 0.13115590810775757, + 0.1583862006664276, + 0.1039738580584526, + 0.35158073902130127 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9315086603164673, + "mean_weights": [ + 0.14864811301231384, + 0.17611286044120789, + 0.1488860547542572, + 0.11854793131351471, + 0.16131754219532013, + 0.1166355237364769, + 0.12985196709632874 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9486479759216309, + "mean_weights": [ + 0.0882779210805893, + 0.09357968717813492, + 0.08933381736278534, + 0.08906754851341248, + 0.06841318309307098, + 0.11134092509746552, + 0.21616394817829132, + 0.24382297694683075 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9349684715270996, + "mean_weights": [ + 0.10477666556835175, + 0.12480375170707703, + 0.10871509462594986, + 0.0876661017537117, + 0.1105533093214035, + 0.07291407883167267, + 0.08379648625850677, + 0.3067745268344879 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8216801881790161, + "mean_weights": [ + 0.07579677551984787, + 0.08220995962619781, + 0.07940877974033356, + 0.08474553376436234, + 0.06977429240942001, + 0.09579598158597946, + 0.13471534848213196, + 0.3775533437728882 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9683878421783447, + "mean_weights": [ + 0.11621449887752533, + 0.12419174611568451, + 0.11125136911869049, + 0.09063617140054703, + 0.13253585994243622, + 0.07837043702602386, + 0.07551991939544678, + 0.2712799906730652 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8001704216003418, + "mean_weights": [ + 0.08999691903591156, + 0.08266323804855347, + 0.0765576958656311, + 0.0837859958410263, + 0.06825517117977142, + 0.07610428333282471, + 0.1253497153520584, + 0.397286981344223 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9856641292572021, + "mean_weights": [ + 0.11320659518241882, + 0.12357879430055618, + 0.12255804240703583, + 0.0981803834438324, + 0.1339874416589737, + 0.07771916687488556, + 0.08289110660552979, + 0.24787844717502594 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9152603149414062, + "mean_weights": [ + 0.08566148579120636, + 0.10291853547096252, + 0.09646031260490417, + 0.10544642806053162, + 0.08053722232580185, + 0.09461864084005356, + 0.14143124222755432, + 0.2929261326789856 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0208582878112793, + "mean_weights": [ + 0.12403465062379837, + 0.1289644092321396, + 0.12299773842096329, + 0.10794863104820251, + 0.124393530189991, + 0.08252135664224625, + 0.10208340734243393, + 0.20705626904964447 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9470036029815674, + "mean_weights": [ + 0.07853354513645172, + 0.08334071934223175, + 0.07690615952014923, + 0.07859408855438232, + 0.056716568768024445, + 0.06552165746688843, + 0.11653584241867065, + 0.11943976581096649, + 0.32441163063049316 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1419777870178223, + "mean_weights": [ + 0.08520209044218063, + 0.1036384105682373, + 0.1072966456413269, + 0.11538195610046387, + 0.10057751089334488, + 0.08314792811870575, + 0.10494177788496017, + 0.13152506947517395, + 0.16828861832618713 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0398616790771484, + "mean_weights": [ + 0.07691620290279388, + 0.08514301478862762, + 0.08627729117870331, + 0.08062928169965744, + 0.0665137991309166, + 0.08139452338218689, + 0.11561635136604309, + 0.15924027562141418, + 0.2482692301273346 + ], + "sources": 9 + }, + { + "entropy_mean": 2.133859634399414, + "mean_weights": [ + 0.10850068926811218, + 0.12340156733989716, + 0.12121301889419556, + 0.09419046342372894, + 0.1215018555521965, + 0.08006064593791962, + 0.08310537785291672, + 0.12693293392658234, + 0.14109346270561218 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0074024200439453, + "mean_weights": [ + 0.09073749929666519, + 0.10495538264513016, + 0.08602896332740784, + 0.0905105322599411, + 0.07132501900196075, + 0.06970499455928802, + 0.09266062080860138, + 0.17847850918769836, + 0.2155984789133072 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1315131187438965, + "mean_weights": [ + 0.08339932560920715, + 0.1049160286784172, + 0.14784187078475952, + 0.1476159691810608, + 0.09480608999729156, + 0.11423666775226593, + 0.1337963342666626, + 0.09029372036457062, + 0.08309400081634521 + ], + "sources": 9 + }, + { + "entropy_mean": 2.114269733428955, + "mean_weights": [ + 0.0949711799621582, + 0.1047864705324173, + 0.09588303416967392, + 0.11574660986661911, + 0.07552529871463776, + 0.07755057513713837, + 0.09274275600910187, + 0.17604602873325348, + 0.1667480617761612 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06273526698350906, + 0.05837090313434601, + 0.06236566603183746, + 0.09409056603908539, + 0.14841069281101227, + 0.0853683277964592, + 0.13471153378486633, + 0.11974260210990906, + 0.14304757118225098, + 0.07015866786241531, + 0.10177619755268097, + 0.10180512070655823, + 0.09450531005859375, + 0.09075505286455154, + 0.09923894703388214, + 0.09139131754636765, + 0.09934699535369873, + 0.06662803888320923, + 0.07585397362709045, + 0.06760125607252121, + 0.07097551971673965, + 0.06389196962118149, + 0.07028494775295258, + 0.05738966539502144, + 0.07284899801015854, + 0.0505077987909317, + 0.054266586899757385, + 0.04711681231856346, + 0.053017228841781616, + 0.049717944115400314, + 0.061314862221479416, + 0.058796487748622894, + 0.06646712124347687, + 0.04949406906962395, + 0.0525309182703495, + 0.04484536498785019, + 0.05494789034128189, + 0.05011478438973427, + 0.05651598423719406, + 0.059141308069229126, + 0.057020124047994614, + 0.04842044413089752, + 0.04913678020238876, + 0.049134157598018646, + 0.05034123733639717, + 0.050292667001485825, + 0.052332911640405655, + 0.05225369334220886, + 0.05686299875378609, + 0.05377275124192238, + 0.04332364350557327, + 0.04606988653540611, + 0.04622604325413704, + 0.05107564106583595, + 0.050495367497205734, + 0.05663415417075157, + 0.054730795323848724, + 0.040449123829603195, + 0.04781271517276764, + 0.04964512959122658, + 0.051698628813028336, + 0.05980329588055611, + 0.05537193641066551, + 0.06257455050945282 + ], + "loss_nats": 1.287712812423706, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.7376251220703125, + "mean_weights": [ + 0.09475357085466385, + 0.06791549921035767, + 0.06304144859313965, + 0.04941102862358093, + 0.051383212208747864, + 0.039511408656835556, + 0.03548463061451912, + 0.2899034321308136, + 0.30859577655792236 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.06238187104463577, + 0.10511645674705505, + 0.23381692171096802, + 0.24062439799308777, + 0.2498445212841034, + 0.2566715478897095, + 0.25889042019844055, + 0.2626059055328369, + 0.06530912220478058, + 0.06572642922401428, + 0.07232747226953506, + 0.07418278604745865, + 0.08615485578775406, + 0.08785579353570938, + 0.09516260772943497, + 0.09792577475309372, + 0.021946560591459274, + 0.026660021394491196, + 0.037179429084062576, + 0.04438937455415726, + 0.05909505486488342, + 0.06966898590326309, + 0.09241165965795517, + 0.11229602992534637, + 0.030420396476984024, + 0.046715885400772095, + 0.057841956615448, + 0.07927730679512024, + 0.0856231078505516, + 0.11152955144643784, + 0.12946195900440216, + 0.15471883118152618, + 0.026337940245866776, + 0.043632324784994125, + 0.05467098951339722, + 0.06648805737495422, + 0.09118234366178513, + 0.10486011952161789, + 0.13852031528949738, + 0.16671040654182434, + 0.06051572784781456, + 0.07499032467603683, + 0.0974416583776474, + 0.11423121392726898, + 0.13639897108078003, + 0.1617974191904068, + 0.17219416797161102, + 0.20152884721755981, + 0.02224583551287651, + 0.05626026913523674, + 0.05840611457824707, + 0.08138494193553925, + 0.0850442424416542, + 0.11230085045099258, + 0.11544818431138992, + 0.15036241710186005, + 0.024911684915423393, + 0.06580562144517899, + 0.06786322593688965, + 0.11371545493602753, + 0.11594865471124649, + 0.1807933896780014, + 0.18226802349090576, + 0.2874876856803894 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 8.034013546791773, + "cross_entropy_nats": 5.5687538385391235, + "step": 0 + }, + { + "bits_per_byte": 5.132682479096697, + "cross_entropy_nats": 3.5577043890953064, + "step": 100 + }, + { + "bits_per_byte": 3.1044505325616547, + "cross_entropy_nats": 2.1518411338329315, + "step": 500 + }, + { + "bits_per_byte": 1.9690143857398397, + "cross_entropy_nats": 1.3648167699575424, + "step": 2000 + }, + { + "bits_per_byte": 1.7962893072710877, + "cross_entropy_nats": 1.2450928688049316, + "step": 4000 + }, + { + "bits_per_byte": 1.7219461947999706, + "cross_entropy_nats": 1.1935621500015259, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.40.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.40.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "40": true, + "41": true, + "42": true, + "43": true, + "44": true, + "45": true, + "46": true, + "47": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.49.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.49.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.51.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.51.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.53.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.53.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.55.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.55.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_group_6_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "b95e2bb4aec55e77b896b278521316e374c02a715393556f84aae8442774726f", + "final_model_state": "bce7611a8a2286b71f3edda7e741f39192cb6a0ae2a06a70fa1c50d58188a732", + "final_optimizer_state": "981d2fbd63281a219bbd804211063705cfcb66f0d31269cb61ad38a5f6615565", + "final_public_parameters": "1860f92a601b1ba646878b266c4c7609d089e988495bac39a9424b106a6c19d4", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "bbf2cfc5c0019fb54bf6927274ab9b185b564a77b4b8ef2dc6f5d81575f835c2" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "add6249175ac3bb3fa911069fbe8f25d17d9ab05df34b66ee89996983522ce83", + "1": "619b5ee4f70e975312ead9adf5df4f9d8584ce2839369c749442e429923405cf", + "7999": "fe22994f7750c2f8323fdb5ac51c9be1e208733abaf6a337f5b1ae1df1cd2730" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "7bb8c454ecb1bae838f9a55f02d5a3c8034be2ba9b13a241643c568567c8433b", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073002, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 212.72869830964808, + "measured_steps": 7980, + "median_ms": 212.7929470007075, + "p95_ms": 215.46092434728052, + "peak_allocated_bytes": 11759431680, + "peak_reserved_bytes": 12266242048, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 8.018870646365954, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.558257579803467, + "step": 1, + "unclipped_grad_norm": 26.8336238861084 + }, + { + "bits_per_byte": 7.214361259062063, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.000614166259766, + "step": 10, + "unclipped_grad_norm": 12.742136001586914 + }, + { + "bits_per_byte": 6.68445106694599, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.633308410644531, + "step": 20, + "unclipped_grad_norm": 3.454619884490967 + }, + { + "bits_per_byte": 6.541280325616926, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.534070014953613, + "step": 30, + "unclipped_grad_norm": 3.060382604598999 + }, + { + "bits_per_byte": 6.411225669173263, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.443922996520996, + "step": 40, + "unclipped_grad_norm": 3.000201463699341 + }, + { + "bits_per_byte": 6.085799409280053, + "learning_rate": 3.75e-05, + "loss_nats": 4.21835470199585, + "step": 50, + "unclipped_grad_norm": 2.9775631427764893 + }, + { + "bits_per_byte": 5.879641053968391, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.075456619262695, + "step": 60, + "unclipped_grad_norm": 2.7089359760284424 + }, + { + "bits_per_byte": 5.710390519497189, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.9581410884857178, + "step": 70, + "unclipped_grad_norm": 2.559527635574341 + }, + { + "bits_per_byte": 5.512353184959268, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.8208720684051514, + "step": 80, + "unclipped_grad_norm": 2.5548105239868164 + }, + { + "bits_per_byte": 5.341608121252069, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.7025206089019775, + "step": 90, + "unclipped_grad_norm": 2.3835411071777344 + }, + { + "bits_per_byte": 5.169907177971373, + "learning_rate": 7.5e-05, + "loss_nats": 3.5835065841674805, + "step": 100, + "unclipped_grad_norm": 2.2460482120513916 + }, + { + "bits_per_byte": 5.001288507759441, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.4666290283203125, + "step": 110, + "unclipped_grad_norm": 2.19248104095459 + }, + { + "bits_per_byte": 4.734167118231137, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.2814745903015137, + "step": 120, + "unclipped_grad_norm": 1.882218599319458 + }, + { + "bits_per_byte": 4.5854447769322215, + "learning_rate": 9.75e-05, + "loss_nats": 3.1783881187438965, + "step": 130, + "unclipped_grad_norm": 1.6648681163787842 + }, + { + "bits_per_byte": 4.391957759370829, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.0442731380462646, + "step": 140, + "unclipped_grad_norm": 1.4589451551437378 + }, + { + "bits_per_byte": 4.26054650377716, + "learning_rate": 0.0001125, + "loss_nats": 2.953185796737671, + "step": 150, + "unclipped_grad_norm": 1.234578013420105 + }, + { + "bits_per_byte": 4.153576390628209, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.879039764404297, + "step": 160, + "unclipped_grad_norm": 1.0436146259307861 + }, + { + "bits_per_byte": 3.9983630801204293, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.771454095840454, + "step": 170, + "unclipped_grad_norm": 0.9488199949264526 + }, + { + "bits_per_byte": 3.861247159997641, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.676412582397461, + "step": 180, + "unclipped_grad_norm": 1.0612412691116333 + }, + { + "bits_per_byte": 3.876052802445119, + "learning_rate": 0.0001425, + "loss_nats": 2.6866750717163086, + "step": 190, + "unclipped_grad_norm": 0.8768960237503052 + }, + { + "bits_per_byte": 3.7462471768799053, + "learning_rate": 0.00015, + "loss_nats": 2.596700668334961, + "step": 200, + "unclipped_grad_norm": 0.6789646148681641 + }, + { + "bits_per_byte": 3.6967457544473645, + "learning_rate": 0.0001575, + "loss_nats": 2.5623888969421387, + "step": 210, + "unclipped_grad_norm": 0.8596236109733582 + }, + { + "bits_per_byte": 3.6620829951312674, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.538362503051758, + "step": 220, + "unclipped_grad_norm": 0.7826887965202332 + }, + { + "bits_per_byte": 3.674413807232829, + "learning_rate": 0.0001725, + "loss_nats": 2.5469095706939697, + "step": 230, + "unclipped_grad_norm": 0.6888794302940369 + }, + { + "bits_per_byte": 3.6587785204775893, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.536072015762329, + "step": 240, + "unclipped_grad_norm": 0.8762317299842834 + }, + { + "bits_per_byte": 3.514202680890976, + "learning_rate": 0.0001875, + "loss_nats": 2.4358596801757812, + "step": 250, + "unclipped_grad_norm": 0.7828739881515503 + }, + { + "bits_per_byte": 3.5291637956548336, + "learning_rate": 0.000195, + "loss_nats": 2.446229934692383, + "step": 260, + "unclipped_grad_norm": 1.0910320281982422 + }, + { + "bits_per_byte": 3.4879116930535785, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.4176361560821533, + "step": 270, + "unclipped_grad_norm": 0.9967095851898193 + }, + { + "bits_per_byte": 3.493534493840887, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.4215335845947266, + "step": 280, + "unclipped_grad_norm": 0.8715087175369263 + }, + { + "bits_per_byte": 3.393445062587655, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.3521568775177, + "step": 290, + "unclipped_grad_norm": 1.011237382888794 + }, + { + "bits_per_byte": 3.478433729164529, + "learning_rate": 0.000225, + "loss_nats": 2.4110665321350098, + "step": 300, + "unclipped_grad_norm": 1.0920255184173584 + }, + { + "bits_per_byte": 3.381976227532218, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.344207286834717, + "step": 310, + "unclipped_grad_norm": 1.060672402381897 + }, + { + "bits_per_byte": 3.357329049871767, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.3271231651306152, + "step": 320, + "unclipped_grad_norm": 1.0780916213989258 + }, + { + "bits_per_byte": 3.349624227113113, + "learning_rate": 0.0002475, + "loss_nats": 2.3217825889587402, + "step": 330, + "unclipped_grad_norm": 1.0222147703170776 + }, + { + "bits_per_byte": 3.358591746494402, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.327998399734497, + "step": 340, + "unclipped_grad_norm": 1.203108549118042 + }, + { + "bits_per_byte": 3.3210269519276454, + "learning_rate": 0.0002625, + "loss_nats": 2.3019604682922363, + "step": 350, + "unclipped_grad_norm": 0.9524287581443787 + }, + { + "bits_per_byte": 3.299855543641132, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.287285566329956, + "step": 360, + "unclipped_grad_norm": 1.0595638751983643 + }, + { + "bits_per_byte": 3.3240817077722706, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.3040778636932373, + "step": 370, + "unclipped_grad_norm": 1.2980397939682007 + }, + { + "bits_per_byte": 3.3684824687455217, + "learning_rate": 0.000285, + "loss_nats": 2.3348541259765625, + "step": 380, + "unclipped_grad_norm": 1.2004672288894653 + }, + { + "bits_per_byte": 3.295546690306415, + "learning_rate": 0.0002925, + "loss_nats": 2.284298896789551, + "step": 390, + "unclipped_grad_norm": 0.8376847505569458 + }, + { + "bits_per_byte": 3.242199735800582, + "learning_rate": 0.0003, + "loss_nats": 2.247321605682373, + "step": 400, + "unclipped_grad_norm": 1.4164752960205078 + }, + { + "bits_per_byte": 3.225045498272922, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.23543119430542, + "step": 410, + "unclipped_grad_norm": 1.1587398052215576 + }, + { + "bits_per_byte": 3.2103038333715657, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.225213050842285, + "step": 420, + "unclipped_grad_norm": 0.9722725749015808 + }, + { + "bits_per_byte": 3.1944738622555495, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.214240550994873, + "step": 430, + "unclipped_grad_norm": 0.9184331893920898 + }, + { + "bits_per_byte": 3.223499718206967, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.2343597412109375, + "step": 440, + "unclipped_grad_norm": 1.4326251745224 + }, + { + "bits_per_byte": 3.177806679595133, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.2026877403259277, + "step": 450, + "unclipped_grad_norm": 1.3872629404067993 + }, + { + "bits_per_byte": 3.134445725728998, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.1726322174072266, + "step": 460, + "unclipped_grad_norm": 1.265127420425415 + }, + { + "bits_per_byte": 3.1533679449075276, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.1857481002807617, + "step": 470, + "unclipped_grad_norm": 0.7593647241592407 + }, + { + "bits_per_byte": 3.1213575020330366, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.163560152053833, + "step": 480, + "unclipped_grad_norm": 1.082176685333252 + }, + { + "bits_per_byte": 3.075586727263012, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.1318342685699463, + "step": 490, + "unclipped_grad_norm": 0.771767258644104 + }, + { + "bits_per_byte": 3.1397366000001417, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.176299571990967, + "step": 500, + "unclipped_grad_norm": 1.4270859956741333 + }, + { + "bits_per_byte": 3.0981456915766867, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.1474709510803223, + "step": 510, + "unclipped_grad_norm": 0.8964738845825195 + }, + { + "bits_per_byte": 2.9721560170688175, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.0601415634155273, + "step": 520, + "unclipped_grad_norm": 1.0101079940795898 + }, + { + "bits_per_byte": 3.0839440521990396, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.137627124786377, + "step": 530, + "unclipped_grad_norm": 1.3795151710510254 + }, + { + "bits_per_byte": 3.029516014749883, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.099900484085083, + "step": 540, + "unclipped_grad_norm": 1.7040444612503052 + }, + { + "bits_per_byte": 2.9807554935772593, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.0661022663116455, + "step": 550, + "unclipped_grad_norm": 0.9698235988616943 + }, + { + "bits_per_byte": 2.9851014951645625, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.0691146850585938, + "step": 560, + "unclipped_grad_norm": 1.2903848886489868 + }, + { + "bits_per_byte": 2.9632115433627444, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.0539417266845703, + "step": 570, + "unclipped_grad_norm": 0.9825631976127625 + }, + { + "bits_per_byte": 2.872915492180244, + "learning_rate": 0.000299626473885547, + "loss_nats": 1.9913532733917236, + "step": 580, + "unclipped_grad_norm": 0.9672371745109558 + }, + { + "bits_per_byte": 2.9178263536959546, + "learning_rate": 0.00029958384005397226, + "loss_nats": 2.0224831104278564, + "step": 590, + "unclipped_grad_norm": 1.0172648429870605 + }, + { + "bits_per_byte": 2.837746071974705, + "learning_rate": 0.0002995389065559004, + "loss_nats": 1.9669756889343262, + "step": 600, + "unclipped_grad_norm": 1.143439531326294 + }, + { + "bits_per_byte": 2.892537680747785, + "learning_rate": 0.000299491674159121, + "loss_nats": 2.0049543380737305, + "step": 610, + "unclipped_grad_norm": 0.8837018013000488 + }, + { + "bits_per_byte": 2.7857125489735113, + "learning_rate": 0.00029944214367070535, + "loss_nats": 1.9309087991714478, + "step": 620, + "unclipped_grad_norm": 1.0084978342056274 + }, + { + "bits_per_byte": 2.79228022244466, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.935461163520813, + "step": 630, + "unclipped_grad_norm": 0.7336203455924988 + }, + { + "bits_per_byte": 2.8026705542965367, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.9426631927490234, + "step": 640, + "unclipped_grad_norm": 1.518286943435669 + }, + { + "bits_per_byte": 2.7257186369944817, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.8893241882324219, + "step": 650, + "unclipped_grad_norm": 1.6367993354797363 + }, + { + "bits_per_byte": 2.719688237324628, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.8851442337036133, + "step": 660, + "unclipped_grad_norm": 1.545776128768921 + }, + { + "bits_per_byte": 2.7049281702796293, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.8749133348464966, + "step": 670, + "unclipped_grad_norm": 0.753908634185791 + }, + { + "bits_per_byte": 2.751855872342008, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.9074411392211914, + "step": 680, + "unclipped_grad_norm": 1.0180727243423462 + }, + { + "bits_per_byte": 2.6299963611946087, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.8229745626449585, + "step": 690, + "unclipped_grad_norm": 0.9542652368545532 + }, + { + "bits_per_byte": 2.560190667001205, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.774588942527771, + "step": 700, + "unclipped_grad_norm": 0.9360936880111694 + }, + { + "bits_per_byte": 2.6536713209297322, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.8393847942352295, + "step": 710, + "unclipped_grad_norm": 1.0009560585021973 + }, + { + "bits_per_byte": 2.7005881880851064, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.8719050884246826, + "step": 720, + "unclipped_grad_norm": 0.8863275647163391 + }, + { + "bits_per_byte": 2.51441972024853, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.7428629398345947, + "step": 730, + "unclipped_grad_norm": 1.0069060325622559 + }, + { + "bits_per_byte": 2.5502567771047184, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.7677032947540283, + "step": 740, + "unclipped_grad_norm": 1.2384967803955078 + }, + { + "bits_per_byte": 2.5553010282545245, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.7711997032165527, + "step": 750, + "unclipped_grad_norm": 1.0719318389892578 + }, + { + "bits_per_byte": 2.530853866417027, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.7542542219161987, + "step": 760, + "unclipped_grad_norm": 1.1390074491500854 + }, + { + "bits_per_byte": 2.5131019891776187, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.7419495582580566, + "step": 770, + "unclipped_grad_norm": 0.9981358647346497 + }, + { + "bits_per_byte": 2.4237852072091313, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.680039882659912, + "step": 780, + "unclipped_grad_norm": 0.7655515670776367 + }, + { + "bits_per_byte": 2.5040773715562437, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.735694169998169, + "step": 790, + "unclipped_grad_norm": 0.8976756930351257 + }, + { + "bits_per_byte": 2.447698706881001, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.69661545753479, + "step": 800, + "unclipped_grad_norm": 0.9077121615409851 + }, + { + "bits_per_byte": 2.496499128028625, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7304413318634033, + "step": 810, + "unclipped_grad_norm": 1.171122431755066 + }, + { + "bits_per_byte": 2.455035142801507, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.7017006874084473, + "step": 820, + "unclipped_grad_norm": 0.9265536665916443 + }, + { + "bits_per_byte": 2.4476441883806777, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.6965776681900024, + "step": 830, + "unclipped_grad_norm": 0.8092235326766968 + }, + { + "bits_per_byte": 2.4251254680073053, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.6809688806533813, + "step": 840, + "unclipped_grad_norm": 0.874322772026062 + }, + { + "bits_per_byte": 2.3621593518560644, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.6373240947723389, + "step": 850, + "unclipped_grad_norm": 0.9783554077148438 + }, + { + "bits_per_byte": 2.4081903363768444, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.669230341911316, + "step": 860, + "unclipped_grad_norm": 1.4636245965957642 + }, + { + "bits_per_byte": 2.3607532217026104, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.6363494396209717, + "step": 870, + "unclipped_grad_norm": 0.9899370074272156 + }, + { + "bits_per_byte": 2.4062409130293143, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.6678791046142578, + "step": 880, + "unclipped_grad_norm": 0.9284471869468689 + }, + { + "bits_per_byte": 2.3647863868479937, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.639145016670227, + "step": 890, + "unclipped_grad_norm": 0.8157685995101929 + }, + { + "bits_per_byte": 2.305498635633407, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.5980498790740967, + "step": 900, + "unclipped_grad_norm": 0.9154189825057983 + }, + { + "bits_per_byte": 2.3490460186930773, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.628234624862671, + "step": 910, + "unclipped_grad_norm": 0.9513880610466003 + }, + { + "bits_per_byte": 2.316342141770307, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.6055660247802734, + "step": 920, + "unclipped_grad_norm": 1.0734350681304932 + }, + { + "bits_per_byte": 2.313089261911888, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.60331130027771, + "step": 930, + "unclipped_grad_norm": 0.9303072690963745 + }, + { + "bits_per_byte": 2.2418796733394912, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.5539525747299194, + "step": 940, + "unclipped_grad_norm": 0.8246778845787048 + }, + { + "bits_per_byte": 2.246466622620653, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.5571320056915283, + "step": 950, + "unclipped_grad_norm": 0.9289151430130005 + }, + { + "bits_per_byte": 2.2606077241051965, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.5669338703155518, + "step": 960, + "unclipped_grad_norm": 1.134368896484375 + }, + { + "bits_per_byte": 2.274281376744695, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5764117240905762, + "step": 970, + "unclipped_grad_norm": 0.9936196804046631 + }, + { + "bits_per_byte": 2.2575570958441924, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.5648193359375, + "step": 980, + "unclipped_grad_norm": 1.3538236618041992 + }, + { + "bits_per_byte": 2.1721451799635445, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.505616307258606, + "step": 990, + "unclipped_grad_norm": 0.8684636950492859 + }, + { + "bits_per_byte": 2.2351432848878456, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.5492832660675049, + "step": 1000, + "unclipped_grad_norm": 0.8402482867240906 + }, + { + "bits_per_byte": 2.22188479836753, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.540093183517456, + "step": 1010, + "unclipped_grad_norm": 0.890442967414856 + }, + { + "bits_per_byte": 2.250056244509462, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.5596201419830322, + "step": 1020, + "unclipped_grad_norm": 0.8663423657417297 + }, + { + "bits_per_byte": 2.1462461405874635, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.4876644611358643, + "step": 1030, + "unclipped_grad_norm": 0.8576785922050476 + }, + { + "bits_per_byte": 2.2370144561292333, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.5505802631378174, + "step": 1040, + "unclipped_grad_norm": 0.7798928618431091 + }, + { + "bits_per_byte": 2.2096559719779303, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5316168069839478, + "step": 1050, + "unclipped_grad_norm": 1.062988042831421 + }, + { + "bits_per_byte": 2.181026536036751, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.5117723941802979, + "step": 1060, + "unclipped_grad_norm": 1.0278452634811401 + }, + { + "bits_per_byte": 2.270514612725498, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.573800802230835, + "step": 1070, + "unclipped_grad_norm": 0.7807672619819641 + }, + { + "bits_per_byte": 2.201701946358176, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.5261034965515137, + "step": 1080, + "unclipped_grad_norm": 0.8375676870346069 + }, + { + "bits_per_byte": 2.109486224826745, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.4621844291687012, + "step": 1090, + "unclipped_grad_norm": 0.8913599848747253 + }, + { + "bits_per_byte": 2.2057005429907366, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5288751125335693, + "step": 1100, + "unclipped_grad_norm": 0.8420668244361877 + }, + { + "bits_per_byte": 2.22141838141839, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.5397698879241943, + "step": 1110, + "unclipped_grad_norm": 0.7918252944946289 + }, + { + "bits_per_byte": 2.1070134582726396, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.46047043800354, + "step": 1120, + "unclipped_grad_norm": 0.8060312271118164 + }, + { + "bits_per_byte": 2.1204196778900464, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.469762921333313, + "step": 1130, + "unclipped_grad_norm": 0.8346309065818787 + }, + { + "bits_per_byte": 2.1395181792857105, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.4830009937286377, + "step": 1140, + "unclipped_grad_norm": 0.8589009642601013 + }, + { + "bits_per_byte": 2.119213907524847, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.4689271450042725, + "step": 1150, + "unclipped_grad_norm": 0.9671322107315063 + }, + { + "bits_per_byte": 2.064344562593577, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.4308946132659912, + "step": 1160, + "unclipped_grad_norm": 0.8266794085502625 + }, + { + "bits_per_byte": 2.186810484567918, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.5157815217971802, + "step": 1170, + "unclipped_grad_norm": 0.8726111054420471 + }, + { + "bits_per_byte": 2.1022884069228347, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.4571952819824219, + "step": 1180, + "unclipped_grad_norm": 1.0611258745193481 + }, + { + "bits_per_byte": 2.105623838453668, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.4595072269439697, + "step": 1190, + "unclipped_grad_norm": 0.8438918590545654 + }, + { + "bits_per_byte": 2.0815397319921423, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.4428133964538574, + "step": 1200, + "unclipped_grad_norm": 0.8341975212097168 + }, + { + "bits_per_byte": 2.217465876136261, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.5370302200317383, + "step": 1210, + "unclipped_grad_norm": 0.9056932330131531 + }, + { + "bits_per_byte": 2.0648071959243977, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4312152862548828, + "step": 1220, + "unclipped_grad_norm": 0.8414756655693054 + }, + { + "bits_per_byte": 1.9950137124992053, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.3828381299972534, + "step": 1230, + "unclipped_grad_norm": 0.728705883026123 + }, + { + "bits_per_byte": 2.021073727636494, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.4009015560150146, + "step": 1240, + "unclipped_grad_norm": 0.918628990650177 + }, + { + "bits_per_byte": 2.1926143830865854, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.5198044776916504, + "step": 1250, + "unclipped_grad_norm": 0.8114829659461975 + }, + { + "bits_per_byte": 2.1120164335862386, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.4639382362365723, + "step": 1260, + "unclipped_grad_norm": 0.8677207827568054 + }, + { + "bits_per_byte": 1.9910613791997271, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.380098581314087, + "step": 1270, + "unclipped_grad_norm": 0.8229067325592041 + }, + { + "bits_per_byte": 2.1358848738035827, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.480482578277588, + "step": 1280, + "unclipped_grad_norm": 0.8881039619445801 + }, + { + "bits_per_byte": 2.0302923416880425, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4072914123535156, + "step": 1290, + "unclipped_grad_norm": 0.7734003067016602 + }, + { + "bits_per_byte": 2.0784213425666977, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.4406518936157227, + "step": 1300, + "unclipped_grad_norm": 0.7850493788719177 + }, + { + "bits_per_byte": 2.1433638833416535, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.4856666326522827, + "step": 1310, + "unclipped_grad_norm": 0.8625432848930359 + }, + { + "bits_per_byte": 2.066748192122035, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.432560682296753, + "step": 1320, + "unclipped_grad_norm": 0.7496529817581177 + }, + { + "bits_per_byte": 1.96163102005348, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.359699010848999, + "step": 1330, + "unclipped_grad_norm": 0.7484292387962341 + }, + { + "bits_per_byte": 1.9905096588557591, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.379716157913208, + "step": 1340, + "unclipped_grad_norm": 0.9107955694198608 + }, + { + "bits_per_byte": 2.162749079816306, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.4991034269332886, + "step": 1350, + "unclipped_grad_norm": 0.8250911235809326 + }, + { + "bits_per_byte": 2.002361499274668, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.387931227684021, + "step": 1360, + "unclipped_grad_norm": 0.7350436449050903 + }, + { + "bits_per_byte": 2.08448338704431, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4448537826538086, + "step": 1370, + "unclipped_grad_norm": 0.8729597330093384 + }, + { + "bits_per_byte": 2.062114291577188, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.4293487071990967, + "step": 1380, + "unclipped_grad_norm": 0.7183053493499756 + }, + { + "bits_per_byte": 2.079455302263685, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.441368579864502, + "step": 1390, + "unclipped_grad_norm": 0.7640445232391357 + }, + { + "bits_per_byte": 2.0571152718645584, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.4258836507797241, + "step": 1400, + "unclipped_grad_norm": 0.7229655385017395 + }, + { + "bits_per_byte": 1.9995009118428668, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.3859484195709229, + "step": 1410, + "unclipped_grad_norm": 0.7203378081321716 + }, + { + "bits_per_byte": 2.0367712721113462, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.4117822647094727, + "step": 1420, + "unclipped_grad_norm": 0.6785973906517029 + }, + { + "bits_per_byte": 2.0482877463610722, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.4197648763656616, + "step": 1430, + "unclipped_grad_norm": 0.8099007606506348 + }, + { + "bits_per_byte": 2.001853462524019, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.387579083442688, + "step": 1440, + "unclipped_grad_norm": 0.753049910068512 + }, + { + "bits_per_byte": 2.051525319763565, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.422008991241455, + "step": 1450, + "unclipped_grad_norm": 0.7571465373039246 + }, + { + "bits_per_byte": 1.988856561615666, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.378570318222046, + "step": 1460, + "unclipped_grad_norm": 0.687399685382843 + }, + { + "bits_per_byte": 1.9988078217598897, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.3854680061340332, + "step": 1470, + "unclipped_grad_norm": 0.7798371911048889 + }, + { + "bits_per_byte": 1.8968069753249683, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.3147664070129395, + "step": 1480, + "unclipped_grad_norm": 0.6712192296981812 + }, + { + "bits_per_byte": 2.0150682654510157, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.396738886833191, + "step": 1490, + "unclipped_grad_norm": 0.670181930065155 + }, + { + "bits_per_byte": 1.9082685871090692, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.3227109909057617, + "step": 1500, + "unclipped_grad_norm": 0.7320581674575806 + }, + { + "bits_per_byte": 1.9830831040262014, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.3745684623718262, + "step": 1510, + "unclipped_grad_norm": 0.759337842464447 + }, + { + "bits_per_byte": 1.9843132959278231, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.375421166419983, + "step": 1520, + "unclipped_grad_norm": 0.8198970556259155 + }, + { + "bits_per_byte": 1.940089161137008, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.3447673320770264, + "step": 1530, + "unclipped_grad_norm": 0.7085598707199097 + }, + { + "bits_per_byte": 2.016095517824619, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3974509239196777, + "step": 1540, + "unclipped_grad_norm": 0.760562539100647 + }, + { + "bits_per_byte": 1.99515422232496, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.3829355239868164, + "step": 1550, + "unclipped_grad_norm": 0.7618942260742188 + }, + { + "bits_per_byte": 2.0757889761125914, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.4388272762298584, + "step": 1560, + "unclipped_grad_norm": 0.8443880081176758 + }, + { + "bits_per_byte": 2.0244234337273492, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.4032233953475952, + "step": 1570, + "unclipped_grad_norm": 0.8402217030525208 + }, + { + "bits_per_byte": 1.9977019733148418, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.3847014904022217, + "step": 1580, + "unclipped_grad_norm": 0.6679587364196777 + }, + { + "bits_per_byte": 1.9955635410340138, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3832192420959473, + "step": 1590, + "unclipped_grad_norm": 0.7231615781784058 + }, + { + "bits_per_byte": 2.002692221912277, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.3881604671478271, + "step": 1600, + "unclipped_grad_norm": 0.745158851146698 + }, + { + "bits_per_byte": 2.015436652289164, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.3969942331314087, + "step": 1610, + "unclipped_grad_norm": 0.7588557004928589 + }, + { + "bits_per_byte": 2.0432919943188095, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.4163020849227905, + "step": 1620, + "unclipped_grad_norm": 0.7977458238601685 + }, + { + "bits_per_byte": 1.9089050948999124, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.3231521844863892, + "step": 1630, + "unclipped_grad_norm": 0.7993338108062744 + }, + { + "bits_per_byte": 2.028824641745579, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.4062740802764893, + "step": 1640, + "unclipped_grad_norm": 0.7733442187309265 + }, + { + "bits_per_byte": 1.9722932564763704, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.3670895099639893, + "step": 1650, + "unclipped_grad_norm": 0.7650595307350159 + }, + { + "bits_per_byte": 1.9473953281109617, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.3498315811157227, + "step": 1660, + "unclipped_grad_norm": 0.7551803588867188 + }, + { + "bits_per_byte": 2.0956505645301338, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.45259428024292, + "step": 1670, + "unclipped_grad_norm": 0.8525713682174683 + }, + { + "bits_per_byte": 2.008782127579329, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.3923816680908203, + "step": 1680, + "unclipped_grad_norm": 0.6863283514976501 + }, + { + "bits_per_byte": 1.8945141026236654, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.3131771087646484, + "step": 1690, + "unclipped_grad_norm": 0.7514925003051758 + }, + { + "bits_per_byte": 1.9367363593584375, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.3424433469772339, + "step": 1700, + "unclipped_grad_norm": 0.7008550763130188 + }, + { + "bits_per_byte": 2.012495404994108, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3949555158615112, + "step": 1710, + "unclipped_grad_norm": 0.7420226335525513 + }, + { + "bits_per_byte": 1.9957871184801352, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.3833742141723633, + "step": 1720, + "unclipped_grad_norm": 0.7137165665626526 + }, + { + "bits_per_byte": 1.8987625900479292, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.3161219358444214, + "step": 1730, + "unclipped_grad_norm": 0.692712128162384 + }, + { + "bits_per_byte": 1.9548323738822222, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.354986548423767, + "step": 1740, + "unclipped_grad_norm": 0.7376338839530945 + }, + { + "bits_per_byte": 2.0575182272155303, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.4261629581451416, + "step": 1750, + "unclipped_grad_norm": 0.7688121199607849 + }, + { + "bits_per_byte": 1.9114107101403328, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.3248889446258545, + "step": 1760, + "unclipped_grad_norm": 0.7555272579193115 + }, + { + "bits_per_byte": 1.9793197796600215, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.371959924697876, + "step": 1770, + "unclipped_grad_norm": 0.7330375909805298 + }, + { + "bits_per_byte": 1.9147368546080195, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.3271944522857666, + "step": 1780, + "unclipped_grad_norm": 0.6358048319816589 + }, + { + "bits_per_byte": 1.8882907384195438, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3088634014129639, + "step": 1790, + "unclipped_grad_norm": 0.7157837152481079 + }, + { + "bits_per_byte": 1.966022081095309, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.3627426624298096, + "step": 1800, + "unclipped_grad_norm": 0.7176512479782104 + }, + { + "bits_per_byte": 1.9491204860817666, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.3510273694992065, + "step": 1810, + "unclipped_grad_norm": 0.7143650054931641 + }, + { + "bits_per_byte": 2.0272141962029, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.4051578044891357, + "step": 1820, + "unclipped_grad_norm": 0.8111060261726379 + }, + { + "bits_per_byte": 1.871968552956758, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.2975497245788574, + "step": 1830, + "unclipped_grad_norm": 0.6767712831497192 + }, + { + "bits_per_byte": 1.9626704831952946, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.360419511795044, + "step": 1840, + "unclipped_grad_norm": 0.6916776299476624 + }, + { + "bits_per_byte": 1.8447792997339039, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.2787035703659058, + "step": 1850, + "unclipped_grad_norm": 0.733634352684021 + }, + { + "bits_per_byte": 2.001037404845675, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.3870134353637695, + "step": 1860, + "unclipped_grad_norm": 0.7013446092605591 + }, + { + "bits_per_byte": 2.016104460922464, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.3974571228027344, + "step": 1870, + "unclipped_grad_norm": 0.7131460309028625 + }, + { + "bits_per_byte": 1.8619130713261114, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.2905797958374023, + "step": 1880, + "unclipped_grad_norm": 0.6984053254127502 + }, + { + "bits_per_byte": 1.9641010348851715, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3614110946655273, + "step": 1890, + "unclipped_grad_norm": 0.7604774832725525 + }, + { + "bits_per_byte": 1.8606962940712568, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.2897363901138306, + "step": 1900, + "unclipped_grad_norm": 0.691594660282135 + }, + { + "bits_per_byte": 1.864379302539484, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.2922892570495605, + "step": 1910, + "unclipped_grad_norm": 0.6990336179733276 + }, + { + "bits_per_byte": 1.9108304406763217, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.3244867324829102, + "step": 1920, + "unclipped_grad_norm": 0.6951581835746765 + }, + { + "bits_per_byte": 1.8916395845971443, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.3111846446990967, + "step": 1930, + "unclipped_grad_norm": 0.6574950814247131 + }, + { + "bits_per_byte": 1.8464982663292777, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.2798950672149658, + "step": 1940, + "unclipped_grad_norm": 0.6538907289505005 + }, + { + "bits_per_byte": 1.9195619998606264, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.3305389881134033, + "step": 1950, + "unclipped_grad_norm": 0.8236934542655945 + }, + { + "bits_per_byte": 1.886723460522231, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.3077770471572876, + "step": 1960, + "unclipped_grad_norm": 0.6981849074363708 + }, + { + "bits_per_byte": 1.843698732738533, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.2779545783996582, + "step": 1970, + "unclipped_grad_norm": 0.660518229007721 + }, + { + "bits_per_byte": 1.8077564225000162, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.2530412673950195, + "step": 1980, + "unclipped_grad_norm": 0.6751270890235901 + }, + { + "bits_per_byte": 1.9189435502481238, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.3301103115081787, + "step": 1990, + "unclipped_grad_norm": 0.6812106370925903 + }, + { + "bits_per_byte": 1.8698170499944642, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.2960584163665771, + "step": 2000, + "unclipped_grad_norm": 0.7189573049545288 + }, + { + "bits_per_byte": 1.938266145037862, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.343503713607788, + "step": 2010, + "unclipped_grad_norm": 0.7090774178504944 + }, + { + "bits_per_byte": 1.9095190729634928, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.3235777616500854, + "step": 2020, + "unclipped_grad_norm": 0.6950410604476929 + }, + { + "bits_per_byte": 1.9133738920999317, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.3262497186660767, + "step": 2030, + "unclipped_grad_norm": 0.6876662373542786 + }, + { + "bits_per_byte": 1.9269355840337186, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.3356499671936035, + "step": 2040, + "unclipped_grad_norm": 0.6909147500991821 + }, + { + "bits_per_byte": 1.8696724125850885, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.295958161354065, + "step": 2050, + "unclipped_grad_norm": 0.7017654180526733 + }, + { + "bits_per_byte": 1.7694662051119205, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.2265005111694336, + "step": 2060, + "unclipped_grad_norm": 0.6528151631355286 + }, + { + "bits_per_byte": 1.8652369800193367, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.2928837537765503, + "step": 2070, + "unclipped_grad_norm": 0.6662172675132751 + }, + { + "bits_per_byte": 1.9804285518101339, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.3727284669876099, + "step": 2080, + "unclipped_grad_norm": 0.7636837959289551 + }, + { + "bits_per_byte": 1.833713592012089, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.2710334062576294, + "step": 2090, + "unclipped_grad_norm": 0.6732650995254517 + }, + { + "bits_per_byte": 1.9135582575016568, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.326377511024475, + "step": 2100, + "unclipped_grad_norm": 0.6848331093788147 + }, + { + "bits_per_byte": 1.844285537543277, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.2783613204956055, + "step": 2110, + "unclipped_grad_norm": 0.7649273872375488 + }, + { + "bits_per_byte": 1.8619990626515426, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.2906394004821777, + "step": 2120, + "unclipped_grad_norm": 0.745818018913269 + }, + { + "bits_per_byte": 1.8897116590809722, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.3098483085632324, + "step": 2130, + "unclipped_grad_norm": 0.6699572801589966 + }, + { + "bits_per_byte": 1.7852617796977643, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.2374491691589355, + "step": 2140, + "unclipped_grad_norm": 0.6564173102378845 + }, + { + "bits_per_byte": 1.8460359769637584, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2795746326446533, + "step": 2150, + "unclipped_grad_norm": 0.7896565198898315 + }, + { + "bits_per_byte": 1.89216499159553, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.3115488290786743, + "step": 2160, + "unclipped_grad_norm": 0.6690707802772522 + }, + { + "bits_per_byte": 1.859092899817263, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.2886250019073486, + "step": 2170, + "unclipped_grad_norm": 0.6643791198730469 + }, + { + "bits_per_byte": 1.786192549804234, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.2380943298339844, + "step": 2180, + "unclipped_grad_norm": 0.6800501346588135 + }, + { + "bits_per_byte": 1.9116076302755707, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.3250254392623901, + "step": 2190, + "unclipped_grad_norm": 0.7342450022697449 + }, + { + "bits_per_byte": 1.942128359428288, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.34618079662323, + "step": 2200, + "unclipped_grad_norm": 0.6729614734649658 + }, + { + "bits_per_byte": 1.9734279980067633, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.3678760528564453, + "step": 2210, + "unclipped_grad_norm": 0.7817486524581909 + }, + { + "bits_per_byte": 1.8626522527595197, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2910921573638916, + "step": 2220, + "unclipped_grad_norm": 0.7089713215827942 + }, + { + "bits_per_byte": 1.8598472157239472, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.2891478538513184, + "step": 2230, + "unclipped_grad_norm": 0.6630467176437378 + }, + { + "bits_per_byte": 1.8348901253266416, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.2718489170074463, + "step": 2240, + "unclipped_grad_norm": 0.6948215365409851 + }, + { + "bits_per_byte": 1.8364145795438895, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2729055881500244, + "step": 2250, + "unclipped_grad_norm": 0.8220188617706299 + }, + { + "bits_per_byte": 1.868465954289286, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.2951219081878662, + "step": 2260, + "unclipped_grad_norm": 0.6416627168655396 + }, + { + "bits_per_byte": 1.9114788152700743, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.3249361515045166, + "step": 2270, + "unclipped_grad_norm": 0.7394387722015381 + }, + { + "bits_per_byte": 1.8000844484276772, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.2477234601974487, + "step": 2280, + "unclipped_grad_norm": 0.6806369423866272 + }, + { + "bits_per_byte": 1.8230345012894138, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.2636312246322632, + "step": 2290, + "unclipped_grad_norm": 0.6535201072692871 + }, + { + "bits_per_byte": 1.8732329694059013, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.2984261512756348, + "step": 2300, + "unclipped_grad_norm": 0.6698590517044067 + }, + { + "bits_per_byte": 1.8449220453341202, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.278802514076233, + "step": 2310, + "unclipped_grad_norm": 0.7061722278594971 + }, + { + "bits_per_byte": 1.8437668378682746, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.2780017852783203, + "step": 2320, + "unclipped_grad_norm": 0.6518182754516602 + }, + { + "bits_per_byte": 1.793533457273662, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.24318265914917, + "step": 2330, + "unclipped_grad_norm": 0.6269276142120361 + }, + { + "bits_per_byte": 1.8247620670173308, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.2648286819458008, + "step": 2340, + "unclipped_grad_norm": 0.6368834376335144 + }, + { + "bits_per_byte": 1.7578133486200602, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2184233665466309, + "step": 2350, + "unclipped_grad_norm": 0.6873382925987244 + }, + { + "bits_per_byte": 1.8378069510852748, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.2738707065582275, + "step": 2360, + "unclipped_grad_norm": 0.6633060574531555 + }, + { + "bits_per_byte": 1.8742523105775653, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.2991327047348022, + "step": 2370, + "unclipped_grad_norm": 0.670342743396759 + }, + { + "bits_per_byte": 1.7763348482220793, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.2312614917755127, + "step": 2380, + "unclipped_grad_norm": 0.6927285194396973 + }, + { + "bits_per_byte": 1.7744062347753038, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.2299246788024902, + "step": 2390, + "unclipped_grad_norm": 0.6405621767044067 + }, + { + "bits_per_byte": 1.869930214578732, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.2961368560791016, + "step": 2400, + "unclipped_grad_norm": 0.6600690484046936 + }, + { + "bits_per_byte": 1.8480951252425388, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2810019254684448, + "step": 2410, + "unclipped_grad_norm": 0.7039197087287903 + }, + { + "bits_per_byte": 1.9394289197403456, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.344309687614441, + "step": 2420, + "unclipped_grad_norm": 0.7103801965713501 + }, + { + "bits_per_byte": 1.829258381441488, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.2679452896118164, + "step": 2430, + "unclipped_grad_norm": 0.667262852191925 + }, + { + "bits_per_byte": 1.7774200587490236, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.2320137023925781, + "step": 2440, + "unclipped_grad_norm": 0.665513277053833 + }, + { + "bits_per_byte": 1.8706994929760412, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.2966700792312622, + "step": 2450, + "unclipped_grad_norm": 0.6866514682769775 + }, + { + "bits_per_byte": 1.8720627994494308, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.2976150512695312, + "step": 2460, + "unclipped_grad_norm": 0.6790883541107178 + }, + { + "bits_per_byte": 1.8135783791970241, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.2570767402648926, + "step": 2470, + "unclipped_grad_norm": 0.611331582069397 + }, + { + "bits_per_byte": 1.8235925849914638, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.2640180587768555, + "step": 2480, + "unclipped_grad_norm": 0.6766824126243591 + }, + { + "bits_per_byte": 1.7500428284887768, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.2130372524261475, + "step": 2490, + "unclipped_grad_norm": 0.6680052876472473 + }, + { + "bits_per_byte": 1.8286870550753218, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2675492763519287, + "step": 2500, + "unclipped_grad_norm": 0.6833259463310242 + }, + { + "bits_per_byte": 1.7979694057873665, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.2462574243545532, + "step": 2510, + "unclipped_grad_norm": 0.6785508990287781 + }, + { + "bits_per_byte": 1.7817966732481805, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.2350473403930664, + "step": 2520, + "unclipped_grad_norm": 0.6775974035263062 + }, + { + "bits_per_byte": 1.7839853244630606, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.2365643978118896, + "step": 2530, + "unclipped_grad_norm": 0.6275198459625244 + }, + { + "bits_per_byte": 1.805110469416492, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.2512072324752808, + "step": 2540, + "unclipped_grad_norm": 0.690688967704773 + }, + { + "bits_per_byte": 1.8468862591896242, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.2801640033721924, + "step": 2550, + "unclipped_grad_norm": 0.66722571849823 + }, + { + "bits_per_byte": 1.7913382707180492, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.2416610717773438, + "step": 2560, + "unclipped_grad_norm": 0.6575818061828613 + }, + { + "bits_per_byte": 1.7826098072214598, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2356109619140625, + "step": 2570, + "unclipped_grad_norm": 0.7133845686912537 + }, + { + "bits_per_byte": 1.8245568917148516, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.2646864652633667, + "step": 2580, + "unclipped_grad_norm": 0.6782506704330444 + }, + { + "bits_per_byte": 1.7816253785279212, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.2349286079406738, + "step": 2590, + "unclipped_grad_norm": 0.6414566040039062 + }, + { + "bits_per_byte": 1.8190807921287289, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.2608907222747803, + "step": 2600, + "unclipped_grad_norm": 0.6193698048591614 + }, + { + "bits_per_byte": 1.7289572395450439, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.198421835899353, + "step": 2610, + "unclipped_grad_norm": 0.6259768605232239 + }, + { + "bits_per_byte": 1.8014212695728338, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.248650074005127, + "step": 2620, + "unclipped_grad_norm": 0.6647894382476807 + }, + { + "bits_per_byte": 1.8030841698240263, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.2498027086257935, + "step": 2630, + "unclipped_grad_norm": 0.6602649688720703 + }, + { + "bits_per_byte": 1.8178046808593267, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.2600061893463135, + "step": 2640, + "unclipped_grad_norm": 0.6987072229385376 + }, + { + "bits_per_byte": 1.745824094063112, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2101130485534668, + "step": 2650, + "unclipped_grad_norm": 0.5887352824211121 + }, + { + "bits_per_byte": 1.7468853990015865, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.2108486890792847, + "step": 2660, + "unclipped_grad_norm": 0.6428946256637573 + }, + { + "bits_per_byte": 1.796217246540376, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.2450429201126099, + "step": 2670, + "unclipped_grad_norm": 0.7706190347671509 + }, + { + "bits_per_byte": 1.7693616396601959, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.2264280319213867, + "step": 2680, + "unclipped_grad_norm": 0.6362439393997192 + }, + { + "bits_per_byte": 1.848741951992434, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.2814502716064453, + "step": 2690, + "unclipped_grad_norm": 0.6491434574127197 + }, + { + "bits_per_byte": 1.8400919125846382, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.2754545211791992, + "step": 2700, + "unclipped_grad_norm": 0.7701624035835266 + }, + { + "bits_per_byte": 1.712789322520082, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.1872150897979736, + "step": 2710, + "unclipped_grad_norm": 0.6629177927970886 + }, + { + "bits_per_byte": 1.8192957704423074, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.2610397338867188, + "step": 2720, + "unclipped_grad_norm": 0.646001935005188 + }, + { + "bits_per_byte": 1.7383142996305365, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.2049076557159424, + "step": 2730, + "unclipped_grad_norm": 0.5994110107421875 + }, + { + "bits_per_byte": 1.7750071421574185, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.2303411960601807, + "step": 2740, + "unclipped_grad_norm": 0.6481388807296753 + }, + { + "bits_per_byte": 1.7664549608779636, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.224413275718689, + "step": 2750, + "unclipped_grad_norm": 0.6262131929397583 + }, + { + "bits_per_byte": 1.8794697482567901, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.3027491569519043, + "step": 2760, + "unclipped_grad_norm": 0.7271934151649475 + }, + { + "bits_per_byte": 1.7847640619061673, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.2371041774749756, + "step": 2770, + "unclipped_grad_norm": 0.658359944820404 + }, + { + "bits_per_byte": 1.7813694683434373, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.2347512245178223, + "step": 2780, + "unclipped_grad_norm": 0.6573207378387451 + }, + { + "bits_per_byte": 1.7738705088178663, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2295533418655396, + "step": 2790, + "unclipped_grad_norm": 0.6746286153793335 + }, + { + "bits_per_byte": 1.7278494992908364, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.1976540088653564, + "step": 2800, + "unclipped_grad_norm": 0.6247284412384033 + }, + { + "bits_per_byte": 1.7930410709442417, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.242841362953186, + "step": 2810, + "unclipped_grad_norm": 0.6581431031227112 + }, + { + "bits_per_byte": 1.7915061257852913, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2417774200439453, + "step": 2820, + "unclipped_grad_norm": 0.6571568846702576 + }, + { + "bits_per_byte": 1.726584050945788, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.1967768669128418, + "step": 2830, + "unclipped_grad_norm": 0.6243956089019775 + }, + { + "bits_per_byte": 1.8094150231849373, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.2541909217834473, + "step": 2840, + "unclipped_grad_norm": 0.6933325529098511 + }, + { + "bits_per_byte": 1.726325733004192, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.1965978145599365, + "step": 2850, + "unclipped_grad_norm": 0.634556233882904 + }, + { + "bits_per_byte": 1.6987789278501941, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.1775038242340088, + "step": 2860, + "unclipped_grad_norm": 0.6379566192626953 + }, + { + "bits_per_byte": 1.8208350151675294, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.2621066570281982, + "step": 2870, + "unclipped_grad_norm": 0.63397616147995 + }, + { + "bits_per_byte": 1.7343494115975453, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2021594047546387, + "step": 2880, + "unclipped_grad_norm": 0.6400126814842224 + }, + { + "bits_per_byte": 1.7925146320499508, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.242476463317871, + "step": 2890, + "unclipped_grad_norm": 0.6657673716545105 + }, + { + "bits_per_byte": 1.7042804808686445, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.1813172101974487, + "step": 2900, + "unclipped_grad_norm": 0.6459061503410339 + }, + { + "bits_per_byte": 1.7322821801341743, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.2007265090942383, + "step": 2910, + "unclipped_grad_norm": 0.6347489356994629 + }, + { + "bits_per_byte": 1.8285410418047392, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.2674480676651, + "step": 2920, + "unclipped_grad_norm": 0.6735997796058655 + }, + { + "bits_per_byte": 1.768944065783901, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.2261385917663574, + "step": 2930, + "unclipped_grad_norm": 0.6750695705413818 + }, + { + "bits_per_byte": 1.8169547425987627, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.2594170570373535, + "step": 2940, + "unclipped_grad_norm": 0.6986971497535706 + }, + { + "bits_per_byte": 1.7661269899627683, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2241859436035156, + "step": 2950, + "unclipped_grad_norm": 0.6931433081626892 + }, + { + "bits_per_byte": 1.7494049448367266, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.2125951051712036, + "step": 2960, + "unclipped_grad_norm": 0.6624579429626465 + }, + { + "bits_per_byte": 1.803795662050646, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.250295877456665, + "step": 2970, + "unclipped_grad_norm": 0.6687899231910706 + }, + { + "bits_per_byte": 1.739176276676661, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.205505132675171, + "step": 2980, + "unclipped_grad_norm": 0.6366226077079773 + }, + { + "bits_per_byte": 1.8637783951573692, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.2918727397918701, + "step": 2990, + "unclipped_grad_norm": 0.6679046750068665 + }, + { + "bits_per_byte": 1.7949330520863835, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.2441527843475342, + "step": 3000, + "unclipped_grad_norm": 0.6487835049629211 + }, + { + "bits_per_byte": 1.7636583509922836, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.2224748134613037, + "step": 3010, + "unclipped_grad_norm": 0.6533840298652649 + }, + { + "bits_per_byte": 1.7288860387275866, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.198372483253479, + "step": 3020, + "unclipped_grad_norm": 0.6559380292892456 + }, + { + "bits_per_byte": 1.7617252659965856, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.221134901046753, + "step": 3030, + "unclipped_grad_norm": 0.6586632132530212 + }, + { + "bits_per_byte": 1.736313109505097, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.2035205364227295, + "step": 3040, + "unclipped_grad_norm": 0.6175543069839478 + }, + { + "bits_per_byte": 1.7495843227415764, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.212719440460205, + "step": 3050, + "unclipped_grad_norm": 0.6556141972541809 + }, + { + "bits_per_byte": 1.7508618098721853, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.2136049270629883, + "step": 3060, + "unclipped_grad_norm": 0.6971888542175293 + }, + { + "bits_per_byte": 1.7287235151225213, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.1982598304748535, + "step": 3070, + "unclipped_grad_norm": 0.6718002557754517 + }, + { + "bits_per_byte": 1.7367391105312842, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.2038158178329468, + "step": 3080, + "unclipped_grad_norm": 0.65813809633255 + }, + { + "bits_per_byte": 1.7027751167256424, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.1802737712860107, + "step": 3090, + "unclipped_grad_norm": 0.6299879550933838 + }, + { + "bits_per_byte": 1.8074814222412867, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.2528506517410278, + "step": 3100, + "unclipped_grad_norm": 0.6740135550498962 + }, + { + "bits_per_byte": 1.7726105639176453, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.2286800146102905, + "step": 3110, + "unclipped_grad_norm": 0.6835277676582336 + }, + { + "bits_per_byte": 1.6626993754562889, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.1524953842163086, + "step": 3120, + "unclipped_grad_norm": 0.6420844793319702 + }, + { + "bits_per_byte": 1.767056384208031, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.224830150604248, + "step": 3130, + "unclipped_grad_norm": 0.6585980653762817 + }, + { + "bits_per_byte": 1.7119149627230954, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.1866090297698975, + "step": 3140, + "unclipped_grad_norm": 0.640397846698761 + }, + { + "bits_per_byte": 1.7056073270200511, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.182236909866333, + "step": 3150, + "unclipped_grad_norm": 0.6397213935852051 + }, + { + "bits_per_byte": 1.7536537762262923, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.2155401706695557, + "step": 3160, + "unclipped_grad_norm": 0.6579908728599548 + }, + { + "bits_per_byte": 1.790991037745957, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.2414203882217407, + "step": 3170, + "unclipped_grad_norm": 0.6711400747299194 + }, + { + "bits_per_byte": 1.6755903350517107, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.1614307165145874, + "step": 3180, + "unclipped_grad_norm": 0.6659626364707947 + }, + { + "bits_per_byte": 1.7605863968825721, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.2203454971313477, + "step": 3190, + "unclipped_grad_norm": 0.6405898332595825 + }, + { + "bits_per_byte": 1.6750425703087128, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.1610510349273682, + "step": 3200, + "unclipped_grad_norm": 0.6211809515953064 + }, + { + "bits_per_byte": 1.6974529416120419, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.1765847206115723, + "step": 3210, + "unclipped_grad_norm": 0.6501805186271667 + }, + { + "bits_per_byte": 1.7106600053197496, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.1857391595840454, + "step": 3220, + "unclipped_grad_norm": 0.615857720375061 + }, + { + "bits_per_byte": 1.6706074817082626, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.1579768657684326, + "step": 3230, + "unclipped_grad_norm": 0.6441828608512878 + }, + { + "bits_per_byte": 1.743054829418919, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.2081935405731201, + "step": 3240, + "unclipped_grad_norm": 0.6937673091888428 + }, + { + "bits_per_byte": 1.7373006338863513, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.20420503616333, + "step": 3250, + "unclipped_grad_norm": 0.6254703998565674 + }, + { + "bits_per_byte": 1.8373800901458333, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.2735748291015625, + "step": 3260, + "unclipped_grad_norm": 0.6900506615638733 + }, + { + "bits_per_byte": 1.6942182919146143, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.1743426322937012, + "step": 3270, + "unclipped_grad_norm": 0.6274329423904419 + }, + { + "bits_per_byte": 1.6260650070307003, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.1271023750305176, + "step": 3280, + "unclipped_grad_norm": 0.616184651851654 + }, + { + "bits_per_byte": 1.67383009262013, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.1602106094360352, + "step": 3290, + "unclipped_grad_norm": 0.621311366558075 + }, + { + "bits_per_byte": 1.7588158354919394, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.2191182374954224, + "step": 3300, + "unclipped_grad_norm": 0.6610729694366455 + }, + { + "bits_per_byte": 1.7486831336510553, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.212094783782959, + "step": 3310, + "unclipped_grad_norm": 0.6942672729492188 + }, + { + "bits_per_byte": 1.7587967454176936, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.2191050052642822, + "step": 3320, + "unclipped_grad_norm": 0.671239972114563 + }, + { + "bits_per_byte": 1.6992685624572006, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.1778432130813599, + "step": 3330, + "unclipped_grad_norm": 0.6172157526016235 + }, + { + "bits_per_byte": 1.7238319845666812, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.1948692798614502, + "step": 3340, + "unclipped_grad_norm": 0.6998726725578308 + }, + { + "bits_per_byte": 1.7052557944816875, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.1819932460784912, + "step": 3350, + "unclipped_grad_norm": 0.6467944383621216 + }, + { + "bits_per_byte": 1.7057184278125084, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.1823139190673828, + "step": 3360, + "unclipped_grad_norm": 0.6332494020462036 + }, + { + "bits_per_byte": 1.6447658845375697, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.1400648355484009, + "step": 3370, + "unclipped_grad_norm": 0.6481251120567322 + }, + { + "bits_per_byte": 1.7522101538549497, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.2145395278930664, + "step": 3380, + "unclipped_grad_norm": 0.7249065041542053 + }, + { + "bits_per_byte": 1.7176927198788314, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.190613865852356, + "step": 3390, + "unclipped_grad_norm": 0.6741031408309937 + }, + { + "bits_per_byte": 1.7259690409863024, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.1963505744934082, + "step": 3400, + "unclipped_grad_norm": 0.6826781034469604 + }, + { + "bits_per_byte": 1.7480068978678627, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.2116260528564453, + "step": 3410, + "unclipped_grad_norm": 0.6849276423454285 + }, + { + "bits_per_byte": 1.724440115220132, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.1952908039093018, + "step": 3420, + "unclipped_grad_norm": 0.6456359028816223 + }, + { + "bits_per_byte": 1.6718536679964147, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1588406562805176, + "step": 3430, + "unclipped_grad_norm": 0.6529037356376648 + }, + { + "bits_per_byte": 1.678087179176937, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.1631613969802856, + "step": 3440, + "unclipped_grad_norm": 0.6204179525375366 + }, + { + "bits_per_byte": 1.6687005380754958, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.1566550731658936, + "step": 3450, + "unclipped_grad_norm": 0.7193098068237305 + }, + { + "bits_per_byte": 1.6579879107359023, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.149229645729065, + "step": 3460, + "unclipped_grad_norm": 0.6362242102622986 + }, + { + "bits_per_byte": 1.7239448051856472, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.1949474811553955, + "step": 3470, + "unclipped_grad_norm": 0.6910108327865601 + }, + { + "bits_per_byte": 1.7796119776342703, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.2335330247879028, + "step": 3480, + "unclipped_grad_norm": 0.7018831968307495 + }, + { + "bits_per_byte": 1.7076085171454907, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.183624029159546, + "step": 3490, + "unclipped_grad_norm": 0.6321303248405457 + }, + { + "bits_per_byte": 1.663780114434311, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1532444953918457, + "step": 3500, + "unclipped_grad_norm": 0.6451124548912048 + }, + { + "bits_per_byte": 1.6715255250985683, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.1586132049560547, + "step": 3510, + "unclipped_grad_norm": 0.6249452233314514 + }, + { + "bits_per_byte": 1.7161655139391698, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.189555287361145, + "step": 3520, + "unclipped_grad_norm": 0.6655569672584534 + }, + { + "bits_per_byte": 1.6822832118926878, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.1660698652267456, + "step": 3530, + "unclipped_grad_norm": 0.6392773389816284 + }, + { + "bits_per_byte": 1.684036918983536, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.167285442352295, + "step": 3540, + "unclipped_grad_norm": 0.673799991607666 + }, + { + "bits_per_byte": 1.733227740748618, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.2013819217681885, + "step": 3550, + "unclipped_grad_norm": 0.6935567259788513 + }, + { + "bits_per_byte": 1.66402708152095, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.1534156799316406, + "step": 3560, + "unclipped_grad_norm": 0.6767885088920593 + }, + { + "bits_per_byte": 1.6966515024590212, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.1760292053222656, + "step": 3570, + "unclipped_grad_norm": 0.6852818131446838 + }, + { + "bits_per_byte": 1.7308131043305044, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.1997082233428955, + "step": 3580, + "unclipped_grad_norm": 0.6906432509422302 + }, + { + "bits_per_byte": 1.7106285324946418, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.1857173442840576, + "step": 3590, + "unclipped_grad_norm": 0.6694921851158142 + }, + { + "bits_per_byte": 1.7506702211991243, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.2134721279144287, + "step": 3600, + "unclipped_grad_norm": 0.7073791027069092 + }, + { + "bits_per_byte": 1.7457346630846633, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.2100510597229004, + "step": 3610, + "unclipped_grad_norm": 0.6925025582313538 + }, + { + "bits_per_byte": 1.682850926623186, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.1664633750915527, + "step": 3620, + "unclipped_grad_norm": 0.655085027217865 + }, + { + "bits_per_byte": 1.6642234857082352, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.1535518169403076, + "step": 3630, + "unclipped_grad_norm": 0.6682478785514832 + }, + { + "bits_per_byte": 1.6081953216754512, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.1147160530090332, + "step": 3640, + "unclipped_grad_norm": 0.6062859892845154 + }, + { + "bits_per_byte": 1.6879799652198675, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.1700185537338257, + "step": 3650, + "unclipped_grad_norm": 0.6663388013839722 + }, + { + "bits_per_byte": 1.686169159888933, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.1687633991241455, + "step": 3660, + "unclipped_grad_norm": 0.6170307397842407 + }, + { + "bits_per_byte": 1.7251712134689499, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.1957975625991821, + "step": 3670, + "unclipped_grad_norm": 0.7081273794174194 + }, + { + "bits_per_byte": 1.65707451087717, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.1485965251922607, + "step": 3680, + "unclipped_grad_norm": 0.6444666981697083 + }, + { + "bits_per_byte": 1.7409618005579186, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.206742763519287, + "step": 3690, + "unclipped_grad_norm": 0.6806919574737549 + }, + { + "bits_per_byte": 1.6311708279695154, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.1306414604187012, + "step": 3700, + "unclipped_grad_norm": 0.6441910862922668 + }, + { + "bits_per_byte": 1.6893040596488602, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.1709363460540771, + "step": 3710, + "unclipped_grad_norm": 0.6544809341430664 + }, + { + "bits_per_byte": 1.7198873904864918, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.1921350955963135, + "step": 3720, + "unclipped_grad_norm": 0.7107911109924316 + }, + { + "bits_per_byte": 1.6843559468008864, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.1675065755844116, + "step": 3730, + "unclipped_grad_norm": 0.6489394307136536 + }, + { + "bits_per_byte": 1.6647236112569441, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.1538984775543213, + "step": 3740, + "unclipped_grad_norm": 0.6457509994506836 + }, + { + "bits_per_byte": 1.6973611028764812, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1765210628509521, + "step": 3750, + "unclipped_grad_norm": 0.6853610277175903 + }, + { + "bits_per_byte": 1.724518023360973, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.1953448057174683, + "step": 3760, + "unclipped_grad_norm": 0.6944870352745056 + }, + { + "bits_per_byte": 1.7199413930388627, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.1921725273132324, + "step": 3770, + "unclipped_grad_norm": 0.640734851360321 + }, + { + "bits_per_byte": 1.7648990338356079, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.223334789276123, + "step": 3780, + "unclipped_grad_norm": 0.7070465683937073 + }, + { + "bits_per_byte": 1.6556283087660646, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.1475940942764282, + "step": 3790, + "unclipped_grad_norm": 0.6397157907485962 + }, + { + "bits_per_byte": 1.6783037053343732, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.16331148147583, + "step": 3800, + "unclipped_grad_norm": 0.6925249695777893 + }, + { + "bits_per_byte": 1.6280104747772606, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.128450870513916, + "step": 3810, + "unclipped_grad_norm": 0.6427530646324158 + }, + { + "bits_per_byte": 1.6365965366389352, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1344022750854492, + "step": 3820, + "unclipped_grad_norm": 0.6470082402229309 + }, + { + "bits_per_byte": 1.7099467932666215, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.1852447986602783, + "step": 3830, + "unclipped_grad_norm": 0.6749695539474487 + }, + { + "bits_per_byte": 1.6337252862827807, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.132412075996399, + "step": 3840, + "unclipped_grad_norm": 0.650758683681488 + }, + { + "bits_per_byte": 1.6059385653308293, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1131517887115479, + "step": 3850, + "unclipped_grad_norm": 0.6182020306587219 + }, + { + "bits_per_byte": 1.6037168934469836, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.1116118431091309, + "step": 3860, + "unclipped_grad_norm": 0.6587263345718384 + }, + { + "bits_per_byte": 1.569579713111923, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.0879497528076172, + "step": 3870, + "unclipped_grad_norm": 0.6396462321281433 + }, + { + "bits_per_byte": 1.6644050993875463, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1536777019500732, + "step": 3880, + "unclipped_grad_norm": 0.6618650555610657 + }, + { + "bits_per_byte": 1.6309677164588465, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.1305006742477417, + "step": 3890, + "unclipped_grad_norm": 0.6646989583969116 + }, + { + "bits_per_byte": 1.6357273363214746, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.1337997913360596, + "step": 3900, + "unclipped_grad_norm": 0.6604899168014526 + }, + { + "bits_per_byte": 1.6545293396270513, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1468323469161987, + "step": 3910, + "unclipped_grad_norm": 0.6787318587303162 + }, + { + "bits_per_byte": 1.6878260407473453, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.1699118614196777, + "step": 3920, + "unclipped_grad_norm": 0.7052112221717834 + }, + { + "bits_per_byte": 1.6817977048693022, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.1657333374023438, + "step": 3930, + "unclipped_grad_norm": 0.6520130634307861 + }, + { + "bits_per_byte": 1.6179482858232301, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.1214762926101685, + "step": 3940, + "unclipped_grad_norm": 0.6605469584465027 + }, + { + "bits_per_byte": 1.7173200334744119, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1903555393218994, + "step": 3950, + "unclipped_grad_norm": 0.708786129951477 + }, + { + "bits_per_byte": 1.699606508366146, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.1780774593353271, + "step": 3960, + "unclipped_grad_norm": 0.6629562973976135 + }, + { + "bits_per_byte": 1.6391136747169635, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1361470222473145, + "step": 3970, + "unclipped_grad_norm": 0.6665799617767334 + }, + { + "bits_per_byte": 1.631017075479644, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.1305348873138428, + "step": 3980, + "unclipped_grad_norm": 0.6867449879646301 + }, + { + "bits_per_byte": 1.6814138395925764, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1654672622680664, + "step": 3990, + "unclipped_grad_norm": 0.6741853356361389 + }, + { + "bits_per_byte": 1.5998187346825272, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.1089098453521729, + "step": 4000, + "unclipped_grad_norm": 0.6545262932777405 + }, + { + "bits_per_byte": 1.663637024868793, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.1531453132629395, + "step": 4010, + "unclipped_grad_norm": 0.690457820892334 + }, + { + "bits_per_byte": 1.6831018493107948, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.1666373014450073, + "step": 4020, + "unclipped_grad_norm": 0.7107905149459839 + }, + { + "bits_per_byte": 1.6391102350639462, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.1361446380615234, + "step": 4030, + "unclipped_grad_norm": 0.7038000822067261 + }, + { + "bits_per_byte": 1.6087578769264235, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1151059865951538, + "step": 4040, + "unclipped_grad_norm": 0.65596604347229 + }, + { + "bits_per_byte": 1.6500808363798338, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.1437488794326782, + "step": 4050, + "unclipped_grad_norm": 0.6746683716773987 + }, + { + "bits_per_byte": 1.6171470186528603, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1209208965301514, + "step": 4060, + "unclipped_grad_norm": 0.6491661071777344 + }, + { + "bits_per_byte": 1.768091891748876, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.2255479097366333, + "step": 4070, + "unclipped_grad_norm": 0.7306786775588989 + }, + { + "bits_per_byte": 1.5881576230234253, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.1008269786834717, + "step": 4080, + "unclipped_grad_norm": 0.6624433994293213 + }, + { + "bits_per_byte": 1.6281174479860971, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.1285250186920166, + "step": 4090, + "unclipped_grad_norm": 0.7040045857429504 + }, + { + "bits_per_byte": 1.7322529430835278, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.2007062435150146, + "step": 4100, + "unclipped_grad_norm": 0.709124743938446 + }, + { + "bits_per_byte": 1.6320334929462432, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.131239414215088, + "step": 4110, + "unclipped_grad_norm": 0.6646102070808411 + }, + { + "bits_per_byte": 1.6337092918962506, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.1324009895324707, + "step": 4120, + "unclipped_grad_norm": 0.6718649864196777 + }, + { + "bits_per_byte": 1.6921957759404676, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.172940731048584, + "step": 4130, + "unclipped_grad_norm": 0.7302323579788208 + }, + { + "bits_per_byte": 1.6807929822229617, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.165036916732788, + "step": 4140, + "unclipped_grad_norm": 0.6884070634841919 + }, + { + "bits_per_byte": 1.6190756320996358, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1222577095031738, + "step": 4150, + "unclipped_grad_norm": 0.6662921905517578 + }, + { + "bits_per_byte": 1.6461508608249678, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.1410248279571533, + "step": 4160, + "unclipped_grad_norm": 0.7144712209701538 + }, + { + "bits_per_byte": 1.6659785686602901, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.1547683477401733, + "step": 4170, + "unclipped_grad_norm": 0.7300424575805664 + }, + { + "bits_per_byte": 1.6421422891986575, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.1382462978363037, + "step": 4180, + "unclipped_grad_norm": 0.6750319600105286 + }, + { + "bits_per_byte": 1.670730965251582, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.15806245803833, + "step": 4190, + "unclipped_grad_norm": 0.7004430890083313 + }, + { + "bits_per_byte": 1.6517516478329661, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.144906997680664, + "step": 4200, + "unclipped_grad_norm": 0.7251765131950378 + }, + { + "bits_per_byte": 1.6842140611139247, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.1674082279205322, + "step": 4210, + "unclipped_grad_norm": 0.695562481880188 + }, + { + "bits_per_byte": 1.6489540060513808, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.1429678201675415, + "step": 4220, + "unclipped_grad_norm": 0.6624053716659546 + }, + { + "bits_per_byte": 1.6461546444432869, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.1410274505615234, + "step": 4230, + "unclipped_grad_norm": 0.6809643507003784 + }, + { + "bits_per_byte": 1.6662085094644936, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.1549277305603027, + "step": 4240, + "unclipped_grad_norm": 0.6776161789894104 + }, + { + "bits_per_byte": 1.6127201852196515, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1178524494171143, + "step": 4250, + "unclipped_grad_norm": 0.6506413221359253 + }, + { + "bits_per_byte": 1.6413755905411112, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1377148628234863, + "step": 4260, + "unclipped_grad_norm": 0.6575179696083069 + }, + { + "bits_per_byte": 1.5658554288074888, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.085368275642395, + "step": 4270, + "unclipped_grad_norm": 0.6578490734100342 + }, + { + "bits_per_byte": 1.6325695628689827, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1316109895706177, + "step": 4280, + "unclipped_grad_norm": 0.6765334010124207 + }, + { + "bits_per_byte": 1.608131516111981, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.1146718263626099, + "step": 4290, + "unclipped_grad_norm": 0.6485001444816589 + }, + { + "bits_per_byte": 1.5837349171738373, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.0977613925933838, + "step": 4300, + "unclipped_grad_norm": 0.6859890818595886 + }, + { + "bits_per_byte": 1.5713755559522324, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.0891945362091064, + "step": 4310, + "unclipped_grad_norm": 0.6751237511634827 + }, + { + "bits_per_byte": 1.6987656851860777, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.1774946451187134, + "step": 4320, + "unclipped_grad_norm": 0.7076508402824402 + }, + { + "bits_per_byte": 1.640483172565784, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.1370962858200073, + "step": 4330, + "unclipped_grad_norm": 0.7175340056419373 + }, + { + "bits_per_byte": 1.5893793877751548, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.1016738414764404, + "step": 4340, + "unclipped_grad_norm": 0.7149683833122253 + }, + { + "bits_per_byte": 1.5492594469645296, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.0738648176193237, + "step": 4350, + "unclipped_grad_norm": 0.6820738911628723 + }, + { + "bits_per_byte": 1.6529066833161608, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.145707607269287, + "step": 4360, + "unclipped_grad_norm": 0.7285676002502441 + }, + { + "bits_per_byte": 1.6119548624233122, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.1173219680786133, + "step": 4370, + "unclipped_grad_norm": 0.6848057508468628 + }, + { + "bits_per_byte": 1.5636072715954101, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.0838099718093872, + "step": 4380, + "unclipped_grad_norm": 0.6535383462905884 + }, + { + "bits_per_byte": 1.699209228442653, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1778020858764648, + "step": 4390, + "unclipped_grad_norm": 0.6795487403869629 + }, + { + "bits_per_byte": 1.6477625102462032, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.1421419382095337, + "step": 4400, + "unclipped_grad_norm": 0.6717441082000732 + }, + { + "bits_per_byte": 1.6503632318925505, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.1439446210861206, + "step": 4410, + "unclipped_grad_norm": 0.696977436542511 + }, + { + "bits_per_byte": 1.688505716183555, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.1703829765319824, + "step": 4420, + "unclipped_grad_norm": 0.7529892325401306 + }, + { + "bits_per_byte": 1.6142178101433649, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.1188905239105225, + "step": 4430, + "unclipped_grad_norm": 0.694868266582489 + }, + { + "bits_per_byte": 1.6221610008561147, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.1243963241577148, + "step": 4440, + "unclipped_grad_norm": 0.6558736562728882 + }, + { + "bits_per_byte": 1.6528647195493502, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.1456785202026367, + "step": 4450, + "unclipped_grad_norm": 0.6744326949119568 + }, + { + "bits_per_byte": 1.6500837600848985, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.1437509059906006, + "step": 4460, + "unclipped_grad_norm": 0.7517809271812439 + }, + { + "bits_per_byte": 1.6192228492487744, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1223597526550293, + "step": 4470, + "unclipped_grad_norm": 0.7247796058654785 + }, + { + "bits_per_byte": 1.6637254239513366, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.1532065868377686, + "step": 4480, + "unclipped_grad_norm": 0.7319335341453552 + }, + { + "bits_per_byte": 1.6952543154034116, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.175060749053955, + "step": 4490, + "unclipped_grad_norm": 0.7066603899002075 + }, + { + "bits_per_byte": 1.6029919865735969, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.1111093759536743, + "step": 4500, + "unclipped_grad_norm": 0.677769660949707 + }, + { + "bits_per_byte": 1.599490247819379, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.1086821556091309, + "step": 4510, + "unclipped_grad_norm": 0.7161181569099426 + }, + { + "bits_per_byte": 1.6742339078843558, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.1604905128479004, + "step": 4520, + "unclipped_grad_norm": 0.7488904595375061 + }, + { + "bits_per_byte": 1.6381440365313988, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.1354749202728271, + "step": 4530, + "unclipped_grad_norm": 0.7328691482543945 + }, + { + "bits_per_byte": 1.6278348804907294, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.1283291578292847, + "step": 4540, + "unclipped_grad_norm": 0.6951945424079895 + }, + { + "bits_per_byte": 1.653657215604526, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.1462278366088867, + "step": 4550, + "unclipped_grad_norm": 0.68836909532547 + }, + { + "bits_per_byte": 1.6466028312314354, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.1413381099700928, + "step": 4560, + "unclipped_grad_norm": 0.7310075163841248 + }, + { + "bits_per_byte": 1.6337182349940953, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.1324071884155273, + "step": 4570, + "unclipped_grad_norm": 0.6918140053749084 + }, + { + "bits_per_byte": 1.6193667987275466, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.1224595308303833, + "step": 4580, + "unclipped_grad_norm": 0.7176027894020081 + }, + { + "bits_per_byte": 1.5916717445285051, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.1032627820968628, + "step": 4590, + "unclipped_grad_norm": 0.6879271864891052 + }, + { + "bits_per_byte": 1.5997678278178717, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.1088745594024658, + "step": 4600, + "unclipped_grad_norm": 0.691472589969635 + }, + { + "bits_per_byte": 1.6045657998116425, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.1122002601623535, + "step": 4610, + "unclipped_grad_norm": 0.6916077136993408 + }, + { + "bits_per_byte": 1.6480117131073033, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.1423146724700928, + "step": 4620, + "unclipped_grad_norm": 0.686012864112854 + }, + { + "bits_per_byte": 1.6420456349488726, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.1381793022155762, + "step": 4630, + "unclipped_grad_norm": 0.7459740042686462 + }, + { + "bits_per_byte": 1.6511240831399676, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.1444720029830933, + "step": 4640, + "unclipped_grad_norm": 0.7440298199653625 + }, + { + "bits_per_byte": 1.6785458569067881, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.1634793281555176, + "step": 4650, + "unclipped_grad_norm": 0.7148184180259705 + }, + { + "bits_per_byte": 1.6225763389579484, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.12468421459198, + "step": 4660, + "unclipped_grad_norm": 0.7037528157234192 + }, + { + "bits_per_byte": 1.6086323295912934, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.1150189638137817, + "step": 4670, + "unclipped_grad_norm": 0.6921704411506653 + }, + { + "bits_per_byte": 1.6026729587562463, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.1108882427215576, + "step": 4680, + "unclipped_grad_norm": 0.7542321085929871 + }, + { + "bits_per_byte": 1.6008523504142127, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.109626293182373, + "step": 4690, + "unclipped_grad_norm": 0.7231722474098206 + }, + { + "bits_per_byte": 1.6341289295643557, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.1326918601989746, + "step": 4700, + "unclipped_grad_norm": 0.6981315612792969 + }, + { + "bits_per_byte": 1.6082362535463564, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.1147444248199463, + "step": 4710, + "unclipped_grad_norm": 0.7216504812240601 + }, + { + "bits_per_byte": 1.6391198660923945, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.1361513137817383, + "step": 4720, + "unclipped_grad_norm": 0.7287876009941101 + }, + { + "bits_per_byte": 1.6620343185454025, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.1520344018936157, + "step": 4730, + "unclipped_grad_norm": 0.7549720406532288 + }, + { + "bits_per_byte": 1.6187323547285137, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.1220197677612305, + "step": 4740, + "unclipped_grad_norm": 0.7869896292686462 + }, + { + "bits_per_byte": 1.585970347669752, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.0993108749389648, + "step": 4750, + "unclipped_grad_norm": 0.7706308960914612 + }, + { + "bits_per_byte": 1.545847827119364, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.0715000629425049, + "step": 4760, + "unclipped_grad_norm": 0.6770147681236267 + }, + { + "bits_per_byte": 1.5563525274340644, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.0787813663482666, + "step": 4770, + "unclipped_grad_norm": 0.6811891794204712 + }, + { + "bits_per_byte": 1.6022997564038741, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.1106295585632324, + "step": 4780, + "unclipped_grad_norm": 0.721454918384552 + }, + { + "bits_per_byte": 1.5960132745668858, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.1062721014022827, + "step": 4790, + "unclipped_grad_norm": 0.7442226409912109 + }, + { + "bits_per_byte": 1.632037792512515, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.1312423944473267, + "step": 4800, + "unclipped_grad_norm": 0.7435004711151123 + }, + { + "bits_per_byte": 1.592038927488097, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.1035172939300537, + "step": 4810, + "unclipped_grad_norm": 0.687485933303833 + }, + { + "bits_per_byte": 1.5873847329904478, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.1002912521362305, + "step": 4820, + "unclipped_grad_norm": 0.7103719115257263 + }, + { + "bits_per_byte": 1.5825477209349312, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.0969384908676147, + "step": 4830, + "unclipped_grad_norm": 0.6801429986953735 + }, + { + "bits_per_byte": 1.5571023717918262, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.079301118850708, + "step": 4840, + "unclipped_grad_norm": 0.7199984788894653 + }, + { + "bits_per_byte": 1.604216675030391, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.1119582653045654, + "step": 4850, + "unclipped_grad_norm": 0.694636881351471 + }, + { + "bits_per_byte": 1.5960309887799247, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.1062843799591064, + "step": 4860, + "unclipped_grad_norm": 0.7184557318687439 + }, + { + "bits_per_byte": 1.6054117824712366, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.1127866506576538, + "step": 4870, + "unclipped_grad_norm": 0.7059701681137085 + }, + { + "bits_per_byte": 1.560965102130205, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.0819785594940186, + "step": 4880, + "unclipped_grad_norm": 0.6785836219787598 + }, + { + "bits_per_byte": 1.5357682679175966, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.0645134449005127, + "step": 4890, + "unclipped_grad_norm": 0.6957149505615234 + }, + { + "bits_per_byte": 1.516216420254258, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.0509611368179321, + "step": 4900, + "unclipped_grad_norm": 0.6988229751586914 + }, + { + "bits_per_byte": 1.5922724799279688, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.1036791801452637, + "step": 4910, + "unclipped_grad_norm": 0.7401829957962036 + }, + { + "bits_per_byte": 1.57446264453522, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.091334342956543, + "step": 4920, + "unclipped_grad_norm": 0.7110046148300171 + }, + { + "bits_per_byte": 1.5705156426979183, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.0885984897613525, + "step": 4930, + "unclipped_grad_norm": 0.7142418026924133 + }, + { + "bits_per_byte": 1.637604698938293, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.135101079940796, + "step": 4940, + "unclipped_grad_norm": 0.773152768611908 + }, + { + "bits_per_byte": 1.5168453608084633, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.0513970851898193, + "step": 4950, + "unclipped_grad_norm": 0.7081190347671509 + }, + { + "bits_per_byte": 1.5893334684073743, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.1016420125961304, + "step": 4960, + "unclipped_grad_norm": 0.7497702240943909 + }, + { + "bits_per_byte": 1.6798649638389058, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.164393663406372, + "step": 4970, + "unclipped_grad_norm": 0.7571852803230286 + }, + { + "bits_per_byte": 1.5442790013781935, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.0704126358032227, + "step": 4980, + "unclipped_grad_norm": 0.7471824288368225 + }, + { + "bits_per_byte": 1.6059244627534586, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.1131420135498047, + "step": 4990, + "unclipped_grad_norm": 0.7206067442893982 + }, + { + "bits_per_byte": 1.631166528403244, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.1306384801864624, + "step": 5000, + "unclipped_grad_norm": 0.7589837908744812 + }, + { + "bits_per_byte": 1.5855355755283709, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.0990095138549805, + "step": 5010, + "unclipped_grad_norm": 0.7112086415290833 + }, + { + "bits_per_byte": 1.5904040604089953, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.102384090423584, + "step": 5020, + "unclipped_grad_norm": 0.7286252379417419 + }, + { + "bits_per_byte": 1.582101769921244, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.0966293811798096, + "step": 5030, + "unclipped_grad_norm": 0.7386847734451294 + }, + { + "bits_per_byte": 1.5587752470367688, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.0804606676101685, + "step": 5040, + "unclipped_grad_norm": 0.7302570343017578 + }, + { + "bits_per_byte": 1.6033209893846974, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.111337423324585, + "step": 5050, + "unclipped_grad_norm": 0.7622739672660828 + }, + { + "bits_per_byte": 1.54526497791559, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.0710960626602173, + "step": 5060, + "unclipped_grad_norm": 0.7352212071418762 + }, + { + "bits_per_byte": 1.6266500920089357, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.1275079250335693, + "step": 5070, + "unclipped_grad_norm": 0.7310096621513367 + }, + { + "bits_per_byte": 1.6437513588801298, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.1393616199493408, + "step": 5080, + "unclipped_grad_norm": 0.8123218417167664 + }, + { + "bits_per_byte": 1.6258579399190616, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.1269588470458984, + "step": 5090, + "unclipped_grad_norm": 0.8349431753158569 + }, + { + "bits_per_byte": 1.549658790679833, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.0741416215896606, + "step": 5100, + "unclipped_grad_norm": 0.7422271966934204 + }, + { + "bits_per_byte": 1.5703663617569694, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.0884950160980225, + "step": 5110, + "unclipped_grad_norm": 0.7262441515922546 + }, + { + "bits_per_byte": 1.5597448852223332, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0811327695846558, + "step": 5120, + "unclipped_grad_norm": 0.749538779258728 + }, + { + "bits_per_byte": 1.5969657144873641, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.106932282447815, + "step": 5130, + "unclipped_grad_norm": 0.7222827672958374 + }, + { + "bits_per_byte": 1.5461644471796023, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.0717195272445679, + "step": 5140, + "unclipped_grad_norm": 0.7582613825798035 + }, + { + "bits_per_byte": 1.549896986651278, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.0743067264556885, + "step": 5150, + "unclipped_grad_norm": 0.7678307890892029 + }, + { + "bits_per_byte": 1.5712579198190422, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.0891129970550537, + "step": 5160, + "unclipped_grad_norm": 0.7660014629364014 + }, + { + "bits_per_byte": 1.584677726065867, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.0984148979187012, + "step": 5170, + "unclipped_grad_norm": 0.7365893721580505 + }, + { + "bits_per_byte": 1.6181584486225844, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.1216219663619995, + "step": 5180, + "unclipped_grad_norm": 0.7808313965797424 + }, + { + "bits_per_byte": 1.56902696087205, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.087566614151001, + "step": 5190, + "unclipped_grad_norm": 0.7353224754333496 + }, + { + "bits_per_byte": 1.5195919237427424, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.0533008575439453, + "step": 5200, + "unclipped_grad_norm": 0.7446879148483276 + }, + { + "bits_per_byte": 1.6084276702367668, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.1148771047592163, + "step": 5210, + "unclipped_grad_norm": 0.7412604689598083 + }, + { + "bits_per_byte": 1.5799695290158469, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.095151424407959, + "step": 5220, + "unclipped_grad_norm": 0.7423371076583862 + }, + { + "bits_per_byte": 1.618757120230238, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.1220369338989258, + "step": 5230, + "unclipped_grad_norm": 0.775067925453186 + }, + { + "bits_per_byte": 1.5733683189277798, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0905758142471313, + "step": 5240, + "unclipped_grad_norm": 0.7500272393226624 + }, + { + "bits_per_byte": 1.5253829235625949, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0573148727416992, + "step": 5250, + "unclipped_grad_norm": 0.7440436482429504 + }, + { + "bits_per_byte": 1.5069763083713519, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.0445563793182373, + "step": 5260, + "unclipped_grad_norm": 0.7253746390342712 + }, + { + "bits_per_byte": 1.6121839433142613, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.117480754852295, + "step": 5270, + "unclipped_grad_norm": 0.764333963394165 + }, + { + "bits_per_byte": 1.5991700161234725, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.1084601879119873, + "step": 5280, + "unclipped_grad_norm": 0.7480786442756653 + }, + { + "bits_per_byte": 1.5744230885255215, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.0913069248199463, + "step": 5290, + "unclipped_grad_norm": 0.7642346024513245 + }, + { + "bits_per_byte": 1.524250933754616, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.056530237197876, + "step": 5300, + "unclipped_grad_norm": 0.7178941965103149 + }, + { + "bits_per_byte": 1.5913442895612622, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.103035807609558, + "step": 5310, + "unclipped_grad_norm": 0.7316482067108154 + }, + { + "bits_per_byte": 1.5618564882096269, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0825964212417603, + "step": 5320, + "unclipped_grad_norm": 0.7464416027069092 + }, + { + "bits_per_byte": 1.6405978849939094, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.1371757984161377, + "step": 5330, + "unclipped_grad_norm": 0.7693371772766113 + }, + { + "bits_per_byte": 1.544090164427546, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.070281744003296, + "step": 5340, + "unclipped_grad_norm": 0.751762330532074 + }, + { + "bits_per_byte": 1.585377523472228, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.0988999605178833, + "step": 5350, + "unclipped_grad_norm": 0.7579334378242493 + }, + { + "bits_per_byte": 1.4929707292156884, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.0348484516143799, + "step": 5360, + "unclipped_grad_norm": 0.6982736587524414 + }, + { + "bits_per_byte": 1.5324773798933367, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.0622323751449585, + "step": 5370, + "unclipped_grad_norm": 0.7345213890075684 + }, + { + "bits_per_byte": 1.5910902711859378, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.1028597354888916, + "step": 5380, + "unclipped_grad_norm": 0.7533562183380127 + }, + { + "bits_per_byte": 1.5100281405109126, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.046671748161316, + "step": 5390, + "unclipped_grad_norm": 0.7339051961898804 + }, + { + "bits_per_byte": 1.5864228340241722, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.099624514579773, + "step": 5400, + "unclipped_grad_norm": 0.744498074054718 + }, + { + "bits_per_byte": 1.5403937413125517, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.067719578742981, + "step": 5410, + "unclipped_grad_norm": 0.735474169254303 + }, + { + "bits_per_byte": 1.4883877355354962, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.0316717624664307, + "step": 5420, + "unclipped_grad_norm": 0.7276646494865417 + }, + { + "bits_per_byte": 1.5535806830501084, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.0768600702285767, + "step": 5430, + "unclipped_grad_norm": 0.7310234904289246 + }, + { + "bits_per_byte": 1.5312303336919304, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.0613679885864258, + "step": 5440, + "unclipped_grad_norm": 0.7094191312789917 + }, + { + "bits_per_byte": 1.5542921752767278, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.0773532390594482, + "step": 5450, + "unclipped_grad_norm": 0.7317906618118286 + }, + { + "bits_per_byte": 1.5324751441188755, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.0622308254241943, + "step": 5460, + "unclipped_grad_norm": 0.7578192353248596 + }, + { + "bits_per_byte": 1.5442955117126762, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0704240798950195, + "step": 5470, + "unclipped_grad_norm": 0.7318540215492249 + }, + { + "bits_per_byte": 1.527471480874673, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.058762550354004, + "step": 5480, + "unclipped_grad_norm": 0.7607505917549133 + }, + { + "bits_per_byte": 1.5954407443221634, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.1058752536773682, + "step": 5490, + "unclipped_grad_norm": 0.845150351524353 + }, + { + "bits_per_byte": 1.567913545190364, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.0867948532104492, + "step": 5500, + "unclipped_grad_norm": 0.7529054880142212 + }, + { + "bits_per_byte": 1.524237175142547, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.056520700454712, + "step": 5510, + "unclipped_grad_norm": 0.7585949301719666 + }, + { + "bits_per_byte": 1.484416656127074, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.0289192199707031, + "step": 5520, + "unclipped_grad_norm": 0.7193371057510376 + }, + { + "bits_per_byte": 1.498793029877998, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.038884162902832, + "step": 5530, + "unclipped_grad_norm": 0.7194657921791077 + }, + { + "bits_per_byte": 1.618510153143599, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.1218657493591309, + "step": 5540, + "unclipped_grad_norm": 0.7707324624061584 + }, + { + "bits_per_byte": 1.5649205311173986, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.084720253944397, + "step": 5550, + "unclipped_grad_norm": 0.7697377800941467 + }, + { + "bits_per_byte": 1.6005926566114097, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.1094462871551514, + "step": 5560, + "unclipped_grad_norm": 0.8203364014625549 + }, + { + "bits_per_byte": 1.5490238307328474, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.0737015008926392, + "step": 5570, + "unclipped_grad_norm": 0.749539852142334 + }, + { + "bits_per_byte": 1.5706370624494275, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.0886826515197754, + "step": 5580, + "unclipped_grad_norm": 0.7590819001197815 + }, + { + "bits_per_byte": 1.5663450634144953, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.085707664489746, + "step": 5590, + "unclipped_grad_norm": 0.7743318676948547 + }, + { + "bits_per_byte": 1.556059640979645, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.0785783529281616, + "step": 5600, + "unclipped_grad_norm": 0.7694693207740784 + }, + { + "bits_per_byte": 1.4900549353529602, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.032827377319336, + "step": 5610, + "unclipped_grad_norm": 0.751623272895813 + }, + { + "bits_per_byte": 1.5553343901409564, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.078075647354126, + "step": 5620, + "unclipped_grad_norm": 0.8062208890914917 + }, + { + "bits_per_byte": 1.477278344220362, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0239713191986084, + "step": 5630, + "unclipped_grad_norm": 0.7192593812942505 + }, + { + "bits_per_byte": 1.4663928743166517, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.0164260864257812, + "step": 5640, + "unclipped_grad_norm": 0.7452183961868286 + }, + { + "bits_per_byte": 1.544778438996299, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.0707588195800781, + "step": 5650, + "unclipped_grad_norm": 0.8047454357147217 + }, + { + "bits_per_byte": 1.477426593265406, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.0240740776062012, + "step": 5660, + "unclipped_grad_norm": 0.7915642261505127 + }, + { + "bits_per_byte": 1.553981230643968, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.0771377086639404, + "step": 5670, + "unclipped_grad_norm": 0.7902792692184448 + }, + { + "bits_per_byte": 1.562685960534738, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0831713676452637, + "step": 5680, + "unclipped_grad_norm": 0.8641749620437622 + }, + { + "bits_per_byte": 1.5716913160992165, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.0894134044647217, + "step": 5690, + "unclipped_grad_norm": 0.7631419897079468 + }, + { + "bits_per_byte": 1.577785693315191, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.0936377048492432, + "step": 5700, + "unclipped_grad_norm": 0.7915937304496765 + }, + { + "bits_per_byte": 1.565421688562013, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.085067629814148, + "step": 5710, + "unclipped_grad_norm": 0.8233179450035095 + }, + { + "bits_per_byte": 1.572203652416137, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0897685289382935, + "step": 5720, + "unclipped_grad_norm": 0.772072970867157 + }, + { + "bits_per_byte": 1.5274866153479487, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.0587730407714844, + "step": 5730, + "unclipped_grad_norm": 0.778079092502594 + }, + { + "bits_per_byte": 1.589287205074292, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.1016099452972412, + "step": 5740, + "unclipped_grad_norm": 0.8043429851531982 + }, + { + "bits_per_byte": 1.4862513670464783, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.0301909446716309, + "step": 5750, + "unclipped_grad_norm": 0.7632706165313721 + }, + { + "bits_per_byte": 1.4988374014019203, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.0389149188995361, + "step": 5760, + "unclipped_grad_norm": 0.787299394607544 + }, + { + "bits_per_byte": 1.5149182952055456, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.0500613451004028, + "step": 5770, + "unclipped_grad_norm": 0.7761343121528625 + }, + { + "bits_per_byte": 1.5968148857025573, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.106827735900879, + "step": 5780, + "unclipped_grad_norm": 0.8154577016830444 + }, + { + "bits_per_byte": 1.4426013103442432, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 0.9999350309371948, + "step": 5790, + "unclipped_grad_norm": 0.7397564053535461 + }, + { + "bits_per_byte": 1.5571016838612228, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.0793006420135498, + "step": 5800, + "unclipped_grad_norm": 0.763837456703186 + }, + { + "bits_per_byte": 1.4762849724289786, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.023282766342163, + "step": 5810, + "unclipped_grad_norm": 0.7477461099624634 + }, + { + "bits_per_byte": 1.4881755089443314, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.031524658203125, + "step": 5820, + "unclipped_grad_norm": 0.8058508634567261 + }, + { + "bits_per_byte": 1.5466410111051432, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.072049856185913, + "step": 5830, + "unclipped_grad_norm": 0.811541736125946 + }, + { + "bits_per_byte": 1.511028047643029, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.0473648309707642, + "step": 5840, + "unclipped_grad_norm": 0.7899665236473083 + }, + { + "bits_per_byte": 1.491222353587017, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.0336365699768066, + "step": 5850, + "unclipped_grad_norm": 0.7476028203964233 + }, + { + "bits_per_byte": 1.5176247861821737, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.0519373416900635, + "step": 5860, + "unclipped_grad_norm": 0.7351008653640747 + }, + { + "bits_per_byte": 1.5753552344931978, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0919530391693115, + "step": 5870, + "unclipped_grad_norm": 0.7911744117736816 + }, + { + "bits_per_byte": 1.505273852110461, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.0433763265609741, + "step": 5880, + "unclipped_grad_norm": 0.7765505909919739 + }, + { + "bits_per_byte": 1.5600044070424852, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.081312656402588, + "step": 5890, + "unclipped_grad_norm": 0.8453527688980103 + }, + { + "bits_per_byte": 1.578068776758511, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.0938339233398438, + "step": 5900, + "unclipped_grad_norm": 0.8509320616722107 + }, + { + "bits_per_byte": 1.5043836699095952, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.0427592992782593, + "step": 5910, + "unclipped_grad_norm": 0.7386367321014404 + }, + { + "bits_per_byte": 1.4825994874380575, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.0276596546173096, + "step": 5920, + "unclipped_grad_norm": 0.7850841879844666 + }, + { + "bits_per_byte": 1.5864175025619953, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.0996208190917969, + "step": 5930, + "unclipped_grad_norm": 0.805269718170166 + }, + { + "bits_per_byte": 1.521077853846197, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.054330825805664, + "step": 5940, + "unclipped_grad_norm": 0.825657308101654 + }, + { + "bits_per_byte": 1.529726345410135, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.0603255033493042, + "step": 5950, + "unclipped_grad_norm": 0.8003990054130554 + }, + { + "bits_per_byte": 1.4804242508699448, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.0261518955230713, + "step": 5960, + "unclipped_grad_norm": 0.8078850507736206 + }, + { + "bits_per_byte": 1.5100630529890378, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.0466959476470947, + "step": 5970, + "unclipped_grad_norm": 0.7944138050079346 + }, + { + "bits_per_byte": 1.4952735769107413, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.0364446640014648, + "step": 5980, + "unclipped_grad_norm": 0.7797579169273376 + }, + { + "bits_per_byte": 1.4963414171899485, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0371848344802856, + "step": 5990, + "unclipped_grad_norm": 0.7583661079406738 + }, + { + "bits_per_byte": 1.577947528989653, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.0937498807907104, + "step": 6000, + "unclipped_grad_norm": 0.8161975145339966 + }, + { + "bits_per_byte": 1.5580080324312697, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0799288749694824, + "step": 6010, + "unclipped_grad_norm": 0.8220515847206116 + }, + { + "bits_per_byte": 1.4971729533068703, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0377612113952637, + "step": 6020, + "unclipped_grad_norm": 0.776614785194397 + }, + { + "bits_per_byte": 1.4978880571691577, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.0382568836212158, + "step": 6030, + "unclipped_grad_norm": 0.8159616589546204 + }, + { + "bits_per_byte": 1.5164909045650352, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.0511513948440552, + "step": 6040, + "unclipped_grad_norm": 0.7814146280288696 + }, + { + "bits_per_byte": 1.5198906576072913, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.053507924079895, + "step": 6050, + "unclipped_grad_norm": 0.7702921628952026 + }, + { + "bits_per_byte": 1.5094605977630653, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.0462783575057983, + "step": 6060, + "unclipped_grad_norm": 0.7836828231811523 + }, + { + "bits_per_byte": 1.4018218160675593, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 0.9716688394546509, + "step": 6070, + "unclipped_grad_norm": 0.7778923511505127 + }, + { + "bits_per_byte": 1.4875319498648027, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.031078577041626, + "step": 6080, + "unclipped_grad_norm": 0.8145858645439148 + }, + { + "bits_per_byte": 1.4918595493084637, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.0340782403945923, + "step": 6090, + "unclipped_grad_norm": 0.8231754899024963 + }, + { + "bits_per_byte": 1.5897618771906736, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.1019389629364014, + "step": 6100, + "unclipped_grad_norm": 0.8403134346008301 + }, + { + "bits_per_byte": 1.5165930622596477, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.0512222051620483, + "step": 6110, + "unclipped_grad_norm": 0.8203406929969788 + }, + { + "bits_per_byte": 1.4717549494052522, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0201427936553955, + "step": 6120, + "unclipped_grad_norm": 0.809775173664093 + }, + { + "bits_per_byte": 1.5419735739433775, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.0688146352767944, + "step": 6130, + "unclipped_grad_norm": 0.7620586156845093 + }, + { + "bits_per_byte": 1.4920430547969343, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.034205436706543, + "step": 6140, + "unclipped_grad_norm": 0.7735241651535034 + }, + { + "bits_per_byte": 1.4664585716892813, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.0164716243743896, + "step": 6150, + "unclipped_grad_norm": 0.7737546563148499 + }, + { + "bits_per_byte": 1.537091674415986, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 1.065430760383606, + "step": 6160, + "unclipped_grad_norm": 0.7964828014373779 + }, + { + "bits_per_byte": 1.481609727282342, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 1.0269736051559448, + "step": 6170, + "unclipped_grad_norm": 0.776137113571167 + }, + { + "bits_per_byte": 1.4726262135145232, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0207467079162598, + "step": 6180, + "unclipped_grad_norm": 0.7977066040039062 + }, + { + "bits_per_byte": 1.4861911731186763, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.030149221420288, + "step": 6190, + "unclipped_grad_norm": 0.7954478859901428 + }, + { + "bits_per_byte": 1.483318374918664, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.0281579494476318, + "step": 6200, + "unclipped_grad_norm": 0.7642073631286621 + }, + { + "bits_per_byte": 1.506708015436006, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.044370412826538, + "step": 6210, + "unclipped_grad_norm": 0.7917945384979248 + }, + { + "bits_per_byte": 1.5370216774770846, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0653822422027588, + "step": 6220, + "unclipped_grad_norm": 0.8426244854927063 + }, + { + "bits_per_byte": 1.4748131449028947, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.0222625732421875, + "step": 6230, + "unclipped_grad_norm": 0.8043993711471558 + }, + { + "bits_per_byte": 1.526992337209369, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.0584304332733154, + "step": 6240, + "unclipped_grad_norm": 0.8457576632499695 + }, + { + "bits_per_byte": 1.4847401554933468, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.0291434526443481, + "step": 6250, + "unclipped_grad_norm": 0.8480363488197327 + }, + { + "bits_per_byte": 1.4337657016561665, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 0.9938106536865234, + "step": 6260, + "unclipped_grad_norm": 0.8023256063461304 + }, + { + "bits_per_byte": 1.5080525758004515, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.045302391052246, + "step": 6270, + "unclipped_grad_norm": 0.8112918734550476 + }, + { + "bits_per_byte": 1.51134122805025, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.0475819110870361, + "step": 6280, + "unclipped_grad_norm": 0.7972313761711121 + }, + { + "bits_per_byte": 1.4974684195010526, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 1.037966012954712, + "step": 6290, + "unclipped_grad_norm": 0.841532289981842 + }, + { + "bits_per_byte": 1.4708343262751835, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.0195046663284302, + "step": 6300, + "unclipped_grad_norm": 0.7936187982559204 + }, + { + "bits_per_byte": 1.5304271747124012, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.0608112812042236, + "step": 6310, + "unclipped_grad_norm": 0.8325384855270386 + }, + { + "bits_per_byte": 1.4908455395989766, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 1.0333753824234009, + "step": 6320, + "unclipped_grad_norm": 0.80328369140625 + }, + { + "bits_per_byte": 1.5107375689457216, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 1.047163486480713, + "step": 6330, + "unclipped_grad_norm": 0.8479825258255005 + }, + { + "bits_per_byte": 1.4319572180910185, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 0.9925571084022522, + "step": 6340, + "unclipped_grad_norm": 0.7563198208808899 + }, + { + "bits_per_byte": 1.5394273707973536, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.0670497417449951, + "step": 6350, + "unclipped_grad_norm": 0.8154724836349487 + }, + { + "bits_per_byte": 1.5363973804544526, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.0649495124816895, + "step": 6360, + "unclipped_grad_norm": 0.8092579245567322 + }, + { + "bits_per_byte": 1.4726698971078425, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 1.0207769870758057, + "step": 6370, + "unclipped_grad_norm": 0.837844729423523 + }, + { + "bits_per_byte": 1.4621105063101676, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.0134577751159668, + "step": 6380, + "unclipped_grad_norm": 0.7885876893997192 + }, + { + "bits_per_byte": 1.5724748690565475, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.089956521987915, + "step": 6390, + "unclipped_grad_norm": 0.8220381736755371 + }, + { + "bits_per_byte": 1.5282840988999997, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.0593258142471313, + "step": 6400, + "unclipped_grad_norm": 0.8607512712478638 + }, + { + "bits_per_byte": 1.55073454216098, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.0748872756958008, + "step": 6410, + "unclipped_grad_norm": 0.8883184194564819 + }, + { + "bits_per_byte": 1.4840087132792275, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0286364555358887, + "step": 6420, + "unclipped_grad_norm": 0.8607943058013916 + }, + { + "bits_per_byte": 1.5312943112380515, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.0614123344421387, + "step": 6430, + "unclipped_grad_norm": 0.8233257532119751 + }, + { + "bits_per_byte": 1.4507710022081792, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.0055978298187256, + "step": 6440, + "unclipped_grad_norm": 0.8178320527076721 + }, + { + "bits_per_byte": 1.4536008047454758, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0075592994689941, + "step": 6450, + "unclipped_grad_norm": 0.7823265194892883 + }, + { + "bits_per_byte": 1.4857582927864548, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.0298491716384888, + "step": 6460, + "unclipped_grad_norm": 0.8048349022865295 + }, + { + "bits_per_byte": 1.5021057596989174, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.0411803722381592, + "step": 6470, + "unclipped_grad_norm": 0.8384398221969604 + }, + { + "bits_per_byte": 1.4571691007855774, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 1.0100326538085938, + "step": 6480, + "unclipped_grad_norm": 0.8145540952682495 + }, + { + "bits_per_byte": 1.520864939324429, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0541832447052002, + "step": 6490, + "unclipped_grad_norm": 0.832450270652771 + }, + { + "bits_per_byte": 1.547503848064522, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.0726479291915894, + "step": 6500, + "unclipped_grad_norm": 0.8682740330696106 + }, + { + "bits_per_byte": 1.534920221466192, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 1.0639256238937378, + "step": 6510, + "unclipped_grad_norm": 0.8737024664878845 + }, + { + "bits_per_byte": 1.5251139426966456, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 1.0571284294128418, + "step": 6520, + "unclipped_grad_norm": 0.8388568162918091 + }, + { + "bits_per_byte": 1.486260654109625, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.0301973819732666, + "step": 6530, + "unclipped_grad_norm": 0.7848463654518127 + }, + { + "bits_per_byte": 1.4542321530567932, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.007996916770935, + "step": 6540, + "unclipped_grad_norm": 0.8017342686653137 + }, + { + "bits_per_byte": 1.5158691872821661, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 1.050720453262329, + "step": 6550, + "unclipped_grad_norm": 0.8034395575523376 + }, + { + "bits_per_byte": 1.5289689338157353, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.0598005056381226, + "step": 6560, + "unclipped_grad_norm": 0.849419116973877 + }, + { + "bits_per_byte": 1.5061327334688699, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 1.0439716577529907, + "step": 6570, + "unclipped_grad_norm": 0.8420149087905884 + }, + { + "bits_per_byte": 1.4788482018574378, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.025059461593628, + "step": 6580, + "unclipped_grad_norm": 0.8419811725616455 + }, + { + "bits_per_byte": 1.5151131515489733, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.0501964092254639, + "step": 6590, + "unclipped_grad_norm": 0.8272185921669006 + }, + { + "bits_per_byte": 1.4130484995505816, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9794505834579468, + "step": 6600, + "unclipped_grad_norm": 0.8023126125335693 + }, + { + "bits_per_byte": 1.4614172442445397, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.0129772424697876, + "step": 6610, + "unclipped_grad_norm": 0.804939329624176 + }, + { + "bits_per_byte": 1.564198032001124, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 1.0842194557189941, + "step": 6620, + "unclipped_grad_norm": 0.8781566023826599 + }, + { + "bits_per_byte": 1.5256852690628118, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.0575244426727295, + "step": 6630, + "unclipped_grad_norm": 0.8214298486709595 + }, + { + "bits_per_byte": 1.4738295761226103, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.0215808153152466, + "step": 6640, + "unclipped_grad_norm": 0.8209505677223206 + }, + { + "bits_per_byte": 1.4730998537349993, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.0210750102996826, + "step": 6650, + "unclipped_grad_norm": 0.8117533922195435 + }, + { + "bits_per_byte": 1.5207756803286312, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.0541213750839233, + "step": 6660, + "unclipped_grad_norm": 0.8577474355697632 + }, + { + "bits_per_byte": 1.4858700815095156, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.0299266576766968, + "step": 6670, + "unclipped_grad_norm": 0.8078488707542419 + }, + { + "bits_per_byte": 1.495475828508156, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.0365848541259766, + "step": 6680, + "unclipped_grad_norm": 0.8518248796463013 + }, + { + "bits_per_byte": 1.5034542756643325, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.0421150922775269, + "step": 6690, + "unclipped_grad_norm": 0.8732016682624817 + }, + { + "bits_per_byte": 1.4940287664837963, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.0355818271636963, + "step": 6700, + "unclipped_grad_norm": 0.8117132782936096 + }, + { + "bits_per_byte": 1.5159397001690198, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.050769329071045, + "step": 6710, + "unclipped_grad_norm": 0.845159113407135 + }, + { + "bits_per_byte": 1.5684515069222629, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.087167739868164, + "step": 6720, + "unclipped_grad_norm": 0.8942575454711914 + }, + { + "bits_per_byte": 1.4556876422310452, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 1.0090057849884033, + "step": 6730, + "unclipped_grad_norm": 0.7959308624267578 + }, + { + "bits_per_byte": 1.4783260625294183, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 1.0246975421905518, + "step": 6740, + "unclipped_grad_norm": 0.8087326884269714 + }, + { + "bits_per_byte": 1.4539184567016195, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 1.0077794790267944, + "step": 6750, + "unclipped_grad_norm": 0.8153780698776245 + }, + { + "bits_per_byte": 1.4494477676924409, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 1.0046806335449219, + "step": 6760, + "unclipped_grad_norm": 0.8494510650634766 + }, + { + "bits_per_byte": 1.4716545115371484, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 1.0200731754302979, + "step": 6770, + "unclipped_grad_norm": 0.8471758961677551 + }, + { + "bits_per_byte": 1.4539220683372875, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 1.007781982421875, + "step": 6780, + "unclipped_grad_norm": 0.8326503038406372 + }, + { + "bits_per_byte": 1.4826679365331008, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.0277070999145508, + "step": 6790, + "unclipped_grad_norm": 0.8103733062744141 + }, + { + "bits_per_byte": 1.4680187982979085, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0175530910491943, + "step": 6800, + "unclipped_grad_norm": 0.8493234515190125 + }, + { + "bits_per_byte": 1.4711782915769092, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0197430849075317, + "step": 6810, + "unclipped_grad_norm": 0.833315372467041 + }, + { + "bits_per_byte": 1.5007637790742347, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 1.0402501821517944, + "step": 6820, + "unclipped_grad_norm": 0.8594779372215271 + }, + { + "bits_per_byte": 1.4810898237287837, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.0266132354736328, + "step": 6830, + "unclipped_grad_norm": 0.8552776575088501 + }, + { + "bits_per_byte": 1.4581277320814867, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 1.0106971263885498, + "step": 6840, + "unclipped_grad_norm": 0.8674116730690002 + }, + { + "bits_per_byte": 1.5447956372613854, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.0707707405090332, + "step": 6850, + "unclipped_grad_norm": 0.9256410002708435 + }, + { + "bits_per_byte": 1.5052100465469909, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.0433320999145508, + "step": 6860, + "unclipped_grad_norm": 0.8572757244110107 + }, + { + "bits_per_byte": 1.4277071828228969, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 0.9896112084388733, + "step": 6870, + "unclipped_grad_norm": 0.797566831111908 + }, + { + "bits_per_byte": 1.45514245722781, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 1.0086278915405273, + "step": 6880, + "unclipped_grad_norm": 0.8778078556060791 + }, + { + "bits_per_byte": 1.4643894484167508, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 1.0150374174118042, + "step": 6890, + "unclipped_grad_norm": 0.8340705633163452 + }, + { + "bits_per_byte": 1.520227915585633, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 1.053741693496704, + "step": 6900, + "unclipped_grad_norm": 0.8448969721794128 + }, + { + "bits_per_byte": 1.5022344027217627, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.0412695407867432, + "step": 6910, + "unclipped_grad_norm": 0.8596795797348022 + }, + { + "bits_per_byte": 1.4379930352143742, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 0.9967408180236816, + "step": 6920, + "unclipped_grad_norm": 0.829356849193573 + }, + { + "bits_per_byte": 1.426546127946922, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.988806426525116, + "step": 6930, + "unclipped_grad_norm": 0.8598417043685913 + }, + { + "bits_per_byte": 1.5131620083749346, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 1.0488439798355103, + "step": 6940, + "unclipped_grad_norm": 0.8814669251441956 + }, + { + "bits_per_byte": 1.4208157520114986, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 0.9848344326019287, + "step": 6950, + "unclipped_grad_norm": 0.8596024513244629 + }, + { + "bits_per_byte": 1.425578983509795, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 0.9881360530853271, + "step": 6960, + "unclipped_grad_norm": 0.8583869338035583 + }, + { + "bits_per_byte": 1.4056750873601405, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 0.9743397235870361, + "step": 6970, + "unclipped_grad_norm": 0.877888560295105 + }, + { + "bits_per_byte": 1.5220512756500806, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.0550055503845215, + "step": 6980, + "unclipped_grad_norm": 0.895971953868866 + }, + { + "bits_per_byte": 1.488323757989375, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.0316274166107178, + "step": 6990, + "unclipped_grad_norm": 0.8794772028923035 + }, + { + "bits_per_byte": 1.507115270353249, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 1.0446527004241943, + "step": 7000, + "unclipped_grad_norm": 0.9036779999732971 + }, + { + "bits_per_byte": 1.3975139086374222, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 0.9686828255653381, + "step": 7010, + "unclipped_grad_norm": 0.8339662551879883 + }, + { + "bits_per_byte": 1.4832072741262066, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 1.028080940246582, + "step": 7020, + "unclipped_grad_norm": 0.8799022436141968 + }, + { + "bits_per_byte": 1.4647886201494034, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 1.0153141021728516, + "step": 7030, + "unclipped_grad_norm": 0.8776206970214844 + }, + { + "bits_per_byte": 1.4563931150648843, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.0094947814941406, + "step": 7040, + "unclipped_grad_norm": 0.8433066606521606 + }, + { + "bits_per_byte": 1.4551758218620774, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.0086510181427002, + "step": 7050, + "unclipped_grad_norm": 0.8651601672172546 + }, + { + "bits_per_byte": 1.4268560406837767, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 0.9890212416648865, + "step": 7060, + "unclipped_grad_norm": 0.8322908282279968 + }, + { + "bits_per_byte": 1.4669136377834642, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 1.016787052154541, + "step": 7070, + "unclipped_grad_norm": 0.875623881816864 + }, + { + "bits_per_byte": 1.4669635127522145, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 1.0168216228485107, + "step": 7080, + "unclipped_grad_norm": 0.859826385974884 + }, + { + "bits_per_byte": 1.4271777342322156, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 0.9892442226409912, + "step": 7090, + "unclipped_grad_norm": 0.8464218974113464 + }, + { + "bits_per_byte": 1.4317552244675802, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 0.9924170970916748, + "step": 7100, + "unclipped_grad_norm": 0.8620118498802185 + }, + { + "bits_per_byte": 1.4118097085164167, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 0.9785919189453125, + "step": 7110, + "unclipped_grad_norm": 0.8274115324020386 + }, + { + "bits_per_byte": 1.4584500275692036, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.010920524597168, + "step": 7120, + "unclipped_grad_norm": 0.8706165552139282 + }, + { + "bits_per_byte": 1.5105655862948588, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 1.047044277191162, + "step": 7130, + "unclipped_grad_norm": 0.905292809009552 + }, + { + "bits_per_byte": 1.4671740195168705, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.016967535018921, + "step": 7140, + "unclipped_grad_norm": 0.8973484039306641 + }, + { + "bits_per_byte": 1.499795688732528, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 1.039579153060913, + "step": 7150, + "unclipped_grad_norm": 0.8849338889122009 + }, + { + "bits_per_byte": 1.4941003112665552, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0356314182281494, + "step": 7160, + "unclipped_grad_norm": 0.8565042018890381 + }, + { + "bits_per_byte": 1.4916631451211784, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.0339421033859253, + "step": 7170, + "unclipped_grad_norm": 0.8975939154624939 + }, + { + "bits_per_byte": 1.4558468981657442, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 1.0091161727905273, + "step": 7180, + "unclipped_grad_norm": 0.8360037803649902 + }, + { + "bits_per_byte": 1.507166865148508, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 1.0446884632110596, + "step": 7190, + "unclipped_grad_norm": 0.8937476277351379 + }, + { + "bits_per_byte": 1.4387650653340973, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 0.9972759485244751, + "step": 7200, + "unclipped_grad_norm": 0.8494468927383423 + }, + { + "bits_per_byte": 1.4236068584523511, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 0.9867690801620483, + "step": 7210, + "unclipped_grad_norm": 0.8318880796432495 + }, + { + "bits_per_byte": 1.4509156396175549, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 1.0056980848312378, + "step": 7220, + "unclipped_grad_norm": 0.8777396082878113 + }, + { + "bits_per_byte": 1.4744760589072037, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 1.022028923034668, + "step": 7230, + "unclipped_grad_norm": 0.9095993638038635 + }, + { + "bits_per_byte": 1.5013884200621685, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 1.0406831502914429, + "step": 7240, + "unclipped_grad_norm": 0.8679016828536987 + }, + { + "bits_per_byte": 1.4849528980324642, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 1.0292909145355225, + "step": 7250, + "unclipped_grad_norm": 0.9112210869789124 + }, + { + "bits_per_byte": 1.4648102899634121, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 1.015329122543335, + "step": 7260, + "unclipped_grad_norm": 0.8722792863845825 + }, + { + "bits_per_byte": 1.4554981173497943, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 1.0088744163513184, + "step": 7270, + "unclipped_grad_norm": 0.8950102925300598 + }, + { + "bits_per_byte": 1.4690169856035162, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.018244981765747, + "step": 7280, + "unclipped_grad_norm": 0.8999441266059875 + }, + { + "bits_per_byte": 1.4590395840963613, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 1.011329174041748, + "step": 7290, + "unclipped_grad_norm": 0.9135274291038513 + }, + { + "bits_per_byte": 1.4596270768317088, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 1.0117363929748535, + "step": 7300, + "unclipped_grad_norm": 0.8378084301948547 + }, + { + "bits_per_byte": 1.4472684035407073, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 1.0031700134277344, + "step": 7310, + "unclipped_grad_norm": 0.8812614679336548 + }, + { + "bits_per_byte": 1.4169898259604046, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 0.982182502746582, + "step": 7320, + "unclipped_grad_norm": 0.9021150469779968 + }, + { + "bits_per_byte": 1.472447179574975, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 1.0206226110458374, + "step": 7330, + "unclipped_grad_norm": 0.9255433678627014 + }, + { + "bits_per_byte": 1.4364492329489043, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.995670735836029, + "step": 7340, + "unclipped_grad_norm": 0.8845773339271545 + }, + { + "bits_per_byte": 1.443955329754486, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 1.0008735656738281, + "step": 7350, + "unclipped_grad_norm": 0.9007892608642578 + }, + { + "bits_per_byte": 1.4441343636940343, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 1.0009976625442505, + "step": 7360, + "unclipped_grad_norm": 0.8298656940460205 + }, + { + "bits_per_byte": 1.4034473100921891, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 0.9727955460548401, + "step": 7370, + "unclipped_grad_norm": 0.8596914410591125 + }, + { + "bits_per_byte": 1.4053444507138566, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 0.9741105437278748, + "step": 7380, + "unclipped_grad_norm": 0.8395463824272156 + }, + { + "bits_per_byte": 1.4235850166556916, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 0.9867539405822754, + "step": 7390, + "unclipped_grad_norm": 0.8371807336807251 + }, + { + "bits_per_byte": 1.430157333658414, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.9913095235824585, + "step": 7400, + "unclipped_grad_norm": 0.8753771781921387 + }, + { + "bits_per_byte": 1.4671196729991978, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 1.0169298648834229, + "step": 7410, + "unclipped_grad_norm": 0.8915236592292786 + }, + { + "bits_per_byte": 1.4566235718170406, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0096545219421387, + "step": 7420, + "unclipped_grad_norm": 0.87628173828125 + }, + { + "bits_per_byte": 1.402019080168099, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 0.9718055725097656, + "step": 7430, + "unclipped_grad_norm": 0.8717849254608154 + }, + { + "bits_per_byte": 1.3877343731700849, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 0.9619041681289673, + "step": 7440, + "unclipped_grad_norm": 0.8654010891914368 + }, + { + "bits_per_byte": 1.4952202622889739, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 1.036407709121704, + "step": 7450, + "unclipped_grad_norm": 0.8981733322143555 + }, + { + "bits_per_byte": 1.421701806628744, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 0.9854485988616943, + "step": 7460, + "unclipped_grad_norm": 0.8844080567359924 + }, + { + "bits_per_byte": 1.3885631575645927, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 0.9624786376953125, + "step": 7470, + "unclipped_grad_norm": 0.8624504804611206 + }, + { + "bits_per_byte": 1.4875814808682513, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 1.0311129093170166, + "step": 7480, + "unclipped_grad_norm": 0.8718687891960144 + }, + { + "bits_per_byte": 1.4513239264307032, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 1.0059810876846313, + "step": 7490, + "unclipped_grad_norm": 0.9119541049003601 + }, + { + "bits_per_byte": 1.4729782620008394, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.0209907293319702, + "step": 7500, + "unclipped_grad_norm": 0.9133355021476746 + }, + { + "bits_per_byte": 1.4663156541064142, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 1.016372561454773, + "step": 7510, + "unclipped_grad_norm": 0.8953580260276794 + }, + { + "bits_per_byte": 1.4675236602460746, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 1.0172098875045776, + "step": 7520, + "unclipped_grad_norm": 0.9007623195648193 + }, + { + "bits_per_byte": 1.4447141172100928, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.0013995170593262, + "step": 7530, + "unclipped_grad_norm": 0.9128515720367432 + }, + { + "bits_per_byte": 1.4648584451056537, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.0153625011444092, + "step": 7540, + "unclipped_grad_norm": 0.8641919493675232 + }, + { + "bits_per_byte": 1.4437878186525457, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 1.0007574558258057, + "step": 7550, + "unclipped_grad_norm": 0.8774824142456055 + }, + { + "bits_per_byte": 1.370168151202283, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 0.9497281908988953, + "step": 7560, + "unclipped_grad_norm": 0.8748355507850647 + }, + { + "bits_per_byte": 1.4127829583376494, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 0.9792665243148804, + "step": 7570, + "unclipped_grad_norm": 0.8943585157394409 + }, + { + "bits_per_byte": 1.388180496166423, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 0.962213397026062, + "step": 7580, + "unclipped_grad_norm": 0.8687419295310974 + }, + { + "bits_per_byte": 1.4652694836412157, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 1.0156474113464355, + "step": 7590, + "unclipped_grad_norm": 0.8941670656204224 + }, + { + "bits_per_byte": 1.4983995335728237, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 1.0386114120483398, + "step": 7600, + "unclipped_grad_norm": 0.9189749360084534 + }, + { + "bits_per_byte": 1.4581232605325642, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 1.0106940269470215, + "step": 7610, + "unclipped_grad_norm": 0.9311223030090332 + }, + { + "bits_per_byte": 1.4659639495853998, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 1.0161287784576416, + "step": 7620, + "unclipped_grad_norm": 0.8894855976104736 + }, + { + "bits_per_byte": 1.4393443029022033, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 0.9976774454116821, + "step": 7630, + "unclipped_grad_norm": 0.8996445536613464 + }, + { + "bits_per_byte": 1.4137304107612527, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.9799232482910156, + "step": 7640, + "unclipped_grad_norm": 0.931556761264801 + }, + { + "bits_per_byte": 1.3955731704137608, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 0.9673376083374023, + "step": 7650, + "unclipped_grad_norm": 0.8838335871696472 + }, + { + "bits_per_byte": 1.4002129183687377, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 0.9705536365509033, + "step": 7660, + "unclipped_grad_norm": 0.9093811511993408 + }, + { + "bits_per_byte": 1.4570760581714606, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 1.0099681615829468, + "step": 7670, + "unclipped_grad_norm": 0.9101974368095398 + }, + { + "bits_per_byte": 1.4362143046478257, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 0.9955078959465027, + "step": 7680, + "unclipped_grad_norm": 0.9222767949104309 + }, + { + "bits_per_byte": 1.420382269739999, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 0.984533965587616, + "step": 7690, + "unclipped_grad_norm": 0.8661118149757385 + }, + { + "bits_per_byte": 1.4718006967903818, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 1.020174503326416, + "step": 7700, + "unclipped_grad_norm": 0.8786870241165161 + }, + { + "bits_per_byte": 1.4153097274441258, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 0.9810179471969604, + "step": 7710, + "unclipped_grad_norm": 0.881650984287262 + }, + { + "bits_per_byte": 1.4085972445809505, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 0.9763652086257935, + "step": 7720, + "unclipped_grad_norm": 0.8675586581230164 + }, + { + "bits_per_byte": 1.4152660438508067, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 0.9809876680374146, + "step": 7730, + "unclipped_grad_norm": 0.8522657752037048 + }, + { + "bits_per_byte": 1.4640666369810813, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 1.0148136615753174, + "step": 7740, + "unclipped_grad_norm": 0.8849946856498718 + }, + { + "bits_per_byte": 1.4133721708995055, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9796749353408813, + "step": 7750, + "unclipped_grad_norm": 0.9055566191673279 + }, + { + "bits_per_byte": 1.4113202458920613, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 0.978252649307251, + "step": 7760, + "unclipped_grad_norm": 0.8938711285591125 + }, + { + "bits_per_byte": 1.4666288345136353, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 1.016589641571045, + "step": 7770, + "unclipped_grad_norm": 0.8973739147186279 + }, + { + "bits_per_byte": 1.4496102912975062, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 1.0047932863235474, + "step": 7780, + "unclipped_grad_norm": 0.9293894171714783 + }, + { + "bits_per_byte": 1.4517860438135715, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 1.0063014030456543, + "step": 7790, + "unclipped_grad_norm": 0.8974235653877258 + }, + { + "bits_per_byte": 1.4363978961276218, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 0.9956351518630981, + "step": 7800, + "unclipped_grad_norm": 0.8982102870941162 + }, + { + "bits_per_byte": 1.4227292309849984, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9861607551574707, + "step": 7810, + "unclipped_grad_norm": 0.9417922496795654 + }, + { + "bits_per_byte": 1.4556315758868639, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 1.0089669227600098, + "step": 7820, + "unclipped_grad_norm": 0.9800649285316467 + }, + { + "bits_per_byte": 1.4507395293830714, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 1.0055760145187378, + "step": 7830, + "unclipped_grad_norm": 0.8959765434265137 + }, + { + "bits_per_byte": 1.4296197158918167, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.9909368753433228, + "step": 7840, + "unclipped_grad_norm": 0.9423794150352478 + }, + { + "bits_per_byte": 1.4292436758257052, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.99067622423172, + "step": 7850, + "unclipped_grad_norm": 0.8978332281112671 + }, + { + "bits_per_byte": 1.4664331182569534, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 1.0164539813995361, + "step": 7860, + "unclipped_grad_norm": 0.9325177669525146 + }, + { + "bits_per_byte": 1.4478889169450202, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0036001205444336, + "step": 7870, + "unclipped_grad_norm": 0.8888780474662781 + }, + { + "bits_per_byte": 1.4035363111140104, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9728572368621826, + "step": 7880, + "unclipped_grad_norm": 0.8764894008636475 + }, + { + "bits_per_byte": 1.4350234967732516, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 0.9946824908256531, + "step": 7890, + "unclipped_grad_norm": 0.8987365365028381 + }, + { + "bits_per_byte": 1.402528406788629, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 0.9721586108207703, + "step": 7900, + "unclipped_grad_norm": 0.8790932297706604 + }, + { + "bits_per_byte": 1.4543580443572248, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.0080841779708862, + "step": 7910, + "unclipped_grad_norm": 0.9732881188392639 + }, + { + "bits_per_byte": 1.4785298619706908, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 1.0248388051986694, + "step": 7920, + "unclipped_grad_norm": 0.9096516966819763 + }, + { + "bits_per_byte": 1.4211735619166188, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.9850824475288391, + "step": 7930, + "unclipped_grad_norm": 0.8723064064979553 + }, + { + "bits_per_byte": 1.4468050822792828, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 1.0028488636016846, + "step": 7940, + "unclipped_grad_norm": 0.9346098303794861 + }, + { + "bits_per_byte": 1.4028910322079733, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 0.9724099636077881, + "step": 7950, + "unclipped_grad_norm": 0.8917330503463745 + }, + { + "bits_per_byte": 1.4679954086573912, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 1.0175368785858154, + "step": 7960, + "unclipped_grad_norm": 0.9202073216438293 + }, + { + "bits_per_byte": 1.4168718458619127, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 0.9821007251739502, + "step": 7970, + "unclipped_grad_norm": 0.9002774357795715 + }, + { + "bits_per_byte": 1.4408839775840527, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 0.9987446665763855, + "step": 7980, + "unclipped_grad_norm": 0.9515719413757324 + }, + { + "bits_per_byte": 1.416254944093268, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 0.9816731214523315, + "step": 7990, + "unclipped_grad_norm": 0.9514462351799011 + }, + { + "bits_per_byte": 1.4430477773058832, + "learning_rate": 3e-05, + "loss_nats": 1.0002444982528687, + "step": 8000, + "unclipped_grad_norm": 0.9392750859260559 + } + ], + "variant": "uniform_group_6_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073003.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073003.json new file mode 100644 index 0000000..34ec04c --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073003.json @@ -0,0 +1,14045 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "783f7ece2469352b3cbfbfe1afd3690a94ebb2d051688e080cae43477cfc59f7", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.00027819047681987286, + 0.0002560806169640273, + 0.0002331122086616233, + 0.00022053472639527172, + 0.00020666928321588784, + 0.00019684624567162246, + 0.0001890837011160329, + 0.00018263237143401057, + 0.00017469045997131616, + 0.0001688334159553051, + 0.0001649101759539917, + 0.00016157960635609925, + 0.00015717606584075838, + 0.00015400603297166526, + 0.00015046694898046553, + 0.00014767753600608557, + 0.00014588078192900866, + 0.00014366162940859795, + 0.00014116008242126554, + 0.00013849612150806934, + 0.0001361419854220003, + 0.00013430001854430884, + 0.00013224364374764264, + 0.00013045054220128804, + 0.00012888714263681322, + 0.00012767451698891819, + 0.0001263424928765744, + 0.00012512138346210122, + 0.00012419422273524106, + 0.00012343283742666245, + 0.0001227961911354214, + 0.00012195061572128907 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00022039370378479362, + "first_to_last_ratio": 1.7624457042066763, + "imbalance_abs_log_ratio": 0.5667024491058967, + "last_quartile_mean": 0.00012504992537287762, + "mean": 0.0001607882525149762, + "normalized": [ + 1.7301666786506156, + 1.592657504255016, + 1.4498087081325215, + 1.3715848200709229, + 1.2853506396348091, + 1.224257634451794, + 1.175979576607571, + 1.1358564358860717, + 1.0864628307036615, + 1.050035766385231, + 1.0256357250890054, + 1.0049217142965674, + 0.9775345112735686, + 0.9578189361646354, + 0.9358080993289649, + 0.9184597363064851, + 0.9072850761620224, + 0.8934833681037547, + 0.8779253472396409, + 0.8613572157279928, + 0.8467159963027754, + 0.8352601414820391, + 0.8224708066612338, + 0.81131886291965, + 0.8015955184587156, + 0.7940537632065271, + 0.7857694259399116, + 0.7781749070905981, + 0.7724085609032495, + 0.7676732316943716, + 0.7637136993200662, + 0.75845476155001 + ], + "population_cv": 0.2498932502086342 + }, + "activation_output_rms_by_block": [ + 0.0034565660171210766, + 0.004928664769977331, + 0.0060537648387253284, + 0.006960673723369837, + 0.0036145171616226435, + 0.0051584807224571705, + 0.006370538379997015, + 0.007546419743448496, + 0.00395671371370554, + 0.005665866658091545, + 0.006755253300070763, + 0.007946331985294819, + 0.004383862018585205, + 0.006652849726378918, + 0.007932198233902454, + 0.009347881190478802, + 0.004880861379206181, + 0.006906712893396616, + 0.008132148534059525, + 0.009816600009799004, + 0.005453306715935469, + 0.007622495759278536, + 0.009448722936213017, + 0.010670131072402, + 0.0055701653473079205, + 0.007738742511719465, + 0.009509840048849583, + 0.010865328833460808, + 0.005987230688333511, + 0.00815181527286768, + 0.009874933399260044, + 0.011666664853692055 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.005511203169589862, + "first_to_last_ratio": 0.6356203088456116, + "imbalance_abs_log_ratio": 0.45315389258709987, + "last_quartile_mean": 0.008670590119436383, + "mean": 0.007157071326219011, + "normalized": [ + 0.4829581625738996, + 0.6886426787339398, + 0.8458438602600115, + 0.9725589429115095, + 0.5050274053272774, + 0.7207530129760953, + 0.8901040788372985, + 1.0544005221525679, + 0.5528397766850007, + 0.7916459680002588, + 0.9438571997073384, + 1.1102770425361632, + 0.6125217730505899, + 0.9295491721602742, + 1.1083022471557418, + 1.3061042379491232, + 0.6819634955048963, + 0.9650194302374437, + 1.1362396940587196, + 1.3715945478756304, + 0.7619466772614072, + 1.0650300118366156, + 1.3201940438400879, + 1.490851576861243, + 0.7782743937317399, + 1.0812722353862223, + 1.3287334463206348, + 1.5181249897087195, + 0.8365475786723071, + 1.138987569259587, + 1.3797450031110483, + 1.6300892253166077 + ], + "population_cv": 0.3025375276561286 + }, + "bits_per_byte": 8.06433666787865, + "branch_output_rms_by_sublayer": [ + 0.0021227982360869646, + 0.0027448106557130814, + 0.00224129855632782, + 0.0027243648655712605, + 0.0024903982412070036, + 0.0027103275060653687, + 0.0023722455371171236, + 0.0027234479784965515, + 0.0023577234242111444, + 0.0027595730498433113, + 0.0023993616923689842, + 0.0027272943407297134, + 0.002688874490559101, + 0.002728298306465149, + 0.0029664887115359306, + 0.0027007071767002344, + 0.0029547461308538914, + 0.002684502862393856, + 0.003052416956052184, + 0.002683063969016075, + 0.0028214550111442804, + 0.0027139110025018454, + 0.0032544767018407583, + 0.002666700165718794, + 0.00348988501355052, + 0.002679243916645646, + 0.003914494998753071, + 0.0026584581937640905, + 0.0035071265883743763, + 0.002684473292902112, + 0.004211232997477055, + 0.0026407993864268064, + 0.004072706215083599, + 0.002663099905475974, + 0.004050548188388348, + 0.0026283918414264917, + 0.003929397091269493, + 0.002597011858597398, + 0.004410331137478352, + 0.0026434636674821377, + 0.004824656527489424, + 0.0025708195753395557, + 0.00433384021744132, + 0.002691218862310052, + 0.004475373309105635, + 0.0026598076801747084, + 0.004755842965096235, + 0.0026596153620630503, + 0.004843301139771938, + 0.0026765079237520695, + 0.0050001065246760845, + 0.002582729561254382, + 0.005346097983419895, + 0.002614464145153761, + 0.004958339501172304, + 0.0025760054122656584, + 0.0054801348596811295, + 0.002634295728057623, + 0.005455824080854654, + 0.00259007909335196, + 0.005286100786179304, + 0.00257018837146461, + 0.005402065813541412, + 0.0025643527042120695 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.006159982634052032, + 0.005514663551817784, + 0.005191681274696802, + 0.00489831575692433, + 0.004666415075367917, + 0.004399813239791133, + 0.004483878567781538, + 0.004868281936291656, + 0.00443828064080646, + 0.004266304440803418, + 0.0039471188591477255, + 0.004494388758593749, + 0.0042334459038140095, + 0.00446434979681851, + 0.004322339302329385, + 0.004638471384339534, + 0.004165330760400444, + 0.004499937764754925, + 0.003981478974932938, + 0.004372386299982334, + 0.00458495381977442, + 0.00429501466057645, + 0.003963900484885007, + 0.004271473918508079, + 0.004041799650627023, + 0.004206875397956982, + 0.003955591472942362, + 0.003916464042690177, + 0.003989587235588314, + 0.003922309599766117, + 0.003679536678174494, + 0.003763244111925236 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.005022879004590398, + "first_to_last_ratio": 1.2766484804448084, + "imbalance_abs_log_ratio": 0.24423826934058726, + "last_quartile_mean": 0.003934426023708838, + "mean": 0.0043936754999019155, + "normalized": [ + 1.4020112851278042, + 1.255136741878387, + 1.1816260155791434, + 1.1148560600421402, + 1.0620754936207943, + 1.0013969488391563, + 1.0205302070855338, + 1.108020366183241, + 1.0101521245493756, + 0.9710103627130996, + 0.8983637638318627, + 1.0229223252136126, + 0.9635317637610054, + 1.0160854612313022, + 0.9837638902613262, + 1.0557155130011497, + 0.9480287655502622, + 1.0241852783291305, + 0.9061841219320409, + 0.995154580733134, + 1.0435349219296635, + 0.9775448051801577, + 0.9021832597727114, + 0.9721869351988865, + 0.9199131002545027, + 0.9574843199164109, + 0.9002921296829196, + 0.891386731400079, + 0.9080295610536958, + 0.892717179467095, + 0.8374620925593245, + 0.8565138941210488 + ], + "population_cv": 0.11556590700625036 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028097106143832207, + 0.014079716056585312, + 0.014136734418570995, + 0.014167090877890587, + 0.014224153943359852, + 0.014255538582801819, + 0.014337832108139992, + 0.014355218037962914, + 0.014408928342163563, + 0.009655896574258804, + 0.009704872034490108, + 0.009734091348946095, + 0.009773381985723972, + 0.009828771464526653, + 0.00987160298973322, + 0.009915035218000412, + 0.00994801800698042, + 0.007479202933609486, + 0.007515520788729191, + 0.007559895049780607, + 0.007602147292345762, + 0.0076358383521437645, + 0.007658426649868488, + 0.007706487085670233, + 0.007734050042927265, + 0.0062490664422512054, + 0.0062688966281712055, + 0.006315518636256456, + 0.006336068734526634, + 0.0063744354993104935, + 0.006399397272616625, + 0.006439714692533016, + 0.006462901830673218, + 0.0054116579703986645, + 0.005432929843664169, + 0.005481995176523924, + 0.005504137370735407, + 0.005541813559830189, + 0.005560056772083044, + 0.005614723544567823, + 0.005631812382489443, + 0.004862982779741287, + 0.004873365629464388, + 0.004923288244754076, + 0.004937449004501104, + 0.0049925679340958595, + 0.005005750339478254, + 0.005004179198294878, + 0.005022916477173567, + 0.004448203835636377, + 0.0044684382155537605, + 0.004490054678171873, + 0.004500201437622309, + 0.00455933902412653, + 0.004573327023535967, + 0.004625064320862293, + 0.00463611725717783, + 0.004160413984209299, + 0.004166214261204004, + 0.004242690280079842, + 0.0042496914975345135, + 0.004270957782864571, + 0.004283483140170574, + 0.004303553607314825 + ], + "loss_nats": 5.5897722244262695, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0021227982360869646, + 0.0034565660171210766, + 0.00413061399012804, + 0.004928664769977331, + 0.005416445434093475, + 0.0060537648387253284, + 0.006388718727976084, + 0.006960673723369837, + 0.0023577234242111444, + 0.0036145171616226435, + 0.004388594534248114, + 0.0051584807224571705, + 0.00574506726115942, + 0.006370538379997015, + 0.007066869642585516, + 0.007546419743448496, + 0.0029547461308538914, + 0.00395671371370554, + 0.004991721361875534, + 0.005665866658091545, + 0.006190058775246143, + 0.006755253300070763, + 0.0074937604367733, + 0.007946331985294819, + 0.00348988501355052, + 0.004383862018585205, + 0.006057556718587875, + 0.006652849726378918, + 0.0074848332442343235, + 0.007932198233902454, + 0.00897275935858488, + 0.009347881190478802, + 0.004072706215083599, + 0.004880861379206181, + 0.0063449000008404255, + 0.006906712893396616, + 0.007735534105449915, + 0.008132148534059525, + 0.00946879480034113, + 0.009816600009799004, + 0.004824656527489424, + 0.005453306715935469, + 0.007074959110468626, + 0.007622495759278536, + 0.009095527231693268, + 0.009448722936213017, + 0.010346688330173492, + 0.010670131072402, + 0.004843301139771938, + 0.0055701653473079205, + 0.007334613241255283, + 0.007738742511719465, + 0.009146629832684994, + 0.009509840048849583, + 0.010587968863546848, + 0.010865328833460808, + 0.0054801348596811295, + 0.005987230688333511, + 0.0077637480571866035, + 0.00815181527286768, + 0.009536821395158768, + 0.009874933399260044, + 0.011367839761078358, + 0.011666664853692055 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.847539771115407e-05, + 6.490082159871235e-05, + 6.151448906166479e-05, + 5.866558421985246e-05, + 5.401135786087252e-05, + 5.0929098506458104e-05, + 4.8038491513580084e-05, + 4.538292341749184e-05, + 4.138937219977379e-05, + 3.859930075122975e-05, + 3.618111077230424e-05, + 3.389207267900929e-05, + 3.1477487937081605e-05, + 2.956116441055201e-05, + 2.789581412798725e-05, + 2.6259358492097817e-05, + 2.4967697754618712e-05, + 2.3655948098166846e-05, + 2.2588923457078636e-05, + 2.171608866774477e-05, + 2.1130250388523564e-05, + 2.0404178940225393e-05, + 1.9834385966532864e-05, + 1.9313640223117545e-05, + 1.905966200865805e-05, + 1.8681286746868864e-05, + 1.834763133956585e-05, + 1.8089995137415826e-05, + 1.7755672161001712e-05, + 1.754466757120099e-05, + 1.7342761566396803e-05, + 1.7196996850543655e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.648977048622328e-05, + "first_to_last_ratio": 3.137913669654462, + "imbalance_abs_log_ratio": 1.1435581426686838, + "last_quartile_mean": 1.800233417270647e-05, + "mean": 3.2337613504296314e-05, + "normalized": [ + 2.1175154963756544, + 2.0069762287835418, + 1.9022581568516828, + 1.8141593600300239, + 1.6702332673280504, + 1.5749182758861213, + 1.4855298925258593, + 1.4034097912470365, + 1.2799142458139303, + 1.1936347976359332, + 1.118855315884621, + 1.0480696936558553, + 0.9734016993214285, + 0.9141418060001419, + 0.8626429443929455, + 0.8120376133696152, + 0.7720946306474209, + 0.7315304233883543, + 0.6985340292374239, + 0.6715427118596619, + 0.6534264003655136, + 0.6309735546042856, + 0.6133534239902306, + 0.5972500172454461, + 0.5893960606006322, + 0.5776952818236541, + 0.5673774082657095, + 0.5594103329552264, + 0.5490718156626718, + 0.5425467642771485, + 0.5363030751818676, + 0.5317954847923114 + ], + "population_cv": 0.4866459599041023 + }, + "activation_output_rms_by_block": [ + 0.005520553328096867, + 0.009550899267196655, + 0.013989307917654514, + 0.018765948712825775, + 0.006620377767831087, + 0.011785997077822685, + 0.017189405858516693, + 0.02231588587164879, + 0.007451951503753662, + 0.013181107118725777, + 0.01907873898744583, + 0.02571253851056099, + 0.008935928344726562, + 0.016163820400834084, + 0.023612193763256073, + 0.03195178136229515, + 0.009834196418523788, + 0.019116180017590523, + 0.02841295301914215, + 0.03848475590348244, + 0.012644016183912754, + 0.0226194616407156, + 0.034674666821956635, + 0.04606693610548973, + 0.014916402287781239, + 0.029746420681476593, + 0.046650227159261703, + 0.062961645424366, + 0.019602956250309944, + 0.0373276025056839, + 0.055456433445215225, + 0.07453885674476624 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013217296975199133, + "first_to_last_ratio": 0.30990095856059247, + "imbalance_abs_log_ratio": 1.1715025210644914, + "last_quartile_mean": 0.042650068062357605, + "mean": 0.02515250457508955, + "normalized": [ + 0.21948324516216539, + 0.37971961156725687, + 0.5561795198522376, + 0.746086683208921, + 0.2632094846883659, + 0.468581450512696, + 0.6834073245946519, + 0.8872232109143483, + 0.29627075433012345, + 0.5240474991019397, + 0.7585224338390923, + 1.0222655335892905, + 0.3552699222476833, + 0.6426326393293792, + 0.9387611358051807, + 1.2703220574677658, + 0.3909827901696655, + 0.7600109945521186, + 1.1296271881918936, + 1.530056610807531, + 0.5026941212222297, + 0.8992926160966642, + 1.3785771002819978, + 1.8315049289808436, + 0.593038448447559, + 1.1826424916322944, + 1.854695106803121, + 2.5031958641097605, + 0.7793639870649005, + 1.4840511168280346, + 2.204807607912652, + 2.963476520687636 + ], + "population_cv": 0.6659510186251899 + }, + "bits_per_byte": 5.183186302409792, + "branch_output_rms_by_sublayer": [ + 0.00272006937302649, + 0.004830322694033384, + 0.0025339166168123484, + 0.00494748167693615, + 0.0027577043510973454, + 0.005500274244695902, + 0.002650825073942542, + 0.0059000770561397076, + 0.0027917264960706234, + 0.006001376546919346, + 0.0025352563243359327, + 0.006052698474377394, + 0.0034094403963536024, + 0.006183222867548466, + 0.003356113564223051, + 0.005940464790910482, + 0.0035560799296945333, + 0.006375520955771208, + 0.0038353921845555305, + 0.006207411643117666, + 0.0033839894458651543, + 0.006487393286079168, + 0.004606761038303375, + 0.0069061568938195705, + 0.0048495507799088955, + 0.007097041234374046, + 0.005102694034576416, + 0.006851170212030411, + 0.005348205100744963, + 0.006814420223236084, + 0.006069976836442947, + 0.0070969522930681705, + 0.005550710018724203, + 0.0070962924510240555, + 0.006322797853499651, + 0.007617968134582043, + 0.007153828628361225, + 0.007433404680341482, + 0.008639195002615452, + 0.00742357037961483, + 0.008386691100895405, + 0.007437188643962145, + 0.007544232998043299, + 0.007736989296972752, + 0.009543494321405888, + 0.0074466983787715435, + 0.008977021090686321, + 0.00726865790784359, + 0.009665686637163162, + 0.00845401268452406, + 0.011951345950365067, + 0.008405361324548721, + 0.014285893179476261, + 0.008240332826972008, + 0.013714144937694073, + 0.007825162261724472, + 0.014480169862508774, + 0.007934519089758396, + 0.014547434635460377, + 0.007785224821418524, + 0.01368383876979351, + 0.008720025420188904, + 0.015162315219640732, + 0.008376669138669968 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00023648881754453216, + 0.00020477862533202742, + 0.0001969623539143767, + 0.00018101517277836545, + 0.00017006397804935624, + 0.00015934910402257528, + 0.00017354602296195662, + 0.00015987439520781115, + 0.00016445880316126357, + 0.00016157962500657773, + 0.00014703135633090014, + 0.00016954514740805267, + 0.00016266620196335487, + 0.00016562773874885197, + 0.00015317147847637442, + 0.00015944930904397407, + 0.0001405569356543629, + 0.00015370943216391574, + 0.000144762004183892, + 0.00016299107021754654, + 0.00015366455425146273, + 0.0001349005228689607, + 0.00015012219431237723, + 0.00014215756711650107, + 0.00014226460036910897, + 0.00015190831469909813, + 0.00015589491598331132, + 0.00014358163707721538, + 0.00014357759126620013, + 0.000140443281272529, + 0.00013532251196204462, + 0.00013400865957486014 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00018525980872637513, + "first_to_last_ratio": 1.2921329693477606, + "imbalance_abs_log_ratio": 0.25629431751659953, + "last_quartile_mean": 0.00014337518902554595, + "mean": 0.00015923356009136678, + "normalized": [ + 1.485169441723447, + 1.2860267974573154, + 1.2369399629158668, + 1.1367903391376828, + 1.0680159254856518, + 1.0007256254971766, + 1.0898834571203297, + 1.004024497826191, + 1.032814961035215, + 1.01473348277753, + 0.9233691455905079, + 1.064757625909822, + 1.0215572764310392, + 1.0401559737395576, + 0.9619296233060795, + 1.001354921365091, + 0.8827092452979924, + 0.9653080172026466, + 0.909117425377093, + 1.0235974760849642, + 0.9650261801801793, + 0.8471865025912627, + 0.9427798651630879, + 0.8927613439964059, + 0.8934335217241819, + 0.9539968497340291, + 0.9790330373437623, + 0.9017046217821767, + 0.9016792137525318, + 0.8819954863280323, + 0.8498366291904658, + 0.8415855269326842 + ], + "population_cv": 0.1343502879334377 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6920387148857117, + "mean_weights": [ + 0.5231621265411377, + 0.4768379330635071 + ], + "sources": 2 + }, + { + "entropy_mean": 0.692237913608551, + "mean_weights": [ + 0.5206801295280457, + 0.47931987047195435 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6930609941482544, + "mean_weights": [ + 0.5023574233055115, + 0.4976426362991333 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6924445629119873, + "mean_weights": [ + 0.517627477645874, + 0.482372522354126 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929888129234314, + "mean_weights": [ + 0.5013466477394104, + 0.498653382062912 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929047703742981, + "mean_weights": [ + 0.509869396686554, + 0.49013060331344604 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927404403686523, + "mean_weights": [ + 0.5128230452537537, + 0.48717695474624634 + ], + "sources": 2 + }, + { + "entropy_mean": 0.692888617515564, + "mean_weights": [ + 0.510038435459137, + 0.4899615943431854 + ], + "sources": 2 + }, + { + "entropy_mean": 1.098108172416687, + "mean_weights": [ + 0.3357090950012207, + 0.32087498903274536, + 0.34341591596603394 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098517656326294, + "mean_weights": [ + 0.33792540431022644, + 0.3309560716152191, + 0.33111852407455444 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098421573638916, + "mean_weights": [ + 0.34076738357543945, + 0.3312253952026367, + 0.3280072510242462 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985374450683594, + "mean_weights": [ + 0.3372262120246887, + 0.33111053705215454, + 0.33166325092315674 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0983846187591553, + "mean_weights": [ + 0.34108543395996094, + 0.3330540657043457, + 0.32586050033569336 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984745025634766, + "mean_weights": [ + 0.337818443775177, + 0.3331182599067688, + 0.3290632963180542 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982074737548828, + "mean_weights": [ + 0.34279391169548035, + 0.33254510164260864, + 0.324660986661911 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985103845596313, + "mean_weights": [ + 0.3366244435310364, + 0.33529800176620483, + 0.3280775547027588 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3860628604888916, + "mean_weights": [ + 0.2556775212287903, + 0.2528025507926941, + 0.24701416492462158, + 0.24450579285621643 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861674070358276, + "mean_weights": [ + 0.25406143069267273, + 0.25269144773483276, + 0.2465747445821762, + 0.24667240679264069 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859940767288208, + "mean_weights": [ + 0.25521907210350037, + 0.25419867038726807, + 0.2482832670211792, + 0.24229897558689117 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862324953079224, + "mean_weights": [ + 0.25186488032341003, + 0.2524600028991699, + 0.24845129251480103, + 0.24722380936145782 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859516382217407, + "mean_weights": [ + 0.25461074709892273, + 0.25498804450035095, + 0.2496698498725891, + 0.2407313585281372 + ], + "sources": 4 + }, + { + "entropy_mean": 1.386136531829834, + "mean_weights": [ + 0.25203216075897217, + 0.25430822372436523, + 0.2492649257183075, + 0.2443946897983551 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859035968780518, + "mean_weights": [ + 0.25416573882102966, + 0.25645381212234497, + 0.24988193809986115, + 0.23949852585792542 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861734867095947, + "mean_weights": [ + 0.249299094080925, + 0.25509464740753174, + 0.25041887164115906, + 0.2451874166727066 + ], + "sources": 4 + }, + { + "entropy_mean": 1.609163761138916, + "mean_weights": [ + 0.20298153162002563, + 0.20569300651550293, + 0.2007213830947876, + 0.194899320602417, + 0.19570477306842804 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093815565109253, + "mean_weights": [ + 0.1999782770872116, + 0.20289158821105957, + 0.1990451216697693, + 0.19804182648658752, + 0.20004317164421082 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6088870763778687, + "mean_weights": [ + 0.20441460609436035, + 0.20701850950717926, + 0.2030748575925827, + 0.19480165839195251, + 0.19069038331508636 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093982458114624, + "mean_weights": [ + 0.20016637444496155, + 0.20252978801727295, + 0.20030274987220764, + 0.1992998570203781, + 0.19770124554634094 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085177659988403, + "mean_weights": [ + 0.2029160112142563, + 0.21054993569850922, + 0.20384445786476135, + 0.1945992410182953, + 0.18809035420417786 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6092829704284668, + "mean_weights": [ + 0.20027539134025574, + 0.20464211702346802, + 0.2017458826303482, + 0.1985521912574768, + 0.19478441774845123 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085519790649414, + "mean_weights": [ + 0.20301291346549988, + 0.2110891491174698, + 0.20327432453632355, + 0.19483619928359985, + 0.18778741359710693 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093014478683472, + "mean_weights": [ + 0.20159733295440674, + 0.20404216647148132, + 0.20026597380638123, + 0.1987612247467041, + 0.1953333020210266 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7908084392547607, + "mean_weights": [ + 0.16949914395809174, + 0.176947221159935, + 0.17052626609802246, + 0.16260865330696106, + 0.15737155079841614, + 0.1630471646785736 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917122840881348, + "mean_weights": [ + 0.16739794611930847, + 0.1690039038658142, + 0.16775791347026825, + 0.1657543182373047, + 0.16472579538822174, + 0.16536012291908264 + ], + "sources": 6 + }, + { + "entropy_mean": 1.79024076461792, + "mean_weights": [ + 0.17057330906391144, + 0.18028055131435394, + 0.17232558131217957, + 0.16426734626293182, + 0.15759997069835663, + 0.1549532413482666 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917349338531494, + "mean_weights": [ + 0.16661964356899261, + 0.16852860152721405, + 0.16664528846740723, + 0.16634956002235413, + 0.16624407470226288, + 0.1656128317117691 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7899181842803955, + "mean_weights": [ + 0.17183387279510498, + 0.18008580803871155, + 0.1745484173297882, + 0.16428706049919128, + 0.1564694046974182, + 0.15277546644210815 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917158603668213, + "mean_weights": [ + 0.16592609882354736, + 0.16857831180095673, + 0.16695892810821533, + 0.1667860448360443, + 0.16571009159088135, + 0.1660405397415161 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7884674072265625, + "mean_weights": [ + 0.17281503975391388, + 0.1848573088645935, + 0.17561936378479004, + 0.16400930285453796, + 0.15400075912475586, + 0.14869821071624756 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458975791931152, + "mean_weights": [ + 0.14283010363578796, + 0.1433570683002472, + 0.14330893754959106, + 0.14272412657737732, + 0.14193668961524963, + 0.14225220680236816, + 0.14359086751937866 + ], + "sources": 7 + }, + { + "entropy_mean": 2.074883222579956, + "mean_weights": [ + 0.13161395490169525, + 0.1447027027606964, + 0.13560038805007935, + 0.1284974217414856, + 0.12059470266103745, + 0.11523722112178802, + 0.10903213918209076, + 0.11472146958112717 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0794177055358887, + "mean_weights": [ + 0.12457513064146042, + 0.12499764561653137, + 0.12486107647418976, + 0.1250150501728058, + 0.1245865523815155, + 0.12604494392871857, + 0.12620270252227783, + 0.12371689081192017 + ], + "sources": 8 + }, + { + "entropy_mean": 2.07416033744812, + "mean_weights": [ + 0.13240060210227966, + 0.1460423767566681, + 0.13675764203071594, + 0.12843625247478485, + 0.12032738327980042, + 0.11532401293516159, + 0.10900983959436417, + 0.11170189827680588 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0793137550354004, + "mean_weights": [ + 0.12426232546567917, + 0.12236000597476959, + 0.12353703379631042, + 0.12372594326734543, + 0.12466074526309967, + 0.12560483813285828, + 0.12777681648731232, + 0.12807230651378632 + ], + "sources": 8 + }, + { + "entropy_mean": 2.07352352142334, + "mean_weights": [ + 0.1330663114786148, + 0.1475958675146103, + 0.1372985988855362, + 0.1289202868938446, + 0.1197829395532608, + 0.11523041129112244, + 0.10973018407821655, + 0.10837538540363312 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0794036388397217, + "mean_weights": [ + 0.12454748153686523, + 0.1242402195930481, + 0.1247481107711792, + 0.12434204667806625, + 0.12395815551280975, + 0.12491720914840698, + 0.1262882649898529, + 0.12695851922035217 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0730199813842773, + "mean_weights": [ + 0.13274523615837097, + 0.14807099103927612, + 0.13688203692436218, + 0.12925183773040771, + 0.121518075466156, + 0.11719447374343872, + 0.1087818592786789, + 0.10555550456047058 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0792930126190186, + "mean_weights": [ + 0.12389382719993591, + 0.12217606604099274, + 0.12295391410589218, + 0.12413577735424042, + 0.12563353776931763, + 0.12554338574409485, + 0.12667298316955566, + 0.1289905160665512 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1897361278533936, + "mean_weights": [ + 0.1203719973564148, + 0.13450884819030762, + 0.12450436502695084, + 0.11663885414600372, + 0.10931286960840225, + 0.10511736571788788, + 0.0989350825548172, + 0.09487420320510864, + 0.09573642164468765 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197061538696289, + "mean_weights": [ + 0.10995423048734665, + 0.10847180336713791, + 0.10944779217243195, + 0.10998472571372986, + 0.1106867641210556, + 0.1111973375082016, + 0.11222705990076065, + 0.11497480422258377, + 0.11305546760559082 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1907856464385986, + "mean_weights": [ + 0.11950328946113586, + 0.1321493685245514, + 0.12339559942483902, + 0.11676183342933655, + 0.11050068587064743, + 0.10565406829118729, + 0.1002497524023056, + 0.09685426950454712, + 0.09493112564086914 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1971702575683594, + "mean_weights": [ + 0.1109868660569191, + 0.11009706556797028, + 0.10998459160327911, + 0.1099836528301239, + 0.10991828888654709, + 0.11259518563747406, + 0.11267602443695068, + 0.1119103878736496, + 0.11184795200824738 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1903560161590576, + "mean_weights": [ + 0.1198592483997345, + 0.1334088146686554, + 0.12379282712936401, + 0.11622756719589233, + 0.1098957434296608, + 0.10603700578212738, + 0.10054641962051392, + 0.09582884609699249, + 0.09440355002880096 + ], + "sources": 9 + }, + { + "entropy_mean": 2.196927547454834, + "mean_weights": [ + 0.10958623886108398, + 0.10723185539245605, + 0.1089189350605011, + 0.10959269851446152, + 0.11027678847312927, + 0.11107141524553299, + 0.1139138787984848, + 0.1143452376127243, + 0.11506295949220657 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1887855529785156, + "mean_weights": [ + 0.12046992033720016, + 0.13502465188503265, + 0.12495312094688416, + 0.11731088906526566, + 0.11063195019960403, + 0.10588784515857697, + 0.09945450723171234, + 0.09479314833879471, + 0.09147396683692932 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028617912903428078, + 0.014959256164729595, + 0.015274961479008198, + 0.014781977981328964, + 0.0157940573990345, + 0.015444205142557621, + 0.016597270965576172, + 0.01670028083026409, + 0.0178354624658823, + 0.01179081667214632, + 0.012896513566374779, + 0.012996076606214046, + 0.013999328017234802, + 0.01415199227631092, + 0.015286009758710861, + 0.015432553365826607, + 0.016473792493343353, + 0.01251901313662529, + 0.013480177149176598, + 0.013605917803943157, + 0.014498019590973854, + 0.014678623527288437, + 0.015643835067749023, + 0.015779949724674225, + 0.01692904904484749, + 0.013776812702417374, + 0.014660787768661976, + 0.014904298819601536, + 0.015783702954649925, + 0.016061274334788322, + 0.01696891337633133, + 0.017220500856637955, + 0.0182294063270092, + 0.015443956479430199, + 0.01628282479941845, + 0.01663891226053238, + 0.017581414431333542, + 0.017740750685334206, + 0.01882229372859001, + 0.01890667900443077, + 0.020223958417773247, + 0.017717810347676277, + 0.018474990501999855, + 0.018883828073740005, + 0.019671211019158363, + 0.020256539806723595, + 0.021038755774497986, + 0.021670879796147346, + 0.02247566729784012, + 0.019627301022410393, + 0.02105524018406868, + 0.020957279950380325, + 0.02264176495373249, + 0.022439396008849144, + 0.024370873346924782, + 0.023861775174736977, + 0.026266612112522125, + 0.02291625738143921, + 0.025011125952005386, + 0.024447107687592506, + 0.026532378047704697, + 0.0257034283131361, + 0.02849755622446537, + 0.02698364295065403 + ], + "loss_nats": 3.5927109718322754, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1923251152038574, + "mean_weights": [ + 0.09994596242904663, + 0.09316341578960419, + 0.10046449303627014, + 0.10793919861316681, + 0.11503102630376816, + 0.11786940693855286, + 0.12105783820152283, + 0.12264540791511536, + 0.12188324332237244 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.00272006937302649, + 0.005520553328096867, + 0.00624432135373354, + 0.009550899267196655, + 0.009988417848944664, + 0.013989307917654514, + 0.014321896247565746, + 0.018765948712825775, + 0.0027917264960706234, + 0.006620377767831087, + 0.007183594163507223, + 0.011785997077822685, + 0.01235294435173273, + 0.017189405858516693, + 0.017732884734869003, + 0.02231588587164879, + 0.0035560799296945333, + 0.007451951503753662, + 0.008640909567475319, + 0.013181107118725777, + 0.01409607008099556, + 0.01907873898744583, + 0.02024942636489868, + 0.02571253851056099, + 0.0048495507799088955, + 0.008935928344726562, + 0.011136100627481937, + 0.016163820400834084, + 0.018431898206472397, + 0.023612193763256073, + 0.02638358809053898, + 0.03195178136229515, + 0.005550710018724203, + 0.009834196418523788, + 0.01351640559732914, + 0.019116180017590523, + 0.022760765627026558, + 0.02841295301914215, + 0.03270368278026581, + 0.03848475590348244, + 0.008386691100895405, + 0.012644016183912754, + 0.017066271975636482, + 0.0226194616407156, + 0.029033595696091652, + 0.034674666821956635, + 0.04045802354812622, + 0.04606693610548973, + 0.009665686637163162, + 0.014916402287781239, + 0.023538386449217796, + 0.029746420681476593, + 0.040347639471292496, + 0.046650227159261703, + 0.056875914335250854, + 0.062961645424366, + 0.014480169862508774, + 0.019602956250309944, + 0.03171956166625023, + 0.0373276025056839, + 0.04853202775120735, + 0.055456433445215225, + 0.06785041838884354, + 0.07453885674476624 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00015286129200831056, + 0.0001596057554706931, + 0.0001548997388454154, + 0.00014926388394087553, + 0.00014973749057389796, + 0.00013395246060099453, + 0.00012230542779434472, + 0.00011163100862177089, + 0.00010697251855162904, + 9.841703285928816e-05, + 9.074148692889139e-05, + 8.466922008665279e-05, + 8.737866301089525e-05, + 8.072105993051082e-05, + 7.312748493859544e-05, + 6.66483465465717e-05, + 6.38419805909507e-05, + 5.7327142712892964e-05, + 5.1727347454288974e-05, + 4.712332884082571e-05, + 4.238562542013824e-05, + 3.858288619085215e-05, + 3.5554763599066064e-05, + 3.310271131340414e-05, + 3.4646320273168385e-05, + 3.30927869072184e-05, + 3.204917811672203e-05, + 3.1358391424873844e-05, + 2.4667071556905285e-05, + 2.4190361727960408e-05, + 2.406916428299155e-05, + 2.4089864382403903e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00014178213223203784, + "first_to_last_ratio": 4.971254622709509, + "imbalance_abs_log_ratio": 1.6036722474280931, + "last_quartile_mean": 2.8520392334030475e-05, + "mean": 7.56481811095e-05, + "normalized": [ + 2.0206869453615193, + 2.1098426046710275, + 2.047633354478141, + 1.9731324897926827, + 1.9793931377828424, + 1.7707294297942162, + 1.6167662725070522, + 1.4756601809128245, + 1.4140791884577815, + 1.3009834660377422, + 1.1995197451944546, + 1.119249912487584, + 1.1550662782548002, + 1.0670588340209777, + 0.9666786942668737, + 0.8810303905403737, + 0.8439327906448941, + 0.7578125722535547, + 0.6837883832185488, + 0.6229274537693794, + 0.5602993330240876, + 0.510030586657514, + 0.4700015661658922, + 0.4375876700259097, + 0.4579927734550309, + 0.4374564784223551, + 0.4236609214745185, + 0.4145293510690347, + 0.3260762016366255, + 0.3197745322249731, + 0.3181724124754787, + 0.3184460489213093 + ], + "population_cv": 0.5949029050557015 + }, + "activation_output_rms_by_block": [ + 0.021161425858736038, + 0.03683347627520561, + 0.048924874514341354, + 0.05841962620615959, + 0.017047826200723648, + 0.025985000655055046, + 0.03293692693114281, + 0.038455791771411896, + 0.01630864106118679, + 0.028415540233254433, + 0.032696306705474854, + 0.03946299850940704, + 0.018893985077738762, + 0.02713344618678093, + 0.036225005984306335, + 0.04083120450377464, + 0.020795846357941628, + 0.031739242374897, + 0.042166925966739655, + 0.05129452794790268, + 0.021753763779997826, + 0.036568306386470795, + 0.04833273962140083, + 0.06496113538742065, + 0.026428475975990295, + 0.05308837816119194, + 0.0783136710524559, + 0.10490671545267105, + 0.02600686252117157, + 0.055555470287799835, + 0.08636210113763809, + 0.12206175923347473 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.034970618551597, + "first_to_last_ratio": 0.5061572050203191, + "imbalance_abs_log_ratio": 0.6809079760879482, + "last_quartile_mean": 0.06909042922779918, + "mean": 0.04343962494749576, + "normalized": [ + 0.4871456851736923, + 0.8479234413217239, + 1.126272949489676, + 1.344846468558827, + 0.3924487428546833, + 0.5981865793376987, + 0.7582230963308899, + 0.8852698847628707, + 0.3754323634446334, + 0.6541387101661096, + 0.7526839088733834, + 0.9084562437430077, + 0.43494816312469053, + 0.6246243198364708, + 0.8339161774092311, + 0.9399529704302503, + 0.4787298781487404, + 0.7306518510060648, + 0.9707018883727846, + 1.18082345346906, + 0.500781574571396, + 0.8418191094115077, + 1.1126417338966266, + 1.4954349966404483, + 0.6083955837080464, + 1.2221187044169548, + 1.8028164641640303, + 2.415000488136553, + 0.5986898494774143, + 1.2789122915989297, + 1.9880949994854125, + 2.809917428638192 + ], + "population_cv": 0.568563140978506 + }, + "bits_per_byte": 3.067092504136898, + "branch_output_rms_by_sublayer": [ + 0.010234483517706394, + 0.017091969028115273, + 0.015844203531742096, + 0.01249503530561924, + 0.011603006161749363, + 0.01348696555942297, + 0.01365630142390728, + 0.012086901813745499, + 0.01432340033352375, + 0.008346238173544407, + 0.013000942766666412, + 0.00798263493925333, + 0.015437191352248192, + 0.007602585479617119, + 0.014045360498130322, + 0.008352738805115223, + 0.013383014127612114, + 0.00802657287567854, + 0.016149764880537987, + 0.007843135856091976, + 0.015759345144033432, + 0.007903078570961952, + 0.01248305942863226, + 0.00855498667806387, + 0.015984252095222473, + 0.008713160641491413, + 0.011801399290561676, + 0.008602410554885864, + 0.01826307363808155, + 0.009256557561457157, + 0.020604459568858147, + 0.009156917221844196, + 0.017283903434872627, + 0.010396110825240612, + 0.015061315149068832, + 0.011026599444448948, + 0.016859732568264008, + 0.011797916144132614, + 0.019787220284342766, + 0.013876549899578094, + 0.016343828290700912, + 0.013472933322191238, + 0.01708521693944931, + 0.015084017999470234, + 0.0159829743206501, + 0.015682615339756012, + 0.013441185466945171, + 0.017962589859962463, + 0.012103093788027763, + 0.022373178973793983, + 0.018175961449742317, + 0.023165129125118256, + 0.012037546373903751, + 0.024680674076080322, + 0.006721542216837406, + 0.027424708008766174, + 0.005014357157051563, + 0.024616269394755363, + 0.00892665982246399, + 0.027556434273719788, + 0.0051607671193778515, + 0.0312117300927639, + 0.010016578249633312, + 0.03412666916847229 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.000676480037560686, + 0.0005110694211201519, + 0.00047228012773426685, + 0.0004129357088210776, + 0.0004282409583756997, + 0.0004154345635765322, + 0.000313826928744342, + 0.0004120928901703585, + 0.0003453507961863634, + 0.0002726503080695178, + 0.0002510530752377891, + 0.0002178222294286696, + 0.00023730041780366525, + 0.0002049813617215844, + 0.0001925568974060751, + 0.0001910139020414276, + 0.0001821130386192414, + 0.0001494933742046571, + 0.0001389964762696074, + 0.00013670952969881389, + 0.0001199375815159, + 0.0001158431254601814, + 0.00010874733490020777, + 0.00010248224866112548, + 0.0001196930668964626, + 0.00012146618943708263, + 0.00011605054339173233, + 0.00010928146823787377, + 8.100500100913196e-05, + 8.185756253411311e-05, + 8.573691911994455e-05, + 9.643895805802721e-05 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00045529507951288937, + "first_to_last_ratio": 4.488265305786549, + "imbalance_abs_log_ratio": 1.501466280953272, + "last_quartile_mean": 0.00010144121358554603, + "mean": 0.00023190443881288466, + "normalized": [ + 2.917063774301075, + 2.2037931819516206, + 2.0365290554672506, + 1.7806287405919838, + 1.8466268285672267, + 1.7914041046524831, + 1.3532596889943864, + 1.7769944045912007, + 1.4891944197110194, + 1.175701118379668, + 1.0825712371997966, + 0.9392758092242577, + 1.0232681143077835, + 0.88390443395944, + 0.8303286405028337, + 0.8236750577920151, + 0.7852934577340361, + 0.6446335178831049, + 0.5993696238895998, + 0.5895080335617029, + 0.5171853636345155, + 0.49952957370363676, + 0.4689316662366781, + 0.44191585631448266, + 0.5161309870098633, + 0.52377690594827, + 0.5004239849215191, + 0.4712349138174498, + 0.34930336574752663, + 0.35297971420099045, + 0.3697079691912348, + 0.41585645601135013 + ], + "population_cv": 0.6467057266129946 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6465992331504822, + "mean_weights": [ + 0.37046489119529724, + 0.6295351386070251 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6799579858779907, + "mean_weights": [ + 0.4533712863922119, + 0.5466287136077881 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6899222135543823, + "mean_weights": [ + 0.485215425491333, + 0.514784574508667 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6784638166427612, + "mean_weights": [ + 0.4707837700843811, + 0.5292162299156189 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6813141703605652, + "mean_weights": [ + 0.4505807161331177, + 0.5494192838668823 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6827169060707092, + "mean_weights": [ + 0.46454811096191406, + 0.5354518890380859 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6873244643211365, + "mean_weights": [ + 0.5028640627861023, + 0.49713587760925293 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6849884390830994, + "mean_weights": [ + 0.46853116154670715, + 0.5314688086509705 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0892460346221924, + "mean_weights": [ + 0.28484630584716797, + 0.35620802640914917, + 0.35894566774368286 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0834758281707764, + "mean_weights": [ + 0.29898303747177124, + 0.31943249702453613, + 0.3815844655036926 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0875498056411743, + "mean_weights": [ + 0.28361061215400696, + 0.3391039967536926, + 0.3772854208946228 + ], + "sources": 3 + }, + { + "entropy_mean": 1.067812442779541, + "mean_weights": [ + 0.29054006934165955, + 0.29684174060821533, + 0.4126181900501251 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0913453102111816, + "mean_weights": [ + 0.30610108375549316, + 0.3296392560005188, + 0.36425966024398804 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0731472969055176, + "mean_weights": [ + 0.29320991039276123, + 0.30639368295669556, + 0.4003963768482208 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0834020376205444, + "mean_weights": [ + 0.27801185846328735, + 0.3334004282951355, + 0.38858771324157715 + ], + "sources": 3 + }, + { + "entropy_mean": 1.055156946182251, + "mean_weights": [ + 0.2844427227973938, + 0.2930121421813965, + 0.42254507541656494 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3796665668487549, + "mean_weights": [ + 0.22888872027397156, + 0.23129937052726746, + 0.25495558977127075, + 0.28485631942749023 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3535382747650146, + "mean_weights": [ + 0.22840344905853271, + 0.21709680557250977, + 0.23951464891433716, + 0.31498509645462036 + ], + "sources": 4 + }, + { + "entropy_mean": 1.38047456741333, + "mean_weights": [ + 0.229506254196167, + 0.24313977360725403, + 0.2601771056652069, + 0.26717686653137207 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3494740724563599, + "mean_weights": [ + 0.2434343844652176, + 0.2201111912727356, + 0.22082674503326416, + 0.31562769412994385 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3820511102676392, + "mean_weights": [ + 0.24899566173553467, + 0.2345060110092163, + 0.24331089854240417, + 0.27318739891052246 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3373212814331055, + "mean_weights": [ + 0.239411860704422, + 0.2179979830980301, + 0.21153053641319275, + 0.33105963468551636 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3797708749771118, + "mean_weights": [ + 0.24076583981513977, + 0.2308681458234787, + 0.24815474450588226, + 0.28021126985549927 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3376049995422363, + "mean_weights": [ + 0.24594180285930634, + 0.21920382976531982, + 0.2080971598625183, + 0.3267572224140167 + ], + "sources": 4 + }, + { + "entropy_mean": 1.60691499710083, + "mean_weights": [ + 0.1933879852294922, + 0.19704721868038177, + 0.19036617875099182, + 0.20468607544898987, + 0.21451255679130554 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5662117004394531, + "mean_weights": [ + 0.19762277603149414, + 0.17512229084968567, + 0.16398051381111145, + 0.21042925119400024, + 0.2528451681137085 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6042964458465576, + "mean_weights": [ + 0.21364352107048035, + 0.18369734287261963, + 0.18157300353050232, + 0.20280620455741882, + 0.2182799130678177 + ], + "sources": 5 + }, + { + "entropy_mean": 1.560100793838501, + "mean_weights": [ + 0.1962088793516159, + 0.16914671659469604, + 0.16035029292106628, + 0.20224899053573608, + 0.2720451354980469 + ], + "sources": 5 + }, + { + "entropy_mean": 1.599726915359497, + "mean_weights": [ + 0.20235487818717957, + 0.18438228964805603, + 0.1783113330602646, + 0.19914957880973816, + 0.23580196499824524 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5747346878051758, + "mean_weights": [ + 0.20395107567310333, + 0.1738116443157196, + 0.16936205327510834, + 0.19264253973960876, + 0.26023268699645996 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5999796390533447, + "mean_weights": [ + 0.20090287923812866, + 0.18981856107711792, + 0.18123285472393036, + 0.19229191541671753, + 0.23575380444526672 + ], + "sources": 5 + }, + { + "entropy_mean": 1.573183536529541, + "mean_weights": [ + 0.2065228521823883, + 0.17315250635147095, + 0.1667284220457077, + 0.19133973121643066, + 0.2622564733028412 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7820056676864624, + "mean_weights": [ + 0.15741385519504547, + 0.16548186540603638, + 0.1547827571630478, + 0.15221598744392395, + 0.17495308816432953, + 0.19515244662761688 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7595186233520508, + "mean_weights": [ + 0.17041948437690735, + 0.14349965751171112, + 0.13848072290420532, + 0.14705434441566467, + 0.18234138190746307, + 0.21820437908172607 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7837495803833008, + "mean_weights": [ + 0.1555388867855072, + 0.16459567844867706, + 0.1546156108379364, + 0.15561310946941376, + 0.17284435033798218, + 0.1967923790216446 + ], + "sources": 6 + }, + { + "entropy_mean": 1.772802710533142, + "mean_weights": [ + 0.16644072532653809, + 0.1547652930021286, + 0.14541912078857422, + 0.15002214908599854, + 0.17192085087299347, + 0.2114318609237671 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7812073230743408, + "mean_weights": [ + 0.14766940474510193, + 0.1754523664712906, + 0.16177824139595032, + 0.1518140286207199, + 0.16747768223285675, + 0.1958082765340805 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7798246145248413, + "mean_weights": [ + 0.16764245927333832, + 0.15854625403881073, + 0.15431074798107147, + 0.15243344008922577, + 0.16363412141799927, + 0.20343300700187683 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7723883390426636, + "mean_weights": [ + 0.13868887722492218, + 0.18229368329048157, + 0.16530905663967133, + 0.15040190517902374, + 0.1618221253156662, + 0.20148436725139618 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9409923553466797, + "mean_weights": [ + 0.14198631048202515, + 0.15129181742668152, + 0.14336252212524414, + 0.13319280743598938, + 0.13481390476226807, + 0.145244762301445, + 0.15010784566402435 + ], + "sources": 7 + }, + { + "entropy_mean": 2.0634469985961914, + "mean_weights": [ + 0.10111280530691147, + 0.1545264571905136, + 0.1330452263355255, + 0.12264932692050934, + 0.12621736526489258, + 0.12829245626926422, + 0.11007536947727203, + 0.12408097833395004 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0642125606536865, + "mean_weights": [ + 0.11415836215019226, + 0.15084657073020935, + 0.13143493235111237, + 0.12089236080646515, + 0.11884783953428268, + 0.12703002989292145, + 0.11854374408721924, + 0.1182461753487587 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0539355278015137, + "mean_weights": [ + 0.10209279507398605, + 0.1648375689983368, + 0.14242276549339294, + 0.12545765936374664, + 0.12551307678222656, + 0.12703318893909454, + 0.11218568682670593, + 0.10045725107192993 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0658535957336426, + "mean_weights": [ + 0.1146954894065857, + 0.14761218428611755, + 0.1362922489643097, + 0.12288784980773926, + 0.12050606310367584, + 0.12361439317464828, + 0.12056631594896317, + 0.11382545530796051 + ], + "sources": 8 + }, + { + "entropy_mean": 2.038452625274658, + "mean_weights": [ + 0.09628117084503174, + 0.17527475953102112, + 0.1463599056005478, + 0.13235297799110413, + 0.128395676612854, + 0.12688350677490234, + 0.10560917854309082, + 0.08884280920028687 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0657854080200195, + "mean_weights": [ + 0.11124911904335022, + 0.14782804250717163, + 0.13756024837493896, + 0.13235944509506226, + 0.12761159241199493, + 0.12093794345855713, + 0.11316381394863129, + 0.10928982496261597 + ], + "sources": 8 + }, + { + "entropy_mean": 2.034518241882324, + "mean_weights": [ + 0.09550792723894119, + 0.18016761541366577, + 0.1520727276802063, + 0.13090895116329193, + 0.12566842138767242, + 0.12382014095783234, + 0.10618985444307327, + 0.08566434681415558 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0708389282226562, + "mean_weights": [ + 0.11836373060941696, + 0.13942141830921173, + 0.13403482735157013, + 0.12536101043224335, + 0.12324266135692596, + 0.12180263549089432, + 0.12098512053489685, + 0.1167885959148407 + ], + "sources": 8 + }, + { + "entropy_mean": 2.150324821472168, + "mean_weights": [ + 0.08731061220169067, + 0.16436058282852173, + 0.13623733818531036, + 0.11943905800580978, + 0.11650224775075912, + 0.11330465227365494, + 0.09659627079963684, + 0.07463054358959198, + 0.09161868691444397 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1769137382507324, + "mean_weights": [ + 0.1032157838344574, + 0.13504555821418762, + 0.12166200578212738, + 0.11436311155557632, + 0.11372028291225433, + 0.11937251687049866, + 0.10532563924789429, + 0.09130994975566864, + 0.09598515927791595 + ], + "sources": 9 + }, + { + "entropy_mean": 2.154665946960449, + "mean_weights": [ + 0.0899716392159462, + 0.16547058522701263, + 0.1376831978559494, + 0.11778485774993896, + 0.11428935825824738, + 0.11093620955944061, + 0.10110829025506973, + 0.08096955716609955, + 0.08178631961345673 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1876935958862305, + "mean_weights": [ + 0.1051335334777832, + 0.12856584787368774, + 0.11881857365369797, + 0.11346863210201263, + 0.11117258667945862, + 0.11112762242555618, + 0.110337033867836, + 0.10235048830509186, + 0.0990256816148758 + ], + "sources": 9 + }, + { + "entropy_mean": 2.139727830886841, + "mean_weights": [ + 0.08707605302333832, + 0.1715715527534485, + 0.14120736718177795, + 0.12170714139938354, + 0.11733704805374146, + 0.11485080420970917, + 0.09878863394260406, + 0.07554350793361664, + 0.07191790640354156 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1848366260528564, + "mean_weights": [ + 0.10987311601638794, + 0.12207551300525665, + 0.11300010979175568, + 0.1114986389875412, + 0.1146916002035141, + 0.11859980970621109, + 0.11078428477048874, + 0.10057757794857025, + 0.09889934957027435 + ], + "sources": 9 + }, + { + "entropy_mean": 2.134784698486328, + "mean_weights": [ + 0.08643749356269836, + 0.17323024570941925, + 0.14556987583637238, + 0.1249169111251831, + 0.11791284382343292, + 0.11342346668243408, + 0.09696559607982635, + 0.07453062385320663, + 0.06701293587684631 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.043852195143699646, + 0.016972972080111504, + 0.023848671466112137, + 0.026571517810225487, + 0.02969934232532978, + 0.03087676502764225, + 0.034429069608449936, + 0.03451152518391609, + 0.03785692900419235, + 0.02532852627336979, + 0.026435313746333122, + 0.027248572558164597, + 0.02803654409945011, + 0.028617987409234047, + 0.029994800686836243, + 0.030634304508566856, + 0.031594887375831604, + 0.02266915887594223, + 0.02385900355875492, + 0.023341434076428413, + 0.024995362386107445, + 0.024166349321603775, + 0.025874190032482147, + 0.02476029470562935, + 0.026863638311624527, + 0.020435020327568054, + 0.021791530773043633, + 0.02066263183951378, + 0.022244635969400406, + 0.021215319633483887, + 0.022682856768369675, + 0.02177605964243412, + 0.023207761347293854, + 0.018326645717024803, + 0.019081639125943184, + 0.01884583942592144, + 0.019717859104275703, + 0.019695758819580078, + 0.020406607538461685, + 0.020676106214523315, + 0.02074965462088585, + 0.0176867488771677, + 0.018479831516742706, + 0.018484534695744514, + 0.019486455246806145, + 0.019449569284915924, + 0.020568842068314552, + 0.020608482882380486, + 0.022208938375115395, + 0.019393013790249825, + 0.02096123807132244, + 0.02112950012087822, + 0.022736722603440285, + 0.02220955677330494, + 0.024083420634269714, + 0.02331419661641121, + 0.027064917609095573, + 0.022320950403809547, + 0.02527429535984993, + 0.024373378604650497, + 0.028173813596367836, + 0.02539936453104019, + 0.030857285484671593, + 0.02675630897283554 + ], + "loss_nats": 2.125946521759033, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1591250896453857, + "mean_weights": [ + 0.12719565629959106, + 0.08464579284191132, + 0.08970484137535095, + 0.08979810774326324, + 0.09367705881595612, + 0.0977322906255722, + 0.12260590493679047, + 0.1657724529504776, + 0.12886787950992584 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.010234483517706394, + 0.021161425858736038, + 0.02897210791707039, + 0.03683347627520561, + 0.04058066010475159, + 0.048924874514341354, + 0.050814058631658554, + 0.05841962620615959, + 0.01432340033352375, + 0.017047826200723648, + 0.02261594496667385, + 0.025985000655055046, + 0.029342494904994965, + 0.03293692693114281, + 0.03476514294743538, + 0.038455791771411896, + 0.013383014127612114, + 0.01630864106118679, + 0.02523905225098133, + 0.028415540233254433, + 0.028834128752350807, + 0.032696306705474854, + 0.03512699902057648, + 0.03946299850940704, + 0.015984252095222473, + 0.018893985077738762, + 0.023144295439124107, + 0.02713344618678093, + 0.03201613202691078, + 0.036225005984306335, + 0.03601640462875366, + 0.04083120450377464, + 0.017283903434872627, + 0.020795846357941628, + 0.026569250971078873, + 0.031739242374897, + 0.03601877763867378, + 0.042166925966739655, + 0.04336150363087654, + 0.05129452794790268, + 0.016343828290700912, + 0.021753763779997826, + 0.02770911157131195, + 0.036568306386470795, + 0.03726854547858238, + 0.04833273962140083, + 0.05117516592144966, + 0.06496113538742065, + 0.012103093788027763, + 0.026428475975990295, + 0.03489123284816742, + 0.05308837816119194, + 0.0573502741754055, + 0.0783136710524559, + 0.08074422925710678, + 0.10490671545267105, + 0.005014357157051563, + 0.02600686252117157, + 0.030181895941495895, + 0.055555470287799835, + 0.05713089555501938, + 0.08636210113763809, + 0.09012933820486069, + 0.12206175923347473 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.492746954085305e-05, + 7.713511149631813e-05, + 7.584479317301884e-05, + 7.141788955777884e-05, + 8.826405246509239e-05, + 9.460418368689716e-05, + 8.256193541456014e-05, + 7.497850310755894e-05, + 7.441766501870006e-05, + 6.938254955457523e-05, + 6.46677945042029e-05, + 6.093592310207896e-05, + 5.3101273806532845e-05, + 5.207999856793322e-05, + 5.16299951414112e-05, + 5.217096986598335e-05, + 7.235184602905065e-05, + 6.494282570201904e-05, + 5.741609857068397e-05, + 5.24955648870673e-05, + 5.885101199964993e-05, + 5.565655374084599e-05, + 5.275748117128387e-05, + 5.083094583824277e-05, + 8.288460230687633e-05, + 6.546315853483975e-05, + 5.173764657229185e-05, + 4.3963729694951326e-05, + 2.6678681024350226e-05, + 2.3330803742283024e-05, + 2.2579977667191997e-05, + 2.250970464956481e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 7.996674230525969e-05, + "first_to_last_ratio": 1.8862955542872177, + "imbalance_abs_log_ratio": 0.6346148815238539, + "last_quartile_mean": 4.2393538024043664e-05, + "mean": 6.0080335629209e-05, + "normalized": [ + 1.2471213543691637, + 1.2838661883043423, + 1.2623896384517816, + 1.188706567795169, + 1.4691005225040958, + 1.5746280824853423, + 1.3741923135066734, + 1.2479707764999062, + 1.2386359736399466, + 1.1548295932095927, + 1.0763554135800073, + 1.0142407239225542, + 0.8838378356314778, + 0.8668393414003117, + 0.8593493128941587, + 0.8683535023499371, + 1.2042516951898594, + 1.0809331376379006, + 0.9556554231825933, + 0.8737561855687397, + 0.9795386690722576, + 0.9263688885550713, + 0.8781156200072057, + 0.8460496318121515, + 1.3795629042155462, + 1.0895937555817148, + 0.8611411043306286, + 0.7317490695504315, + 0.4440501329586442, + 0.3883267877574969, + 0.375829752459212, + 0.3746601015760864 + ], + "population_cv": 0.30469905986026097 + }, + "activation_output_rms_by_block": [ + 0.091498002409935, + 0.1524418443441391, + 0.18927344679832458, + 0.20122204720973969, + 0.09716148674488068, + 0.11803841590881348, + 0.12927795946598053, + 0.13284066319465637, + 0.04031922295689583, + 0.06599447876214981, + 0.09237803518772125, + 0.12235672026872635, + 0.07300658524036407, + 0.10997047275304794, + 0.13365790247917175, + 0.15966735780239105, + 0.044320181012153625, + 0.07193361222743988, + 0.09863750636577606, + 0.13064858317375183, + 0.045498576015233994, + 0.07957836240530014, + 0.11307200789451599, + 0.1419886350631714, + 0.0492420569062233, + 0.07780471444129944, + 0.10795509815216064, + 0.1515592485666275, + 0.06438802927732468, + 0.14722995460033417, + 0.23145347833633423, + 0.344204843044281 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.13896923325955868, + "first_to_last_ratio": 0.9471105998032673, + "imbalance_abs_log_ratio": 0.05433940296045832, + "last_quartile_mean": 0.14672967791557312, + "mean": 0.11901936028152704, + "normalized": [ + 0.768765704953435, + 1.280815524327764, + 1.5902744423312252, + 1.6906665162186332, + 0.8163502686878504, + 0.9917581108620207, + 1.0861926930222774, + 1.1161265098420676, + 0.3387618859782576, + 0.5544852417795493, + 0.7761597354347335, + 1.028040479963135, + 0.6134009212255479, + 0.9239712975512978, + 1.122992949743797, + 1.3415242480275116, + 0.37237791320100505, + 0.604385807966779, + 0.8287517772945513, + 1.0977086657558666, + 0.3822787814319635, + 0.6686169562419625, + 0.9500303784783983, + 1.1929877183620639, + 0.4137314869593207, + 0.653714775959651, + 0.9070381314166441, + 1.2733999595370953, + 0.5409878621849585, + 1.2370252558245827, + 1.9446708316097212, + 2.8920071678263333 + ], + "population_cv": 0.512118916980833 + }, + "bits_per_byte": 2.070896585643497, + "branch_output_rms_by_sublayer": [ + 0.04483851417899132, + 0.0769026055932045, + 0.07648502290248871, + 0.02227567695081234, + 0.05391820892691612, + 0.008645663969218731, + 0.034694772213697433, + 0.024034852162003517, + 0.0951480120420456, + 0.010152613744139671, + 0.03470344468951225, + 0.006408527493476868, + 0.03328023850917816, + 0.009709968231618404, + 0.03295121714472771, + 0.018027858808636665, + 0.03393593803048134, + 0.020053111016750336, + 0.028536589816212654, + 0.021738557144999504, + 0.02645471878349781, + 0.027084683999419212, + 0.02775159664452076, + 0.029649054631590843, + 0.0624675415456295, + 0.028460506349802017, + 0.03417849540710449, + 0.026108473539352417, + 0.033046722412109375, + 0.026846151798963547, + 0.03612860292196274, + 0.026833457872271538, + 0.03177119418978691, + 0.028621261939406395, + 0.02923821099102497, + 0.03319534659385681, + 0.028336195275187492, + 0.03987955302000046, + 0.029424920678138733, + 0.05270272493362427, + 0.03732937201857567, + 0.026832399889826775, + 0.04573337361216545, + 0.03161422163248062, + 0.04859265685081482, + 0.03612437844276428, + 0.03604976460337639, + 0.04247966781258583, + 0.023885536938905716, + 0.04003679007291794, + 0.02049342356622219, + 0.041669707745313644, + 0.015570039860904217, + 0.047472283244132996, + 0.01260604988783598, + 0.06517942249774933, + 0.013619035482406616, + 0.06221276521682739, + 0.021267028525471687, + 0.0901259332895279, + 0.014324037358164787, + 0.09489317983388901, + 0.018173350021243095, + 0.1292373090982437 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00026604026419561135, + 0.00016295364970886383, + 0.00013531488693005372, + 0.0001535454595092414, + 0.00017632706475316363, + 0.00014222709253675864, + 0.0001286399817301883, + 0.00015205345731464097, + 0.0001904662739914667, + 0.00016940698852805803, + 0.00016848411565751776, + 0.0001832549889608726, + 0.00015364818549896595, + 0.00013487143345347673, + 0.0001563477967298781, + 0.0001657491060780051, + 0.00023095741315168052, + 0.0002602707272097987, + 0.00025633077913206337, + 0.00025657064458866713, + 0.00020581571860629118, + 0.0002529129705783329, + 0.00023730419062687903, + 0.00023402594942544908, + 0.0003707127327738903, + 0.0002903920661674258, + 0.0002396897424931522, + 0.00021849163975841562, + 0.00012661973251191643, + 0.00012861896151560002, + 0.00012385693034570536, + 0.0001317298463083473 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00016463773208481522, + "first_to_last_ratio": 0.8079826036234979, + "imbalance_abs_log_ratio": 0.21321475086175729, + "last_quartile_mean": 0.00020376395648430664, + "mean": 0.0001938634622115743, + "normalized": [ + 1.3723074021306105, + 0.8405588544117879, + 0.6979906651124224, + 0.7920288730924883, + 0.909542534429349, + 0.7336456850313448, + 0.6635597046636674, + 0.7843327235572441, + 0.9824763873431701, + 0.8738469157389466, + 0.8690864886836767, + 0.9452786351296869, + 0.7925587614404661, + 0.6957032125335914, + 0.8064840839334995, + 0.8549785719658385, + 1.191340598774737, + 1.3425465750000398, + 1.3222232606798021, + 1.3234605513681421, + 1.0616529605855933, + 1.3045932827832947, + 1.2240789879626486, + 1.2071689360940183, + 1.9122362127697392, + 1.4979205614851978, + 1.2363843075884255, + 1.127038779076189, + 0.6531387145749469, + 0.6634512767301699, + 0.6388874362025639, + 0.679498059126701 + ], + "population_cv": 0.3030429126978485 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.40725070238113403, + "mean_weights": [ + 0.8447163105010986, + 0.1552836298942566 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6303530931472778, + "mean_weights": [ + 0.5865391492843628, + 0.41346079111099243 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5607507824897766, + "mean_weights": [ + 0.7307177782058716, + 0.2692822515964508 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6765038967132568, + "mean_weights": [ + 0.5413137674331665, + 0.4586862027645111 + ], + "sources": 2 + }, + { + "entropy_mean": 0.670610785484314, + "mean_weights": [ + 0.5058683753013611, + 0.4941316246986389 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6892119646072388, + "mean_weights": [ + 0.5047469735145569, + 0.4952530264854431 + ], + "sources": 2 + }, + { + "entropy_mean": 0.576317310333252, + "mean_weights": [ + 0.7205114364624023, + 0.27948853373527527 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6848311424255371, + "mean_weights": [ + 0.5329695343971252, + 0.46703049540519714 + ], + "sources": 2 + }, + { + "entropy_mean": 1.053565502166748, + "mean_weights": [ + 0.278648316860199, + 0.2728675603866577, + 0.4484841227531433 + ], + "sources": 3 + }, + { + "entropy_mean": 1.081671953201294, + "mean_weights": [ + 0.30949199199676514, + 0.28478163480758667, + 0.4057263731956482 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9490886926651001, + "mean_weights": [ + 0.21884064376354218, + 0.2031792849302292, + 0.5779800415039062 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0769644975662231, + "mean_weights": [ + 0.33828768134117126, + 0.2510800361633301, + 0.41063231229782104 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0134786367416382, + "mean_weights": [ + 0.3179722726345062, + 0.19739001989364624, + 0.48463770747184753 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0799745321273804, + "mean_weights": [ + 0.3330957889556885, + 0.257799357175827, + 0.4091048836708069 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9970211982727051, + "mean_weights": [ + 0.2562355399131775, + 0.21757182478904724, + 0.5261926054954529 + ], + "sources": 3 + }, + { + "entropy_mean": 1.084675908088684, + "mean_weights": [ + 0.3783033490180969, + 0.2626742422580719, + 0.3590223789215088 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3473670482635498, + "mean_weights": [ + 0.267598032951355, + 0.16219636797904968, + 0.2368270754814148, + 0.33337855339050293 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3785642385482788, + "mean_weights": [ + 0.2764087915420532, + 0.21344706416130066, + 0.23307646811008453, + 0.2770676612854004 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3698399066925049, + "mean_weights": [ + 0.2425079047679901, + 0.20580732822418213, + 0.26610618829727173, + 0.2855786085128784 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3757984638214111, + "mean_weights": [ + 0.27757763862609863, + 0.21923497319221497, + 0.21938714385032654, + 0.28380024433135986 + ], + "sources": 4 + }, + { + "entropy_mean": 1.342599868774414, + "mean_weights": [ + 0.30494293570518494, + 0.1875782459974289, + 0.1876712143421173, + 0.31980758905410767 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3775323629379272, + "mean_weights": [ + 0.26181769371032715, + 0.209392249584198, + 0.24447140097618103, + 0.2843186855316162 + ], + "sources": 4 + }, + { + "entropy_mean": 1.358901858329773, + "mean_weights": [ + 0.260298490524292, + 0.1959088146686554, + 0.21282958984375, + 0.3309631049633026 + ], + "sources": 4 + }, + { + "entropy_mean": 1.376914620399475, + "mean_weights": [ + 0.26105737686157227, + 0.21985405683517456, + 0.23669737577438354, + 0.28239119052886963 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5840137004852295, + "mean_weights": [ + 0.16954834759235382, + 0.16273891925811768, + 0.19214710593223572, + 0.26211944222450256, + 0.2134462296962738 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5902224779129028, + "mean_weights": [ + 0.19109177589416504, + 0.17331960797309875, + 0.16441664099693298, + 0.20784860849380493, + 0.2633233368396759 + ], + "sources": 5 + }, + { + "entropy_mean": 1.596144676208496, + "mean_weights": [ + 0.21844935417175293, + 0.16117408871650696, + 0.1771986484527588, + 0.23071235418319702, + 0.2124655842781067 + ], + "sources": 5 + }, + { + "entropy_mean": 1.556012749671936, + "mean_weights": [ + 0.20308932662010193, + 0.16124466061592102, + 0.12989482283592224, + 0.19262486696243286, + 0.31314635276794434 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5915946960449219, + "mean_weights": [ + 0.1935623288154602, + 0.15926021337509155, + 0.184815913438797, + 0.23437398672103882, + 0.22798755764961243 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5651438236236572, + "mean_weights": [ + 0.21180111169815063, + 0.19541677832603455, + 0.12937679886817932, + 0.16290953755378723, + 0.30049577355384827 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5943063497543335, + "mean_weights": [ + 0.1853662133216858, + 0.16870340704917908, + 0.19402793049812317, + 0.2201918363571167, + 0.23171064257621765 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5861427783966064, + "mean_weights": [ + 0.21089500188827515, + 0.19157911837100983, + 0.13813158869743347, + 0.19270305335521698, + 0.2666912078857422 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7657980918884277, + "mean_weights": [ + 0.13579657673835754, + 0.1306893527507782, + 0.16402600705623627, + 0.18097393214702606, + 0.16079631447792053, + 0.2277178168296814 + ], + "sources": 6 + }, + { + "entropy_mean": 1.758882999420166, + "mean_weights": [ + 0.15976183116436005, + 0.14843232929706573, + 0.11410017311573029, + 0.14428351819515228, + 0.18656599521636963, + 0.24685615301132202 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7183139324188232, + "mean_weights": [ + 0.11669094860553741, + 0.12010741233825684, + 0.1611485630273819, + 0.18112441897392273, + 0.1298353374004364, + 0.29109328985214233 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7624797821044922, + "mean_weights": [ + 0.1648901402950287, + 0.16732367873191833, + 0.11384207010269165, + 0.13662001490592957, + 0.1801655888557434, + 0.23715850710868835 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6784703731536865, + "mean_weights": [ + 0.11376311630010605, + 0.1189732626080513, + 0.14842449128627777, + 0.1726904809474945, + 0.11740631610155106, + 0.3287423253059387 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7734556198120117, + "mean_weights": [ + 0.16557791829109192, + 0.17164838314056396, + 0.13243623077869415, + 0.13632801175117493, + 0.171006977558136, + 0.22300247848033905 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6679855585098267, + "mean_weights": [ + 0.1163882240653038, + 0.1227155327796936, + 0.1510334014892578, + 0.1780768185853958, + 0.11049839854240417, + 0.3212876319885254 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9353042840957642, + "mean_weights": [ + 0.1553524136543274, + 0.1700681447982788, + 0.1242346316576004, + 0.11826618760824203, + 0.1569168120622635, + 0.13176673650741577, + 0.1433950662612915 + ], + "sources": 7 + }, + { + "entropy_mean": 1.964665412902832, + "mean_weights": [ + 0.08267911523580551, + 0.09424921870231628, + 0.10068939626216888, + 0.09415201097726822, + 0.07310959696769714, + 0.13375617563724518, + 0.18746790289878845, + 0.23389658331871033 + ], + "sources": 8 + }, + { + "entropy_mean": 2.008932590484619, + "mean_weights": [ + 0.10240210592746735, + 0.12103405594825745, + 0.10526444017887115, + 0.10498595237731934, + 0.1143268272280693, + 0.09818154573440552, + 0.11157755553722382, + 0.24222753942012787 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9140675067901611, + "mean_weights": [ + 0.08204050362110138, + 0.09491553902626038, + 0.09153153002262115, + 0.08785407245159149, + 0.07723700255155563, + 0.10408923774957657, + 0.14840492606163025, + 0.31392717361450195 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0044901371002197, + "mean_weights": [ + 0.10885269939899445, + 0.1156979352235794, + 0.1109280213713646, + 0.11524852365255356, + 0.10071282088756561, + 0.09537497907876968, + 0.11124420166015625, + 0.24194082617759705 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9380353689193726, + "mean_weights": [ + 0.08100489526987076, + 0.10177023708820343, + 0.10513961315155029, + 0.1066896989941597, + 0.08088363707065582, + 0.09770825505256653, + 0.138932466506958, + 0.28787118196487427 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9984654188156128, + "mean_weights": [ + 0.10779916495084763, + 0.11929400265216827, + 0.1236351951956749, + 0.13136842846870422, + 0.09354512393474579, + 0.0907527357339859, + 0.1102253794670105, + 0.22337999939918518 + ], + "sources": 8 + }, + { + "entropy_mean": 2.00014328956604, + "mean_weights": [ + 0.08765444159507751, + 0.11157547682523727, + 0.11302599310874939, + 0.12166635692119598, + 0.08641816675662994, + 0.10199566185474396, + 0.15030288696289062, + 0.22736100852489471 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0488078594207764, + "mean_weights": [ + 0.10723283886909485, + 0.1219443529844284, + 0.12234292924404144, + 0.14430664479732513, + 0.1068975180387497, + 0.1095016598701477, + 0.13023510575294495, + 0.15753895044326782 + ], + "sources": 8 + }, + { + "entropy_mean": 2.134258508682251, + "mean_weights": [ + 0.09382475912570953, + 0.12434307485818863, + 0.1074826568365097, + 0.11427435278892517, + 0.07799828052520752, + 0.09851260483264923, + 0.12680630385875702, + 0.14773844182491302, + 0.10901952534914017 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1467552185058594, + "mean_weights": [ + 0.09808872640132904, + 0.12375220656394958, + 0.11769671738147736, + 0.11697162687778473, + 0.11387702077627182, + 0.11408111453056335, + 0.11529303342103958, + 0.10075700283050537, + 0.09948255121707916 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1012933254241943, + "mean_weights": [ + 0.09751901030540466, + 0.11587081104516983, + 0.09766259789466858, + 0.10800798237323761, + 0.07261601090431213, + 0.0746348574757576, + 0.1129271388053894, + 0.2021249532699585, + 0.11863663047552109 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1556379795074463, + "mean_weights": [ + 0.09949111938476562, + 0.10229455679655075, + 0.12046648561954498, + 0.12406884133815765, + 0.08918923139572144, + 0.10173333436250687, + 0.10880154371261597, + 0.13875965774059296, + 0.11519522219896317 + ], + "sources": 9 + }, + { + "entropy_mean": 2.135862112045288, + "mean_weights": [ + 0.08859945088624954, + 0.11215460300445557, + 0.11093514412641525, + 0.12573575973510742, + 0.0869036614894867, + 0.10194189846515656, + 0.1471358984708786, + 0.13915079832077026, + 0.0874428078532219 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1723732948303223, + "mean_weights": [ + 0.09969168901443481, + 0.10641041398048401, + 0.13913322985172272, + 0.1252598762512207, + 0.09513836354017258, + 0.10382498800754547, + 0.11247606575489044, + 0.11626758426427841, + 0.10179778188467026 + ], + "sources": 9 + }, + { + "entropy_mean": 2.14325213432312, + "mean_weights": [ + 0.08698053658008575, + 0.12473929673433304, + 0.12396970391273499, + 0.13742369413375854, + 0.08447833359241486, + 0.10599758476018906, + 0.1492115706205368, + 0.11881858855485916, + 0.0683806836605072 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05576629564166069, + 0.048379842191934586, + 0.05707862228155136, + 0.06348917633295059, + 0.08294793218374252, + 0.10258980095386505, + 0.10450658202171326, + 0.07536187767982483, + 0.10511407256126404, + 0.08322826772928238, + 0.08434967696666718, + 0.0912788137793541, + 0.08562305569648743, + 0.08653967082500458, + 0.08805308490991592, + 0.091586172580719, + 0.08400599658489227, + 0.053851574659347534, + 0.06219805032014847, + 0.061099451035261154, + 0.060256246477365494, + 0.05317429080605507, + 0.060655634850263596, + 0.05588795244693756, + 0.06256718933582306, + 0.04701605439186096, + 0.04898771643638611, + 0.04751140624284744, + 0.05384722724556923, + 0.050974659621715546, + 0.05927867814898491, + 0.05479227751493454, + 0.06086777150630951, + 0.04300158470869064, + 0.04659106209874153, + 0.04196516051888466, + 0.0495486781001091, + 0.045776620507240295, + 0.05155850574374199, + 0.050973519682884216, + 0.05232362821698189, + 0.044989023357629776, + 0.045673467218875885, + 0.045741744339466095, + 0.04657372459769249, + 0.04639090225100517, + 0.04738600179553032, + 0.04718850553035736, + 0.05023302882909775, + 0.04232577234506607, + 0.03951188921928406, + 0.04039522632956505, + 0.041993338614702225, + 0.04450058192014694, + 0.04661031439900398, + 0.04799575358629227, + 0.04939674213528633, + 0.04567969590425491, + 0.04665130749344826, + 0.05162831023335457, + 0.05335777997970581, + 0.053171806037425995, + 0.05649247020483017, + 0.05376771092414856 + ], + "loss_nats": 1.4354361295700073, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.9188275337219238, + "mean_weights": [ + 0.11096015572547913, + 0.07863883674144745, + 0.07829269766807556, + 0.06037574261426926, + 0.06428550183773041, + 0.05204963684082031, + 0.0706396996974945, + 0.28138381242752075, + 0.20337392389774323 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.04483851417899132, + 0.091498002409935, + 0.14079734683036804, + 0.1524418443441391, + 0.18620415031909943, + 0.18927344679832458, + 0.19053027033805847, + 0.20122204720973969, + 0.0951480120420456, + 0.09716148674488068, + 0.11671996116638184, + 0.11803841590881348, + 0.1282212883234024, + 0.12927795946598053, + 0.13007308542728424, + 0.13284066319465637, + 0.03393593803048134, + 0.04031922295689583, + 0.055263202637434006, + 0.06599447876214981, + 0.07879888266324997, + 0.09237803518772125, + 0.10551883280277252, + 0.12235672026872635, + 0.0624675415456295, + 0.07300658524036407, + 0.09785781800746918, + 0.10997047275304794, + 0.12080339342355728, + 0.13365790247917175, + 0.1471952497959137, + 0.15966735780239105, + 0.03177119418978691, + 0.044320181012153625, + 0.05607033520936966, + 0.07193361222743988, + 0.07977130264043808, + 0.09863750636577606, + 0.10540571808815002, + 0.13064858317375183, + 0.03732937201857567, + 0.045498576015233994, + 0.06753379106521606, + 0.07957836240530014, + 0.09847283363342285, + 0.11307200789451599, + 0.12412801384925842, + 0.1419886350631714, + 0.023885536938905716, + 0.0492420569062233, + 0.054955773055553436, + 0.07780471444129944, + 0.08039270341396332, + 0.10795509815216064, + 0.10950618237257004, + 0.1515592485666275, + 0.013619035482406616, + 0.06438802927732468, + 0.07004562765359879, + 0.14722995460033417, + 0.14974848926067352, + 0.23145347833633423, + 0.2348189353942871, + 0.344204843044281 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.377212412189692e-05, + 6.547482189489529e-05, + 6.454151298385113e-05, + 6.153762660687789e-05, + 7.872180867707357e-05, + 8.128974150167778e-05, + 7.331831875490025e-05, + 6.799855327699333e-05, + 6.842906441306695e-05, + 6.388261681422591e-05, + 6.002560621709563e-05, + 5.6137647334253415e-05, + 4.663488289224915e-05, + 4.5897737436462194e-05, + 4.5840533857699484e-05, + 4.774069384438917e-05, + 7.272446964634582e-05, + 6.376936653396115e-05, + 5.520603372133337e-05, + 4.907705442747101e-05, + 5.3741958254249766e-05, + 5.0967148126801476e-05, + 4.8129717470146716e-05, + 4.637950041797012e-05, + 9.337218216387555e-05, + 7.235947123263031e-05, + 5.499512189999223e-05, + 4.418334356159903e-05, + 2.8606293199118227e-05, + 2.3939544917084277e-05, + 2.272939855174627e-05, + 2.271045377710834e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 6.958181347727077e-05, + "first_to_last_ratio": 1.5339237697097534, + "imbalance_abs_log_ratio": 0.4278290079096044, + "last_quartile_mean": 4.536197616289428e-05, + "mean": 5.6066698391532555e-05, + "normalized": [ + 1.137433199232721, + 1.167802345657357, + 1.1511559416810333, + 1.0975789260344886, + 1.4040742710999814, + 1.4498756629827614, + 1.3076981676875967, + 1.212815365052114, + 1.2204939184255839, + 1.1394039357929047, + 1.070610682261272, + 1.001265438214774, + 0.8317750862835213, + 0.8186274339884062, + 0.8176071567043178, + 0.8514982193351194, + 1.29710633464604, + 1.1373840151713281, + 0.9846492714054808, + 0.8753334124429708, + 0.958536168456927, + 0.9090449337837015, + 0.8584368056424646, + 0.8272201101282355, + 1.6653768608207726, + 1.290596259607082, + 0.9808874693484331, + 0.7880496770659104, + 0.5102189716853112, + 0.4269833181527189, + 0.4053992691529517, + 0.40506137205571885 + ], + "population_cv": 0.2961000359572155 + }, + "activation_output_rms_by_block": [ + 0.12991468608379364, + 0.2110949605703354, + 0.24801407754421234, + 0.2648625373840332, + 0.13388457894325256, + 0.15026015043258667, + 0.16143088042736053, + 0.16682761907577515, + 0.048765525221824646, + 0.07774458825588226, + 0.1121370792388916, + 0.16072051227092743, + 0.08850602060556412, + 0.13462567329406738, + 0.16438885033130646, + 0.20456722378730774, + 0.05670234560966492, + 0.09389284998178482, + 0.1309870481491089, + 0.17517942190170288, + 0.06527502089738846, + 0.11259422451257706, + 0.1566854566335678, + 0.19526530802249908, + 0.06247122585773468, + 0.09589836746454239, + 0.12854547798633575, + 0.17463485896587372, + 0.07638392597436905, + 0.171359121799469, + 0.27035632729530334, + 0.40826740860939026 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.18328618630766869, + "first_to_last_ratio": 1.0564679247107793, + "imbalance_abs_log_ratio": 0.054931197661165394, + "last_quartile_mean": 0.17348958924412727, + "mean": 0.15100760478526354, + "normalized": [ + 0.8603188314160434, + 1.397909468668929, + 1.6423946191113643, + 1.7539682041886178, + 0.8866081885984463, + 0.9950502204591631, + 1.069024839225277, + 1.104763030398423, + 0.3229342342802563, + 0.5148388941499797, + 0.7425922647958905, + 1.0643206512643908, + 0.5861030689906103, + 0.8915158510427892, + 1.0886130573693382, + 1.3546816008253841, + 0.37549331167987876, + 0.6217756391494502, + 0.8674202093025423, + 1.1600701974633147, + 0.4322631366162725, + 0.7456195644761651, + 1.0375997742390413, + 1.293082611966272, + 0.4136958926444153, + 0.635056543019222, + 0.8512516847686613, + 1.1564640020230021, + 0.505828339460048, + 1.134771470901388, + 1.7903490865890932, + 2.70362151091633 + ], + "population_cv": 0.48836550314550853 + }, + "bits_per_byte": 1.9235860499255144, + "branch_output_rms_by_sublayer": [ + 0.06858986616134644, + 0.10709022730588913, + 0.10697305202484131, + 0.02342848666012287, + 0.055876992642879486, + 0.012324743904173374, + 0.033786918967962265, + 0.032528962939977646, + 0.13012711703777313, + 0.012621155008673668, + 0.03902044519782066, + 0.004458175972104073, + 0.04220427945256233, + 0.010553214699029922, + 0.035928770899772644, + 0.02080267295241356, + 0.04122508317232132, + 0.02460864931344986, + 0.03184391185641289, + 0.027303222566843033, + 0.034540969878435135, + 0.03519929200410843, + 0.04416383057832718, + 0.04499252140522003, + 0.07039257138967514, + 0.04205214977264404, + 0.03983128070831299, + 0.037533096969127655, + 0.03637707233428955, + 0.03727266192436218, + 0.04788806289434433, + 0.03646649792790413, + 0.039353542029857635, + 0.038644660264253616, + 0.0381646603345871, + 0.04691791534423828, + 0.03741531819105148, + 0.05956745892763138, + 0.04033776745200157, + 0.07839375734329224, + 0.05449078232049942, + 0.036974094808101654, + 0.06478945165872574, + 0.04500679299235344, + 0.06316104531288147, + 0.053396474570035934, + 0.05014883726835251, + 0.06321974843740463, + 0.03010101430118084, + 0.05116280913352966, + 0.025841858237981796, + 0.05484970659017563, + 0.021724559366703033, + 0.06025480851531029, + 0.01672915741801262, + 0.07887377589941025, + 0.019863896071910858, + 0.0733853131532669, + 0.02619529329240322, + 0.11109374463558197, + 0.019914794713258743, + 0.11783956736326218, + 0.023350918665528297, + 0.16346506774425507 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0002950378310382648, + 0.00014847223993931047, + 0.00011710376597652883, + 0.0001377775090297729, + 0.00016995708660904672, + 0.00011361802420731338, + 0.00012338259405490434, + 0.00014164260582306468, + 0.00019093183126556295, + 0.00016954677713837217, + 0.00018296108816108484, + 0.00021392917211422575, + 0.00016526815062807302, + 0.00014231309692063464, + 0.00016267214757338577, + 0.00018064973695327717, + 0.0002733841225923413, + 0.00031035624396506345, + 0.0003185268264789981, + 0.0003186495235864043, + 0.0002329266876355598, + 0.000287248960925402, + 0.0002781617076192276, + 0.0002784432881768582, + 0.0005040242579576307, + 0.00038669408898217805, + 0.0003175490110817138, + 0.00027301250209970756, + 0.00015947821619985384, + 0.00015536274603992123, + 0.00014735168782838124, + 0.00014620029036749587 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00015587395708477577, + "first_to_last_ratio": 0.5967401481925266, + "imbalance_abs_log_ratio": 0.5162735230029069, + "last_quartile_mean": 0.0002612091000696103, + "mean": 0.00022008230684279873, + "normalized": [ + 1.3405795098694853, + 0.6746214271797951, + 0.5320907784748647, + 0.6260271941269008, + 0.7722432986421046, + 0.5162524231830636, + 0.5606202326070426, + 0.6435892455645593, + 0.8675473917216913, + 0.7703789530862957, + 0.8313302908614595, + 0.972041665607549, + 0.7509379240836539, + 0.6466357925913899, + 0.7391423231926585, + 0.8208280781167583, + 1.2421903719303304, + 1.4101826194812923, + 1.4473077415828648, + 1.4478652471323403, + 1.058361714655852, + 1.3051887952564007, + 1.2638985460013106, + 1.2651779789628697, + 2.2901625541286568, + 1.7570430559799042, + 1.4428647883472705, + 1.240501819597502, + 0.7246298827364009, + 0.7059301961556335, + 0.6695299133411583, + 0.6642982458009421 + ], + "population_cv": 0.40747215756498506 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.23262560367584229, + "mean_weights": [ + 0.9319208860397339, + 0.06807906925678253 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5635989904403687, + "mean_weights": [ + 0.6802793145179749, + 0.31972068548202515 + ], + "sources": 2 + }, + { + "entropy_mean": 0.537754237651825, + "mean_weights": [ + 0.7549498081207275, + 0.24505013227462769 + ], + "sources": 2 + }, + { + "entropy_mean": 0.673224687576294, + "mean_weights": [ + 0.5554800033569336, + 0.444519966840744 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6434458494186401, + "mean_weights": [ + 0.40043848752975464, + 0.5995614528656006 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6889988780021667, + "mean_weights": [ + 0.5034067630767822, + 0.49659326672554016 + ], + "sources": 2 + }, + { + "entropy_mean": 0.536893367767334, + "mean_weights": [ + 0.7555723786354065, + 0.2444276213645935 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6844105124473572, + "mean_weights": [ + 0.5416830778121948, + 0.4583168625831604 + ], + "sources": 2 + }, + { + "entropy_mean": 0.9747234582901001, + "mean_weights": [ + 0.23145374655723572, + 0.22331956028938293, + 0.5452266931533813 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0803881883621216, + "mean_weights": [ + 0.3016301691532135, + 0.281657874584198, + 0.4167119562625885 + ], + "sources": 3 + }, + { + "entropy_mean": 0.74129718542099, + "mean_weights": [ + 0.13594412803649902, + 0.1349472999572754, + 0.7291085720062256 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0687153339385986, + "mean_weights": [ + 0.3499948978424072, + 0.23351827263832092, + 0.41648679971694946 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9891505241394043, + "mean_weights": [ + 0.3034753203392029, + 0.1797100305557251, + 0.516814649105072 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0735982656478882, + "mean_weights": [ + 0.346267968416214, + 0.24244481325149536, + 0.41128724813461304 + ], + "sources": 3 + }, + { + "entropy_mean": 0.974714457988739, + "mean_weights": [ + 0.24753746390342712, + 0.2023107260465622, + 0.5501518249511719 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0766172409057617, + "mean_weights": [ + 0.40780121088027954, + 0.24722465872764587, + 0.3449741005897522 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3367538452148438, + "mean_weights": [ + 0.2777651250362396, + 0.14565420150756836, + 0.25300854444503784, + 0.32357215881347656 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3676683902740479, + "mean_weights": [ + 0.29528555274009705, + 0.1919308602809906, + 0.22549307346343994, + 0.2872905433177948 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3611929416656494, + "mean_weights": [ + 0.23971755802631378, + 0.20354682207107544, + 0.28248876333236694, + 0.27424687147140503 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3689255714416504, + "mean_weights": [ + 0.29830437898635864, + 0.21305717527866364, + 0.2095387578010559, + 0.279099702835083 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3248838186264038, + "mean_weights": [ + 0.32920846343040466, + 0.17001411318778992, + 0.19967728853225708, + 0.3011001646518707 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3714501857757568, + "mean_weights": [ + 0.26589149236679077, + 0.19682028889656067, + 0.2416974902153015, + 0.29559072852134705 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3474481105804443, + "mean_weights": [ + 0.27136653661727905, + 0.18102365732192993, + 0.20809906721115112, + 0.3395107090473175 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3790339231491089, + "mean_weights": [ + 0.2655395269393921, + 0.22196024656295776, + 0.24067091941833496, + 0.2718293070793152 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5632529258728027, + "mean_weights": [ + 0.16829654574394226, + 0.1492592841386795, + 0.1793084442615509, + 0.2865314483642578, + 0.2166043072938919 + ], + "sources": 5 + }, + { + "entropy_mean": 1.583046793937683, + "mean_weights": [ + 0.1907578408718109, + 0.16862604022026062, + 0.15888771414756775, + 0.19696356356143951, + 0.28476482629776 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5878093242645264, + "mean_weights": [ + 0.23036769032478333, + 0.1456999033689499, + 0.17779815196990967, + 0.23517638444900513, + 0.21095788478851318 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5599913597106934, + "mean_weights": [ + 0.20395052433013916, + 0.1652589589357376, + 0.13161113858222961, + 0.18477579951286316, + 0.31440359354019165 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5846099853515625, + "mean_weights": [ + 0.1986508071422577, + 0.15068933367729187, + 0.19555959105491638, + 0.22942668199539185, + 0.2256735861301422 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5648300647735596, + "mean_weights": [ + 0.20817363262176514, + 0.19657433032989502, + 0.13692165911197662, + 0.15221485495567322, + 0.3061155080795288 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5876898765563965, + "mean_weights": [ + 0.18916656076908112, + 0.15277862548828125, + 0.1903425008058548, + 0.21992729604244232, + 0.24778501689434052 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5913512706756592, + "mean_weights": [ + 0.21221742033958435, + 0.19118179380893707, + 0.1483897715806961, + 0.1869838833808899, + 0.2612271308898926 + ], + "sources": 5 + }, + { + "entropy_mean": 1.74949049949646, + "mean_weights": [ + 0.1333819031715393, + 0.12170803546905518, + 0.15897899866104126, + 0.18179675936698914, + 0.14956891536712646, + 0.25456541776657104 + ], + "sources": 6 + }, + { + "entropy_mean": 1.749058723449707, + "mean_weights": [ + 0.15612728893756866, + 0.14588959515094757, + 0.11389230191707611, + 0.13761168718338013, + 0.17871133983135223, + 0.2677677869796753 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6866942644119263, + "mean_weights": [ + 0.11552339792251587, + 0.10913115739822388, + 0.15327559411525726, + 0.18143582344055176, + 0.11612141132354736, + 0.3245126008987427 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7629921436309814, + "mean_weights": [ + 0.16343337297439575, + 0.16359388828277588, + 0.12182332575321198, + 0.13560590147972107, + 0.1715523600578308, + 0.24399115145206451 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6641472578048706, + "mean_weights": [ + 0.1189078614115715, + 0.11148512363433838, + 0.14851495623588562, + 0.17322975397109985, + 0.11026395112276077, + 0.33759835362434387 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7752645015716553, + "mean_weights": [ + 0.1680198311805725, + 0.1747690886259079, + 0.13175584375858307, + 0.13679710030555725, + 0.1696045696735382, + 0.21905356645584106 + ], + "sources": 6 + }, + { + "entropy_mean": 1.643704891204834, + "mean_weights": [ + 0.1199546754360199, + 0.11406206339597702, + 0.14940081536769867, + 0.17184020578861237, + 0.10169733315706253, + 0.34304487705230713 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9351837635040283, + "mean_weights": [ + 0.15311850607395172, + 0.17045700550079346, + 0.12090195715427399, + 0.11945034563541412, + 0.15857462584972382, + 0.13318118453025818, + 0.14431636035442352 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9252710342407227, + "mean_weights": [ + 0.08240433037281036, + 0.0854264348745346, + 0.09299176931381226, + 0.09383343905210495, + 0.06261328607797623, + 0.1278754025697708, + 0.19161781668663025, + 0.26323753595352173 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9649277925491333, + "mean_weights": [ + 0.1026010662317276, + 0.11450468003749847, + 0.09586406499147415, + 0.09347793459892273, + 0.10788068920373917, + 0.09829258918762207, + 0.10090316832065582, + 0.28647580742836 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8292760848999023, + "mean_weights": [ + 0.07899133861064911, + 0.08346068114042282, + 0.08307052403688431, + 0.0824592262506485, + 0.06337855756282806, + 0.09794235974550247, + 0.13825617730617523, + 0.3724411129951477 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9737095832824707, + "mean_weights": [ + 0.11180566251277924, + 0.115575410425663, + 0.1011018231511116, + 0.10107971727848053, + 0.10309582948684692, + 0.0978727638721466, + 0.09650644659996033, + 0.2729623317718506 + ], + "sources": 8 + }, + { + "entropy_mean": 1.84842848777771, + "mean_weights": [ + 0.07836711406707764, + 0.09220220148563385, + 0.08729387819766998, + 0.09924592077732086, + 0.06906762719154358, + 0.09387990832328796, + 0.12276017665863037, + 0.35718315839767456 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9916129112243652, + "mean_weights": [ + 0.10888969898223877, + 0.11976490914821625, + 0.11755050718784332, + 0.13214971125125885, + 0.0919133797287941, + 0.094005286693573, + 0.09874803572893143, + 0.2369784712791443 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9611189365386963, + "mean_weights": [ + 0.08629890531301498, + 0.10353914648294449, + 0.10439523309469223, + 0.11415326595306396, + 0.07884065806865692, + 0.10495521873235703, + 0.14107395708560944, + 0.26674360036849976 + ], + "sources": 8 + }, + { + "entropy_mean": 2.044013500213623, + "mean_weights": [ + 0.10659720003604889, + 0.12391743063926697, + 0.110470712184906, + 0.1376512050628662, + 0.10832816362380981, + 0.11791528016328812, + 0.12954822182655334, + 0.16557176411151886 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0704360008239746, + "mean_weights": [ + 0.08733996748924255, + 0.11058264225721359, + 0.08801080286502838, + 0.10169872641563416, + 0.06532149016857147, + 0.09871777892112732, + 0.11666804552078247, + 0.15798911452293396, + 0.1736714392900467 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1404314041137695, + "mean_weights": [ + 0.09997774660587311, + 0.11364627629518509, + 0.10841815918684006, + 0.12224972248077393, + 0.1191805899143219, + 0.11442385613918304, + 0.10672082006931305, + 0.10375681519508362, + 0.1116260290145874 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0337002277374268, + "mean_weights": [ + 0.09811638295650482, + 0.11396120488643646, + 0.08495737612247467, + 0.09679488837718964, + 0.06505166739225388, + 0.06692498177289963, + 0.09284307062625885, + 0.2465619444847107, + 0.13478848338127136 + ], + "sources": 9 + }, + { + "entropy_mean": 2.144615888595581, + "mean_weights": [ + 0.0934646874666214, + 0.09936770796775818, + 0.1254754364490509, + 0.13800542056560516, + 0.08781319856643677, + 0.11397683620452881, + 0.1105157732963562, + 0.12372783571481705, + 0.10765309631824493 + ], + "sources": 9 + }, + { + "entropy_mean": 2.101943016052246, + "mean_weights": [ + 0.09116317331790924, + 0.11131736636161804, + 0.10448693484067917, + 0.12092888355255127, + 0.074070505797863, + 0.09986056387424469, + 0.1327168196439743, + 0.16121982038021088, + 0.10423591732978821 + ], + "sources": 9 + }, + { + "entropy_mean": 2.162642478942871, + "mean_weights": [ + 0.09671883285045624, + 0.09925203025341034, + 0.12671872973442078, + 0.1371961086988449, + 0.09933674335479736, + 0.10610610246658325, + 0.11141124367713928, + 0.12104853987693787, + 0.10221166908740997 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1396477222442627, + "mean_weights": [ + 0.08908043801784515, + 0.11732056736946106, + 0.1194748505949974, + 0.1345033198595047, + 0.07751815766096115, + 0.11012174189090729, + 0.1400572955608368, + 0.13388410210609436, + 0.07803953438997269 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05928435176610947, + 0.05591506510972977, + 0.06663759052753448, + 0.075626902282238, + 0.10697659105062485, + 0.1525050550699234, + 0.13438783586025238, + 0.08619081228971481, + 0.1336100846529007, + 0.10896878689527512, + 0.11143296211957932, + 0.12661238014698029, + 0.10696384310722351, + 0.11037663370370865, + 0.10912629961967468, + 0.11784035712480545, + 0.10305396467447281, + 0.0680573433637619, + 0.07468587905168533, + 0.08221447467803955, + 0.07684502750635147, + 0.0671975314617157, + 0.0759248286485672, + 0.06841029226779938, + 0.07922825217247009, + 0.05793682858347893, + 0.05927792564034462, + 0.05704478919506073, + 0.06561227142810822, + 0.06245425343513489, + 0.07391341030597687, + 0.06796003133058548, + 0.07564540207386017, + 0.051437702029943466, + 0.05707206204533577, + 0.05180804803967476, + 0.06205566227436066, + 0.05852527916431427, + 0.06596656143665314, + 0.06766477227210999, + 0.06732893735170364, + 0.05793819576501846, + 0.058673642575740814, + 0.059030115604400635, + 0.060210879892110825, + 0.0604022741317749, + 0.06194175034761429, + 0.0619652234017849, + 0.06674404442310333, + 0.05654582008719444, + 0.050215352326631546, + 0.05224108323454857, + 0.054014310240745544, + 0.058316633105278015, + 0.05913076549768448, + 0.06228234991431236, + 0.0633753314614296, + 0.05592459812760353, + 0.058598291128873825, + 0.06695902347564697, + 0.06395619362592697, + 0.06826630979776382, + 0.06905022263526917, + 0.06775816529989243 + ], + "loss_nats": 1.3333282470703125, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.8613641262054443, + "mean_weights": [ + 0.10053815692663193, + 0.0808011144399643, + 0.07359045743942261, + 0.05753888934850693, + 0.05637887865304947, + 0.04913659021258354, + 0.05702194944024086, + 0.2843329608440399, + 0.24066099524497986 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.06858986616134644, + 0.12991468608379364, + 0.19991879165172577, + 0.2110949605703354, + 0.2439028024673462, + 0.24801407754421234, + 0.25081589818000793, + 0.2648625373840332, + 0.13012711703777313, + 0.13388457894325256, + 0.14940428733825684, + 0.15026015043258667, + 0.16069534420967102, + 0.16143088042736053, + 0.16462913155555725, + 0.16682761907577515, + 0.04122508317232132, + 0.048765525221824646, + 0.06456131488084793, + 0.07774458825588226, + 0.09435933828353882, + 0.1121370792388916, + 0.1343858242034912, + 0.16072051227092743, + 0.07039257138967514, + 0.08850602060556412, + 0.11566867679357529, + 0.13462567329406738, + 0.14476297795772552, + 0.16438885033130646, + 0.18775273859500885, + 0.20456722378730774, + 0.039353542029857635, + 0.05670234560966492, + 0.07167500257492065, + 0.09389284998178482, + 0.10342796891927719, + 0.1309870481491089, + 0.13953039050102234, + 0.17517942190170288, + 0.05449078232049942, + 0.06527502089738846, + 0.09667403995990753, + 0.11259422451257706, + 0.13678009808063507, + 0.1566854566335678, + 0.17104315757751465, + 0.19526530802249908, + 0.03010101430118084, + 0.06247122585773468, + 0.06944862008094788, + 0.09589836746454239, + 0.09973802417516708, + 0.12854547798633575, + 0.13028548657894135, + 0.17463485896587372, + 0.019863896071910858, + 0.07638392597436905, + 0.08277778327465057, + 0.171359121799469, + 0.17420724034309387, + 0.27035632729530334, + 0.2734096646308899, + 0.40826740860939026 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00010937357728835195, + 0.00011177909618709236, + 0.00011071364860981703, + 0.00010557123459875584, + 0.00014026847202330828, + 0.0001429846597602591, + 0.0001286787010030821, + 0.00011820554209407419, + 0.00012224233068991452, + 0.00011418478970881552, + 0.00010696215758798644, + 9.960799070540816e-05, + 8.516090747434646e-05, + 8.364950190298259e-05, + 8.40622087707743e-05, + 8.718625758774579e-05, + 0.00013687518367078155, + 0.00011597294360399246, + 9.767105802893639e-05, + 8.500106923747808e-05, + 9.275639604311436e-05, + 8.73586322995834e-05, + 8.271219849120826e-05, + 7.955932233016938e-05, + 0.00017083239799831063, + 0.00013092967856209725, + 9.736601350596175e-05, + 7.3001574492082e-05, + 5.768415576312691e-05, + 4.2060197301907465e-05, + 3.693173857755028e-05, + 3.6230074329068884e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.0001209468664455926, + "first_to_last_ratio": 1.50003284432985, + "imbalance_abs_log_ratio": 0.4054870040883457, + "last_quartile_mean": 8.062947881626314e-05, + "mean": 9.917417844462761e-05, + "normalized": [ + 1.1028432904984338, + 1.1270987866010154, + 1.1163555912049457, + 1.0645032432277715, + 1.4143648500362929, + 1.4417529038578565, + 1.297502061738055, + 1.1918983746366243, + 1.2326024032371155, + 1.1513560434742482, + 1.0785282950209376, + 1.004374246074776, + 0.8587004078072071, + 0.8434604976303082, + 0.847621932333013, + 0.8791225594717167, + 1.3801494080155523, + 1.1693864816711759, + 0.9848436312832127, + 0.8570887157380096, + 0.93528776842758, + 0.8808606602005659, + 0.8340094144302829, + 0.8022181134032798, + 1.7225491622669935, + 1.320199275814519, + 0.9817677850522812, + 0.7360945725690212, + 0.581644906646078, + 0.4241043178935042, + 0.37239268483752097, + 0.3653176149001062 + ], + "population_cv": 0.3051121408325057 + }, + "activation_output_rms_by_block": [ + 0.11405637860298157, + 0.19576610624790192, + 0.22915078699588776, + 0.241273432970047, + 0.11842832714319229, + 0.1310930997133255, + 0.13807407021522522, + 0.14413440227508545, + 0.043267782777547836, + 0.0690729096531868, + 0.09961047768592834, + 0.14613077044487, + 0.08204278349876404, + 0.11789657920598984, + 0.1456192433834076, + 0.18111956119537354, + 0.0510682575404644, + 0.0844164788722992, + 0.11892655491828918, + 0.1606190800666809, + 0.060701869428157806, + 0.10386133193969727, + 0.1428302675485611, + 0.17762337625026703, + 0.0571737065911293, + 0.08602745085954666, + 0.11427143961191177, + 0.15259574353694916, + 0.06466396898031235, + 0.14282406866550446, + 0.22156114876270294, + 0.3222712576389313 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.16399707552045584, + "first_to_last_ratio": 1.129661850973041, + "imbalance_abs_log_ratio": 0.12191834101444085, + "last_quartile_mean": 0.1451735980808735, + "mean": 0.13306789728812873, + "normalized": [ + 0.8571291868843318, + 1.4711745675518886, + 1.7220591267006575, + 1.8131603330863748, + 0.889984207736914, + 0.9851594741102186, + 1.0376211921065885, + 1.0831643485204752, + 0.3251556811171453, + 0.5190801918484131, + 0.7485688112305867, + 1.0981669765808093, + 0.6165482822736633, + 0.8859881429606661, + 1.0943228708882484, + 1.3611063591333354, + 0.38377594131428655, + 0.6343865093886184, + 0.8937283698169541, + 1.2070460519782338, + 0.45617215470626626, + 0.7805138132964468, + 1.0733638274849577, + 1.334832667158356, + 0.42965815013468234, + 0.6464929003369873, + 0.8587453618845615, + 1.1467509943930145, + 0.4859471765778113, + 1.0733172431232676, + 1.6650232947091805, + 2.4218557909660587 + ], + "population_cv": 0.45957997854102167 + }, + "bits_per_byte": 1.8700714123350903, + "branch_output_rms_by_sublayer": [ + 0.06266748160123825, + 0.09198914468288422, + 0.10721665620803833, + 0.018869051709771156, + 0.04951969161629677, + 0.012155124917626381, + 0.028980351984500885, + 0.025462880730628967, + 0.11456266790628433, + 0.012394234538078308, + 0.027164561673998833, + 0.0037419795989990234, + 0.029356203973293304, + 0.00951854232698679, + 0.027091041207313538, + 0.01824989728629589, + 0.03705674782395363, + 0.021580733358860016, + 0.028705578297376633, + 0.02515704743564129, + 0.0313430018723011, + 0.032159000635147095, + 0.04253105819225311, + 0.04203430563211441, + 0.06314649432897568, + 0.04122118651866913, + 0.03035840019583702, + 0.03564886748790741, + 0.033261485397815704, + 0.03557557985186577, + 0.043308015912771225, + 0.03478488326072693, + 0.03456971421837807, + 0.03566031530499458, + 0.03342197462916374, + 0.04537735506892204, + 0.03468935936689377, + 0.058925095945596695, + 0.03602323308587074, + 0.07822802662849426, + 0.047963351011276245, + 0.03763965517282486, + 0.05742645263671875, + 0.04522239416837692, + 0.05511033162474632, + 0.05345170572400093, + 0.04611408710479736, + 0.0629618763923645, + 0.02691013365983963, + 0.047519251704216, + 0.02107274904847145, + 0.05213481932878494, + 0.01906774379312992, + 0.05802415683865547, + 0.015758879482746124, + 0.07248375564813614, + 0.01650204323232174, + 0.06214911490678787, + 0.02049560658633709, + 0.09728313982486725, + 0.017270134761929512, + 0.09871190041303635, + 0.02022666111588478, + 0.12426252663135529 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0004195540085483418, + 0.00022290430717419827, + 0.00018579506214368963, + 0.00020497867305137473, + 0.00026845595478514707, + 0.00016210038534466737, + 0.00020239958883723667, + 0.00024237713828714863, + 0.0003230329442295159, + 0.0002827543974714729, + 0.0003033090558088245, + 0.0003591246180736339, + 0.0002962552614134452, + 0.00026307085929084926, + 0.0003001047685282543, + 0.00032675080429261737, + 0.0005025480001152495, + 0.0005689264310260318, + 0.0005791526105580144, + 0.0005616197589417262, + 0.0004071088830331888, + 0.0004914191621052311, + 0.0004722503673668984, + 0.00048685385652756235, + 0.000904196290148671, + 0.0006897123578071538, + 0.0005669288948972495, + 0.0004734512496503878, + 0.0002977476151726678, + 0.00026930590861566004, + 0.00022508673939780185, + 0.00020885577642255332 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00023857063977147553, + "first_to_last_ratio": 0.5250111631728468, + "imbalance_abs_log_ratio": 0.6443357534301968, + "last_quartile_mean": 0.00045441060401401814, + "mean": 0.000377129116533327, + "normalized": [ + 1.1124943425344505, + 0.5910556819987676, + 0.4926563715142659, + 0.5435238597740588, + 0.7118409664384096, + 0.4298272878921098, + 0.5366851297448166, + 0.6426900699557354, + 0.8565579534110818, + 0.7497548851985439, + 0.8042578589447654, + 0.952259060172295, + 0.7855539347815512, + 0.6975617839032633, + 0.7957613331129101, + 0.8664162748721161, + 1.3325621864861743, + 1.5085720144224282, + 1.535687872317423, + 1.4891975568057092, + 1.0794947013782545, + 1.3030528287566052, + 1.2522246272256878, + 1.2909474108041692, + 2.3975775152560166, + 1.828849398177411, + 1.503275324134645, + 1.2554089008095688, + 0.7895110775578521, + 0.7140947140098672, + 0.596842644945742, + 0.5538044326633069 + ], + "population_cv": 0.44389199132289925 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.23296202719211578, + "mean_weights": [ + 0.931702733039856, + 0.06829722225666046 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5422546863555908, + "mean_weights": [ + 0.7066349983215332, + 0.2933650016784668 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5889854431152344, + "mean_weights": [ + 0.7063794136047363, + 0.29362058639526367 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6843193769454956, + "mean_weights": [ + 0.5396547317504883, + 0.4603452980518341 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6168524026870728, + "mean_weights": [ + 0.3534718453884125, + 0.6465281844139099 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6881426572799683, + "mean_weights": [ + 0.4820123016834259, + 0.5179877281188965 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5877853631973267, + "mean_weights": [ + 0.701586902141571, + 0.29841312766075134 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6865841746330261, + "mean_weights": [ + 0.5296852588653564, + 0.47031474113464355 + ], + "sources": 2 + }, + { + "entropy_mean": 0.9759489893913269, + "mean_weights": [ + 0.2243807315826416, + 0.23289287090301514, + 0.5427263975143433 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0660829544067383, + "mean_weights": [ + 0.27302923798561096, + 0.27547094225883484, + 0.4514998197555542 + ], + "sources": 3 + }, + { + "entropy_mean": 0.7023833990097046, + "mean_weights": [ + 0.12144232541322708, + 0.12603233754634857, + 0.7525253295898438 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0623631477355957, + "mean_weights": [ + 0.327129602432251, + 0.2307818979024887, + 0.4420885443687439 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9793835878372192, + "mean_weights": [ + 0.2861265242099762, + 0.17773064970970154, + 0.5361428260803223 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0699747800827026, + "mean_weights": [ + 0.3226298391819, + 0.24580581486225128, + 0.4315643310546875 + ], + "sources": 3 + }, + { + "entropy_mean": 0.964678168296814, + "mean_weights": [ + 0.23837798833847046, + 0.196360245347023, + 0.5652617812156677 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0828839540481567, + "mean_weights": [ + 0.38775932788848877, + 0.25720009207725525, + 0.35504060983657837 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3299640417099, + "mean_weights": [ + 0.26227158308029175, + 0.13554993271827698, + 0.2709197998046875, + 0.3312586545944214 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3674194812774658, + "mean_weights": [ + 0.2877163887023926, + 0.1929437667131424, + 0.22255335748195648, + 0.29678648710250854 + ], + "sources": 4 + }, + { + "entropy_mean": 1.355736494064331, + "mean_weights": [ + 0.2245943397283554, + 0.19579559564590454, + 0.2855016589164734, + 0.29410839080810547 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3665947914123535, + "mean_weights": [ + 0.28862667083740234, + 0.20502260327339172, + 0.209466814994812, + 0.2968839108943939 + ], + "sources": 4 + }, + { + "entropy_mean": 1.323761224746704, + "mean_weights": [ + 0.30563855171203613, + 0.16321708261966705, + 0.20236292481422424, + 0.32878145575523376 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3707349300384521, + "mean_weights": [ + 0.2636043429374695, + 0.19438841938972473, + 0.24279670417308807, + 0.2992105185985565 + ], + "sources": 4 + }, + { + "entropy_mean": 1.341447114944458, + "mean_weights": [ + 0.26287490129470825, + 0.17503273487091064, + 0.2093992829322815, + 0.3526930809020996 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3801867961883545, + "mean_weights": [ + 0.2665756642818451, + 0.2240101397037506, + 0.2382856011390686, + 0.2711285948753357 + ], + "sources": 4 + }, + { + "entropy_mean": 1.547540307044983, + "mean_weights": [ + 0.16689950227737427, + 0.13884101808071136, + 0.1795312762260437, + 0.31771424412727356, + 0.1970139741897583 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5811553001403809, + "mean_weights": [ + 0.19019567966461182, + 0.16833053529262543, + 0.15643972158432007, + 0.19371269643306732, + 0.29132136702537537 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5820006132125854, + "mean_weights": [ + 0.22218428552150726, + 0.13288334012031555, + 0.17689163982868195, + 0.24057719111442566, + 0.22746355831623077 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5623219013214111, + "mean_weights": [ + 0.20177140831947327, + 0.16661575436592102, + 0.13263191282749176, + 0.18349617719650269, + 0.31548476219177246 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5745264291763306, + "mean_weights": [ + 0.19355633854866028, + 0.13506048917770386, + 0.18528637290000916, + 0.2319391816854477, + 0.2541576027870178 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5670980215072632, + "mean_weights": [ + 0.2082798331975937, + 0.19887971878051758, + 0.1370011270046234, + 0.1529916375875473, + 0.3028477132320404 + ], + "sources": 5 + }, + { + "entropy_mean": 1.581557273864746, + "mean_weights": [ + 0.1857318878173828, + 0.14141929149627686, + 0.19106487929821014, + 0.21634675562381744, + 0.26543718576431274 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5962812900543213, + "mean_weights": [ + 0.20828455686569214, + 0.18983566761016846, + 0.15771199762821198, + 0.19020363688468933, + 0.2539641261100769 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7401354312896729, + "mean_weights": [ + 0.13164246082305908, + 0.11492803692817688, + 0.15124152600765228, + 0.17682284116744995, + 0.15447166562080383, + 0.27089351415634155 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7471147775650024, + "mean_weights": [ + 0.15525710582733154, + 0.14720062911510468, + 0.11431462317705154, + 0.13530737161636353, + 0.17593735456466675, + 0.27198290824890137 + ], + "sources": 6 + }, + { + "entropy_mean": 1.668056607246399, + "mean_weights": [ + 0.11600004136562347, + 0.105022132396698, + 0.143685981631279, + 0.17166738212108612, + 0.11442092061042786, + 0.34920355677604675 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7639002799987793, + "mean_weights": [ + 0.16225872933864594, + 0.1667022705078125, + 0.12461072206497192, + 0.13374266028404236, + 0.1680135726928711, + 0.244672030210495 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6540379524230957, + "mean_weights": [ + 0.12023074924945831, + 0.10879058390855789, + 0.1446353942155838, + 0.16682270169258118, + 0.10848808288574219, + 0.3510324954986572 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7788331508636475, + "mean_weights": [ + 0.16649304330348969, + 0.17372167110443115, + 0.138288214802742, + 0.13879621028900146, + 0.1686137169599533, + 0.2140871286392212 + ], + "sources": 6 + }, + { + "entropy_mean": 1.642655849456787, + "mean_weights": [ + 0.12200242280960083, + 0.11414028704166412, + 0.1496569961309433, + 0.16737672686576843, + 0.0977313220500946, + 0.3490922451019287 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9349592924118042, + "mean_weights": [ + 0.15229427814483643, + 0.17081031203269958, + 0.12453728169202805, + 0.11576162278652191, + 0.15978330373764038, + 0.13195550441741943, + 0.14485770463943481 + ], + "sources": 7 + }, + { + "entropy_mean": 1.903893232345581, + "mean_weights": [ + 0.08074598014354706, + 0.08101454377174377, + 0.09124866127967834, + 0.09122030436992645, + 0.05705389380455017, + 0.12553787231445312, + 0.18838009238243103, + 0.28479865193367004 + ], + "sources": 8 + }, + { + "entropy_mean": 1.953556776046753, + "mean_weights": [ + 0.10463656485080719, + 0.1164495050907135, + 0.09414154291152954, + 0.08574216812849045, + 0.11312596499919891, + 0.09258253872394562, + 0.09613300859928131, + 0.29718872904777527 + ], + "sources": 8 + }, + { + "entropy_mean": 1.7883436679840088, + "mean_weights": [ + 0.07738837599754333, + 0.07766823470592499, + 0.0796431303024292, + 0.07871648669242859, + 0.05931882560253143, + 0.09263359755277634, + 0.1354067176580429, + 0.3992246389389038 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9697327613830566, + "mean_weights": [ + 0.11042678356170654, + 0.11892420798540115, + 0.10254333168268204, + 0.09763654321432114, + 0.11291946470737457, + 0.09096341580152512, + 0.09030745923519135, + 0.2762787640094757 + ], + "sources": 8 + }, + { + "entropy_mean": 1.8165807723999023, + "mean_weights": [ + 0.07695324718952179, + 0.08730179816484451, + 0.08890746533870697, + 0.0937752053141594, + 0.06407897919416428, + 0.08820104598999023, + 0.1218956708908081, + 0.3788866102695465 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9957175254821777, + "mean_weights": [ + 0.1092485785484314, + 0.12386660277843475, + 0.1201288253068924, + 0.1279536783695221, + 0.10448656231164932, + 0.08814899623394012, + 0.09167329221963882, + 0.2344934642314911 + ], + "sources": 8 + }, + { + "entropy_mean": 1.925641417503357, + "mean_weights": [ + 0.08538340032100677, + 0.09712239354848862, + 0.10027294605970383, + 0.11003522574901581, + 0.0732574313879013, + 0.0973970890045166, + 0.135427325963974, + 0.30110421776771545 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0462050437927246, + "mean_weights": [ + 0.1081143319606781, + 0.12867756187915802, + 0.12420770525932312, + 0.12876126170158386, + 0.11138664186000824, + 0.11174632608890533, + 0.12069624662399292, + 0.1664099246263504 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0244405269622803, + "mean_weights": [ + 0.08513332903385162, + 0.10402016341686249, + 0.0838124006986618, + 0.09660559892654419, + 0.05869138985872269, + 0.09330513328313828, + 0.10670876502990723, + 0.16347047686576843, + 0.20825274288654327 + ], + "sources": 9 + }, + { + "entropy_mean": 2.143857479095459, + "mean_weights": [ + 0.10113483667373657, + 0.12130618095397949, + 0.10747639834880829, + 0.1146310567855835, + 0.12467692792415619, + 0.10899603366851807, + 0.09931069612503052, + 0.10536618530750275, + 0.11710168421268463 + ], + "sources": 9 + }, + { + "entropy_mean": 1.973613977432251, + "mean_weights": [ + 0.09347088634967804, + 0.0957702025771141, + 0.08175105601549149, + 0.09041690826416016, + 0.05975842475891113, + 0.05595116317272186, + 0.08291695266962051, + 0.2648966312408447, + 0.17506778240203857 + ], + "sources": 9 + }, + { + "entropy_mean": 2.148151397705078, + "mean_weights": [ + 0.09436874836683273, + 0.10471053421497345, + 0.12474234402179718, + 0.13783809542655945, + 0.09644931554794312, + 0.1054454892873764, + 0.09779924154281616, + 0.12052080035209656, + 0.11812540888786316 + ], + "sources": 9 + }, + { + "entropy_mean": 2.067427635192871, + "mean_weights": [ + 0.08839868009090424, + 0.09813904017210007, + 0.09896856546401978, + 0.11030212044715881, + 0.07329557090997696, + 0.08898115158081055, + 0.12178729474544525, + 0.1831226348876953, + 0.13700494170188904 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1650590896606445, + "mean_weights": [ + 0.0997687578201294, + 0.10276617854833603, + 0.12921583652496338, + 0.12793958187103271, + 0.10596772283315659, + 0.10084287822246552, + 0.10392510890960693, + 0.11880208551883698, + 0.11077185720205307 + ], + "sources": 9 + }, + { + "entropy_mean": 2.130171060562134, + "mean_weights": [ + 0.08872106671333313, + 0.11083720624446869, + 0.11231885105371475, + 0.12914228439331055, + 0.07280043512582779, + 0.10742412507534027, + 0.1417485773563385, + 0.1481785774230957, + 0.08882887661457062 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06223799288272858, + 0.058558978140354156, + 0.06285873800516129, + 0.07834011316299438, + 0.10349078476428986, + 0.1494315266609192, + 0.12946709990501404, + 0.09041519463062286, + 0.12573887407779694, + 0.0998830571770668, + 0.10262355208396912, + 0.1128225028514862, + 0.09769245237112045, + 0.09893641620874405, + 0.10000422596931458, + 0.10479776561260223, + 0.09525298327207565, + 0.06153881549835205, + 0.06782867759466171, + 0.0726822018623352, + 0.06832144409418106, + 0.05976621061563492, + 0.06804215162992477, + 0.06079404056072235, + 0.0709940493106842, + 0.0526244081556797, + 0.05209306627511978, + 0.04965999722480774, + 0.05721710994839668, + 0.05419401451945305, + 0.06466542929410934, + 0.05974492058157921, + 0.06542418897151947, + 0.043975457549095154, + 0.04941371828317642, + 0.04448845237493515, + 0.054095908999443054, + 0.051386669278144836, + 0.05759691447019577, + 0.059779901057481766, + 0.059096306562423706, + 0.0506875142455101, + 0.051275305449962616, + 0.05152525007724762, + 0.05263759195804596, + 0.052927058190107346, + 0.05443885922431946, + 0.05457156151533127, + 0.05899491533637047, + 0.049417972564697266, + 0.044221431016922, + 0.046047866344451904, + 0.04773344472050667, + 0.05176534131169319, + 0.05198556184768677, + 0.0555768720805645, + 0.05468311160802841, + 0.04748184233903885, + 0.050643667578697205, + 0.05846012011170387, + 0.05456284433603287, + 0.06102273240685463, + 0.05901077762246132, + 0.0596291646361351 + ], + "loss_nats": 1.2962347269058228, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.7550932168960571, + "mean_weights": [ + 0.09226249158382416, + 0.06850043684244156, + 0.06784362345933914, + 0.049600355327129364, + 0.05115930736064911, + 0.0401824489235878, + 0.046749405562877655, + 0.27866458892822266, + 0.30503731966018677 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.06266748160123825, + 0.11405637860298157, + 0.18758949637413025, + 0.19576610624790192, + 0.2249695509672165, + 0.22915078699588776, + 0.23145067691802979, + 0.241273432970047, + 0.11456266790628433, + 0.11842832714319229, + 0.13031284511089325, + 0.1310930997133255, + 0.137657031416893, + 0.13807407021522522, + 0.14228036999702454, + 0.14413440227508545, + 0.03705674782395363, + 0.043267782777547836, + 0.057188767939805984, + 0.0690729096531868, + 0.08344521373510361, + 0.09961047768592834, + 0.1215684786438942, + 0.14613077044487, + 0.06314649432897568, + 0.08204278349876404, + 0.09940356016159058, + 0.11789657920598984, + 0.12692086398601532, + 0.1456192433834076, + 0.1643700897693634, + 0.18111956119537354, + 0.03456971421837807, + 0.0510682575404644, + 0.06312253326177597, + 0.0844164788722992, + 0.09254997968673706, + 0.11892655491828918, + 0.12588626146316528, + 0.1606190800666809, + 0.047963351011276245, + 0.060701869428157806, + 0.08735287934541702, + 0.10386133193969727, + 0.12329500913619995, + 0.1428302675485611, + 0.15417346358299255, + 0.17762337625026703, + 0.02691013365983963, + 0.0571737065911293, + 0.06209211051464081, + 0.08602745085954666, + 0.08898380398750305, + 0.11427143961191177, + 0.11587047576904297, + 0.15259574353694916, + 0.01650204323232174, + 0.06466396898031235, + 0.06881320476531982, + 0.14282406866550446, + 0.1444701850414276, + 0.22156114876270294, + 0.2232610583305359, + 0.3222712576389313 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 8.061194200882085, + "cross_entropy_nats": 5.587594032287598, + "step": 0 + }, + { + "bits_per_byte": 5.192153993773732, + "cross_entropy_nats": 3.5989269018173218, + "step": 100 + }, + { + "bits_per_byte": 3.090815919983902, + "cross_entropy_nats": 2.142390340566635, + "step": 500 + }, + { + "bits_per_byte": 1.954220330623464, + "cross_entropy_nats": 1.3545623123645782, + "step": 2000 + }, + { + "bits_per_byte": 1.7888187678785716, + "cross_entropy_nats": 1.2399146854877472, + "step": 4000 + }, + { + "bits_per_byte": 1.7130004957175102, + "cross_entropy_nats": 1.1873614639043808, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.40.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.40.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "40": true, + "41": true, + "42": true, + "43": true, + "44": true, + "45": true, + "46": true, + "47": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.49.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.49.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.51.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.51.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.53.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.53.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.55.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.55.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_group_6_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "e1ef26e87f2eb2dcc47a22d93cfd93f069e2d29d7666640235252c3c965680f7", + "final_model_state": "df85a72b302fe9802bec5cb320f771b19b18c35aa56da5ab4aca1f5af3c613ae", + "final_optimizer_state": "b8fd16695f95c15b06ffc9a67a36babac3c010dab2883b5b5482a7649981445c", + "final_public_parameters": "3b56c7a1ba4e39b696ac1c7cbdf15eb6d4cce3ec30d27e0b87bc28b40ddc29a2", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "3f93c11463780b386fff00c53b0424f687e1136b1b80f7b626f058adef96c764" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "3eff5c07fa0cfbd5ceed5922cbd09525ad514f801cc0a440c03fd2fa15df4c84", + "1": "e83e4001c5811b19c69a54c7ea7a802af996a34aa57072cde43a76bc8369e2d0", + "7999": "d04c4ca879f0209183a48ac54a0c4ffe3a0f62a78888479d4641f6c304426442" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "0f77f5ecd3463182d2094728da24c8d1b2e47d99748001b671106c21c4c4ea02", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073003, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 211.15612545946263, + "measured_steps": 7980, + "median_ms": 211.53193300415296, + "p95_ms": 214.05222918401705, + "peak_allocated_bytes": 11759431680, + "peak_reserved_bytes": 12266242048, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 8.082576459898554, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.602415084838867, + "step": 1, + "unclipped_grad_norm": 19.082603454589844 + }, + { + "bits_per_byte": 7.337206402781557, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.085763931274414, + "step": 10, + "unclipped_grad_norm": 11.737322807312012 + }, + { + "bits_per_byte": 6.794616273782671, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.70966911315918, + "step": 20, + "unclipped_grad_norm": 3.370713472366333 + }, + { + "bits_per_byte": 6.6697609968398925, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.623126029968262, + "step": 30, + "unclipped_grad_norm": 3.037761926651001 + }, + { + "bits_per_byte": 6.488848318813684, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.497726917266846, + "step": 40, + "unclipped_grad_norm": 3.0712175369262695 + }, + { + "bits_per_byte": 6.170700364704988, + "learning_rate": 3.75e-05, + "loss_nats": 4.277203559875488, + "step": 50, + "unclipped_grad_norm": 2.826653480529785 + }, + { + "bits_per_byte": 5.94597545033678, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.12143611907959, + "step": 60, + "unclipped_grad_norm": 2.602217674255371 + }, + { + "bits_per_byte": 5.7640797194782385, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.9953556060791016, + "step": 70, + "unclipped_grad_norm": 2.521389961242676 + }, + { + "bits_per_byte": 5.595806390429239, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.8787174224853516, + "step": 80, + "unclipped_grad_norm": 2.595237970352173 + }, + { + "bits_per_byte": 5.40536380975342, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.7467126846313477, + "step": 90, + "unclipped_grad_norm": 2.317441940307617 + }, + { + "bits_per_byte": 5.178297179611064, + "learning_rate": 7.5e-05, + "loss_nats": 3.589322090148926, + "step": 100, + "unclipped_grad_norm": 2.215688943862915 + }, + { + "bits_per_byte": 4.995869678396056, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.4628729820251465, + "step": 110, + "unclipped_grad_norm": 2.034050703048706 + }, + { + "bits_per_byte": 4.79938225150771, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.3266782760620117, + "step": 120, + "unclipped_grad_norm": 1.8054733276367188 + }, + { + "bits_per_byte": 4.633903984500534, + "learning_rate": 9.75e-05, + "loss_nats": 3.211977481842041, + "step": 130, + "unclipped_grad_norm": 1.596373200416565 + }, + { + "bits_per_byte": 4.47743829632917, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.1035237312316895, + "step": 140, + "unclipped_grad_norm": 1.3897442817687988 + }, + { + "bits_per_byte": 4.276164592267314, + "learning_rate": 0.0001125, + "loss_nats": 2.9640114307403564, + "step": 150, + "unclipped_grad_norm": 1.2067102193832397 + }, + { + "bits_per_byte": 4.129668050435865, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.8624677658081055, + "step": 160, + "unclipped_grad_norm": 1.0454909801483154 + }, + { + "bits_per_byte": 4.022488806383464, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.7881767749786377, + "step": 170, + "unclipped_grad_norm": 0.9107083678245544 + }, + { + "bits_per_byte": 3.9199599732103936, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.717109203338623, + "step": 180, + "unclipped_grad_norm": 0.8681561350822449 + }, + { + "bits_per_byte": 3.831916550788894, + "learning_rate": 0.0001425, + "loss_nats": 2.6560821533203125, + "step": 190, + "unclipped_grad_norm": 0.8218041658401489 + }, + { + "bits_per_byte": 3.7630299318817015, + "learning_rate": 0.00015, + "loss_nats": 2.6083335876464844, + "step": 200, + "unclipped_grad_norm": 0.7164890766143799 + }, + { + "bits_per_byte": 3.6870682906833143, + "learning_rate": 0.0001575, + "loss_nats": 2.555680990219116, + "step": 210, + "unclipped_grad_norm": 1.2349090576171875 + }, + { + "bits_per_byte": 3.6559899937764997, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.5341391563415527, + "step": 220, + "unclipped_grad_norm": 0.981060266494751 + }, + { + "bits_per_byte": 3.6039913832534314, + "learning_rate": 0.0001725, + "loss_nats": 2.498096466064453, + "step": 230, + "unclipped_grad_norm": 0.8902397155761719 + }, + { + "bits_per_byte": 3.5877672279016375, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.4868507385253906, + "step": 240, + "unclipped_grad_norm": 0.9120654463768005 + }, + { + "bits_per_byte": 3.510829757142255, + "learning_rate": 0.0001875, + "loss_nats": 2.4335217475891113, + "step": 250, + "unclipped_grad_norm": 0.972642183303833 + }, + { + "bits_per_byte": 3.5162248528998212, + "learning_rate": 0.000195, + "loss_nats": 2.4372613430023193, + "step": 260, + "unclipped_grad_norm": 0.915929913520813 + }, + { + "bits_per_byte": 3.4991438799814287, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.425421714782715, + "step": 270, + "unclipped_grad_norm": 0.9777015447616577 + }, + { + "bits_per_byte": 3.445589858363957, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.388300895690918, + "step": 280, + "unclipped_grad_norm": 0.9333418011665344 + }, + { + "bits_per_byte": 3.4375018582591808, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.382694721221924, + "step": 290, + "unclipped_grad_norm": 1.219210147857666 + }, + { + "bits_per_byte": 3.426231835148141, + "learning_rate": 0.000225, + "loss_nats": 2.374882936477661, + "step": 300, + "unclipped_grad_norm": 1.1797494888305664 + }, + { + "bits_per_byte": 3.3153456770390433, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.298022508621216, + "step": 310, + "unclipped_grad_norm": 0.8957244157791138 + }, + { + "bits_per_byte": 3.3951937821816283, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.3533689975738525, + "step": 320, + "unclipped_grad_norm": 1.042117953300476 + }, + { + "bits_per_byte": 3.4010102354338083, + "learning_rate": 0.0002475, + "loss_nats": 2.35740065574646, + "step": 330, + "unclipped_grad_norm": 0.9022042155265808 + }, + { + "bits_per_byte": 3.3227364594772215, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.303145408630371, + "step": 340, + "unclipped_grad_norm": 0.9215548038482666 + }, + { + "bits_per_byte": 3.3096905435133723, + "learning_rate": 0.0002625, + "loss_nats": 2.294102668762207, + "step": 350, + "unclipped_grad_norm": 1.0587217807769775 + }, + { + "bits_per_byte": 3.3072986088051723, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.2924447059631348, + "step": 360, + "unclipped_grad_norm": 1.1592196226119995 + }, + { + "bits_per_byte": 3.3057968562978384, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.2914037704467773, + "step": 370, + "unclipped_grad_norm": 0.93410724401474 + }, + { + "bits_per_byte": 3.205422965740079, + "learning_rate": 0.000285, + "loss_nats": 2.221829891204834, + "step": 380, + "unclipped_grad_norm": 1.0708391666412354 + }, + { + "bits_per_byte": 3.2849271055809384, + "learning_rate": 0.0002925, + "loss_nats": 2.276937961578369, + "step": 390, + "unclipped_grad_norm": 1.0729241371154785 + }, + { + "bits_per_byte": 3.273288007701147, + "learning_rate": 0.0003, + "loss_nats": 2.2688703536987305, + "step": 400, + "unclipped_grad_norm": 0.9226364493370056 + }, + { + "bits_per_byte": 3.2799526793873826, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.2734899520874023, + "step": 410, + "unclipped_grad_norm": 1.096868872642517 + }, + { + "bits_per_byte": 3.207251485284053, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.223097324371338, + "step": 420, + "unclipped_grad_norm": 1.038764238357544 + }, + { + "bits_per_byte": 3.221543931501355, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.233004093170166, + "step": 430, + "unclipped_grad_norm": 0.9258162975311279 + }, + { + "bits_per_byte": 3.1691143324552247, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.196662664413452, + "step": 440, + "unclipped_grad_norm": 1.2732911109924316 + }, + { + "bits_per_byte": 3.242310148662436, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.2473981380462646, + "step": 450, + "unclipped_grad_norm": 0.930205762386322 + }, + { + "bits_per_byte": 3.1718436971244177, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.198554515838623, + "step": 460, + "unclipped_grad_norm": 0.9386995434761047 + }, + { + "bits_per_byte": 3.1297127631772534, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.169351577758789, + "step": 470, + "unclipped_grad_norm": 0.8871971368789673 + }, + { + "bits_per_byte": 3.1283950321063427, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.168438196182251, + "step": 480, + "unclipped_grad_norm": 1.1018937826156616 + }, + { + "bits_per_byte": 3.1044402565982656, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.151834011077881, + "step": 490, + "unclipped_grad_norm": 1.415128231048584 + }, + { + "bits_per_byte": 3.041280659964804, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.1080551147460938, + "step": 500, + "unclipped_grad_norm": 1.0303574800491333 + }, + { + "bits_per_byte": 3.109993920359927, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.1556835174560547, + "step": 510, + "unclipped_grad_norm": 0.9062967300415039 + }, + { + "bits_per_byte": 3.1283812734942735, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.168428659439087, + "step": 520, + "unclipped_grad_norm": 1.1208547353744507 + }, + { + "bits_per_byte": 2.9833940514067967, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.0679311752319336, + "step": 530, + "unclipped_grad_norm": 1.1758267879486084 + }, + { + "bits_per_byte": 2.996769142164397, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.077202081680298, + "step": 540, + "unclipped_grad_norm": 1.405828833580017 + }, + { + "bits_per_byte": 2.990747341627087, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.073028087615967, + "step": 550, + "unclipped_grad_norm": 0.937627375125885 + }, + { + "bits_per_byte": 3.0122448290196364, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.0879290103912354, + "step": 560, + "unclipped_grad_norm": 1.1108545064926147 + }, + { + "bits_per_byte": 2.969415989475271, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.0582423210144043, + "step": 570, + "unclipped_grad_norm": 1.5421582460403442 + }, + { + "bits_per_byte": 2.95570002910366, + "learning_rate": 0.000299626473885547, + "loss_nats": 2.0487351417541504, + "step": 580, + "unclipped_grad_norm": 1.1428570747375488 + }, + { + "bits_per_byte": 2.9149576830795625, + "learning_rate": 0.00029958384005397226, + "loss_nats": 2.0204946994781494, + "step": 590, + "unclipped_grad_norm": 1.0507310628890991 + }, + { + "bits_per_byte": 2.8734224970349875, + "learning_rate": 0.0002995389065559004, + "loss_nats": 1.9917047023773193, + "step": 600, + "unclipped_grad_norm": 1.1938703060150146 + }, + { + "bits_per_byte": 2.8868911463546576, + "learning_rate": 0.000299491674159121, + "loss_nats": 2.001040458679199, + "step": 610, + "unclipped_grad_norm": 0.9599721431732178 + }, + { + "bits_per_byte": 2.8778992054369463, + "learning_rate": 0.00029944214367070535, + "loss_nats": 1.9948077201843262, + "step": 620, + "unclipped_grad_norm": 1.095024585723877 + }, + { + "bits_per_byte": 2.7956495345577133, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.9377965927124023, + "step": 630, + "unclipped_grad_norm": 1.385589599609375 + }, + { + "bits_per_byte": 2.7874158651476564, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.9320894479751587, + "step": 640, + "unclipped_grad_norm": 1.2236021757125854 + }, + { + "bits_per_byte": 2.783205557871884, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.929171085357666, + "step": 650, + "unclipped_grad_norm": 1.0698883533477783 + }, + { + "bits_per_byte": 2.807495355583842, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.946007490158081, + "step": 660, + "unclipped_grad_norm": 1.2208949327468872 + }, + { + "bits_per_byte": 2.7686520419905714, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.9190833568572998, + "step": 670, + "unclipped_grad_norm": 0.8931671977043152 + }, + { + "bits_per_byte": 2.7595820209493684, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.9127964973449707, + "step": 680, + "unclipped_grad_norm": 1.2853102684020996 + }, + { + "bits_per_byte": 2.733979307680724, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.895050048828125, + "step": 690, + "unclipped_grad_norm": 1.4168299436569214 + }, + { + "bits_per_byte": 2.7111513625011, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.8792269229888916, + "step": 700, + "unclipped_grad_norm": 1.0547316074371338 + }, + { + "bits_per_byte": 2.6706659585573913, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.8511645793914795, + "step": 710, + "unclipped_grad_norm": 0.8782731294631958 + }, + { + "bits_per_byte": 2.7018438334190558, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.8727754354476929, + "step": 720, + "unclipped_grad_norm": 1.0192604064941406 + }, + { + "bits_per_byte": 2.6606472812140294, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.8442201614379883, + "step": 730, + "unclipped_grad_norm": 1.0695395469665527 + }, + { + "bits_per_byte": 2.6260323330748716, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.8202269077301025, + "step": 740, + "unclipped_grad_norm": 1.480175495147705 + }, + { + "bits_per_byte": 2.6255402907107532, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.8198858499526978, + "step": 750, + "unclipped_grad_norm": 0.9765214323997498 + }, + { + "bits_per_byte": 2.5284526446456805, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.7525898218154907, + "step": 760, + "unclipped_grad_norm": 0.9698607921600342 + }, + { + "bits_per_byte": 2.573440210423676, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.7837728261947632, + "step": 770, + "unclipped_grad_norm": 0.9029173851013184 + }, + { + "bits_per_byte": 2.5022725856180896, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.734443187713623, + "step": 780, + "unclipped_grad_norm": 1.0331801176071167 + }, + { + "bits_per_byte": 2.5215539045716207, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.7478079795837402, + "step": 790, + "unclipped_grad_norm": 0.9747472405433655 + }, + { + "bits_per_byte": 2.419015612352753, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.6767338514328003, + "step": 800, + "unclipped_grad_norm": 1.0393155813217163 + }, + { + "bits_per_byte": 2.4560752938739254, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7024216651916504, + "step": 810, + "unclipped_grad_norm": 1.0629769563674927 + }, + { + "bits_per_byte": 2.449581916907949, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.697920799255371, + "step": 820, + "unclipped_grad_norm": 0.8911488652229309 + }, + { + "bits_per_byte": 2.459890213035364, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.7050659656524658, + "step": 830, + "unclipped_grad_norm": 0.9616649746894836 + }, + { + "bits_per_byte": 2.451658435434467, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.6993601322174072, + "step": 840, + "unclipped_grad_norm": 1.0941317081451416 + }, + { + "bits_per_byte": 2.3913141947956293, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.6575326919555664, + "step": 850, + "unclipped_grad_norm": 1.0704880952835083 + }, + { + "bits_per_byte": 2.4410832222329124, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.6920299530029297, + "step": 860, + "unclipped_grad_norm": 0.9012364745140076 + }, + { + "bits_per_byte": 2.4418848333685843, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.6925855875015259, + "step": 870, + "unclipped_grad_norm": 1.1609525680541992 + }, + { + "bits_per_byte": 2.380194140556142, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.649824857711792, + "step": 880, + "unclipped_grad_norm": 1.036691665649414 + }, + { + "bits_per_byte": 2.340990523309314, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.6226509809494019, + "step": 890, + "unclipped_grad_norm": 0.9522600173950195 + }, + { + "bits_per_byte": 2.3831959257443014, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.6519055366516113, + "step": 900, + "unclipped_grad_norm": 0.9157639741897583 + }, + { + "bits_per_byte": 2.368985859216762, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.642055869102478, + "step": 910, + "unclipped_grad_norm": 1.0320746898651123 + }, + { + "bits_per_byte": 2.343887743045749, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.6246591806411743, + "step": 920, + "unclipped_grad_norm": 0.8064703345298767 + }, + { + "bits_per_byte": 2.297710573271736, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.592651605606079, + "step": 930, + "unclipped_grad_norm": 0.9956693053245544 + }, + { + "bits_per_byte": 2.342918448825486, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.6239873170852661, + "step": 940, + "unclipped_grad_norm": 0.8546867370605469 + }, + { + "bits_per_byte": 2.365580602729678, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.6396955251693726, + "step": 950, + "unclipped_grad_norm": 0.8900701999664307 + }, + { + "bits_per_byte": 2.2890865032442202, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.5866738557815552, + "step": 960, + "unclipped_grad_norm": 0.8689322471618652 + }, + { + "bits_per_byte": 2.2414366460308686, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5536454916000366, + "step": 970, + "unclipped_grad_norm": 1.1370890140533447 + }, + { + "bits_per_byte": 2.3883671000904445, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.6554899215698242, + "step": 980, + "unclipped_grad_norm": 1.0618544816970825 + }, + { + "bits_per_byte": 2.221072352324854, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.5395300388336182, + "step": 990, + "unclipped_grad_norm": 0.8828047513961792 + }, + { + "bits_per_byte": 2.2383739789843036, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.5515226125717163, + "step": 1000, + "unclipped_grad_norm": 0.8012931942939758 + }, + { + "bits_per_byte": 2.2306934057794217, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.546198844909668, + "step": 1010, + "unclipped_grad_norm": 1.2355647087097168 + }, + { + "bits_per_byte": 2.203646554191482, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.5274513959884644, + "step": 1020, + "unclipped_grad_norm": 0.9753696322441101 + }, + { + "bits_per_byte": 2.1944582090864855, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.5210825204849243, + "step": 1030, + "unclipped_grad_norm": 0.9449633955955505 + }, + { + "bits_per_byte": 2.293086819703289, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.5894466638565063, + "step": 1040, + "unclipped_grad_norm": 1.1114345788955688 + }, + { + "bits_per_byte": 2.187330904069429, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5161422491073608, + "step": 1050, + "unclipped_grad_norm": 0.8976044058799744 + }, + { + "bits_per_byte": 2.2351890322729755, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.5493149757385254, + "step": 1060, + "unclipped_grad_norm": 0.9080505967140198 + }, + { + "bits_per_byte": 2.2739128179238963, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.5761562585830688, + "step": 1070, + "unclipped_grad_norm": 0.9085085391998291 + }, + { + "bits_per_byte": 2.2252809397741182, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.5424472093582153, + "step": 1080, + "unclipped_grad_norm": 0.7398526668548584 + }, + { + "bits_per_byte": 2.1792685293796312, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.5105538368225098, + "step": 1090, + "unclipped_grad_norm": 0.951080858707428 + }, + { + "bits_per_byte": 2.179716372202478, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5108642578125, + "step": 1100, + "unclipped_grad_norm": 0.8834229111671448 + }, + { + "bits_per_byte": 2.141559613351452, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.4844160079956055, + "step": 1110, + "unclipped_grad_norm": 1.1409209966659546 + }, + { + "bits_per_byte": 2.182815155605724, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.513012170791626, + "step": 1120, + "unclipped_grad_norm": 0.8327498435974121 + }, + { + "bits_per_byte": 2.2026463030940637, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.526758074760437, + "step": 1130, + "unclipped_grad_norm": 0.8334863185882568 + }, + { + "bits_per_byte": 2.124281204349869, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.4724395275115967, + "step": 1140, + "unclipped_grad_norm": 0.8589467406272888 + }, + { + "bits_per_byte": 2.1806211729286673, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.5114914178848267, + "step": 1150, + "unclipped_grad_norm": 0.9057679772377014 + }, + { + "bits_per_byte": 2.098079991456222, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.4542782306671143, + "step": 1160, + "unclipped_grad_norm": 0.8293759822845459 + }, + { + "bits_per_byte": 2.1552697263129335, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.4939191341400146, + "step": 1170, + "unclipped_grad_norm": 0.8850358724594116 + }, + { + "bits_per_byte": 2.159579955508857, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.4969067573547363, + "step": 1180, + "unclipped_grad_norm": 0.8352997899055481 + }, + { + "bits_per_byte": 2.115943829401342, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.466660499572754, + "step": 1190, + "unclipped_grad_norm": 0.8651363849639893 + }, + { + "bits_per_byte": 2.1414894444299, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.4843673706054688, + "step": 1200, + "unclipped_grad_norm": 0.8325709104537964 + }, + { + "bits_per_byte": 2.1574195094487183, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.4954092502593994, + "step": 1210, + "unclipped_grad_norm": 0.8778349161148071 + }, + { + "bits_per_byte": 2.1014025242882406, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4565812349319458, + "step": 1220, + "unclipped_grad_norm": 0.9311639666557312 + }, + { + "bits_per_byte": 2.1224206960328353, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.4711499214172363, + "step": 1230, + "unclipped_grad_norm": 1.0151524543762207 + }, + { + "bits_per_byte": 2.0797149960664876, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.4415485858917236, + "step": 1240, + "unclipped_grad_norm": 0.8917344212532043 + }, + { + "bits_per_byte": 2.0192885477205382, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.3996641635894775, + "step": 1250, + "unclipped_grad_norm": 1.091322422027588 + }, + { + "bits_per_byte": 2.0817507547047507, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.4429596662521362, + "step": 1260, + "unclipped_grad_norm": 0.9715288281440735 + }, + { + "bits_per_byte": 2.108256548873076, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.461332082748413, + "step": 1270, + "unclipped_grad_norm": 0.9675281047821045 + }, + { + "bits_per_byte": 2.1349988191863374, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.4798684120178223, + "step": 1280, + "unclipped_grad_norm": 0.8518584370613098 + }, + { + "bits_per_byte": 2.0826158274385906, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4435592889785767, + "step": 1290, + "unclipped_grad_norm": 0.832960307598114 + }, + { + "bits_per_byte": 2.0385781218413106, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.4130346775054932, + "step": 1300, + "unclipped_grad_norm": 0.8997964859008789 + }, + { + "bits_per_byte": 2.061499969548306, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.4289228916168213, + "step": 1310, + "unclipped_grad_norm": 1.0375984907150269 + }, + { + "bits_per_byte": 2.0359749924378514, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.4112303256988525, + "step": 1320, + "unclipped_grad_norm": 0.8797754645347595 + }, + { + "bits_per_byte": 2.053762126120686, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.4235594272613525, + "step": 1330, + "unclipped_grad_norm": 0.8674715757369995 + }, + { + "bits_per_byte": 2.033246143716611, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.4093388319015503, + "step": 1340, + "unclipped_grad_norm": 0.7660273313522339 + }, + { + "bits_per_byte": 2.0604095995418357, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.4281671047210693, + "step": 1350, + "unclipped_grad_norm": 0.899191677570343 + }, + { + "bits_per_byte": 2.0210247125809984, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.4008675813674927, + "step": 1360, + "unclipped_grad_norm": 0.8374674916267395 + }, + { + "bits_per_byte": 2.0670173449706355, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4327472448349, + "step": 1370, + "unclipped_grad_norm": 0.8804115056991577 + }, + { + "bits_per_byte": 2.0214882058250736, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.401188850402832, + "step": 1380, + "unclipped_grad_norm": 0.9003057479858398 + }, + { + "bits_per_byte": 2.054174196552154, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.4238450527191162, + "step": 1390, + "unclipped_grad_norm": 0.8733509182929993 + }, + { + "bits_per_byte": 2.165684995649185, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.50113844871521, + "step": 1400, + "unclipped_grad_norm": 0.8712846040725708 + }, + { + "bits_per_byte": 2.0424778284496248, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.4157377481460571, + "step": 1410, + "unclipped_grad_norm": 0.7329201698303223 + }, + { + "bits_per_byte": 2.0420748730986533, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.4154584407806396, + "step": 1420, + "unclipped_grad_norm": 0.7460573315620422 + }, + { + "bits_per_byte": 1.9945694813120267, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.3825302124023438, + "step": 1430, + "unclipped_grad_norm": 0.8160272240638733 + }, + { + "bits_per_byte": 2.0108012039004586, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.3937811851501465, + "step": 1440, + "unclipped_grad_norm": 0.8164739012718201 + }, + { + "bits_per_byte": 1.9800395270538822, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.372458815574646, + "step": 1450, + "unclipped_grad_norm": 0.7698927521705627 + }, + { + "bits_per_byte": 2.1176428460092156, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.467838168144226, + "step": 1460, + "unclipped_grad_norm": 0.8664708137512207 + }, + { + "bits_per_byte": 2.090348511386684, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.4489191770553589, + "step": 1470, + "unclipped_grad_norm": 0.7238553166389465 + }, + { + "bits_per_byte": 2.035280870458969, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.4107491970062256, + "step": 1480, + "unclipped_grad_norm": 0.8418828845024109 + }, + { + "bits_per_byte": 2.062513979257793, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.4296257495880127, + "step": 1490, + "unclipped_grad_norm": 0.9077610969543457 + }, + { + "bits_per_byte": 1.9110426672674863, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.3246338367462158, + "step": 1500, + "unclipped_grad_norm": 0.6985825896263123 + }, + { + "bits_per_byte": 1.947417341890272, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.3498468399047852, + "step": 1510, + "unclipped_grad_norm": 0.778479278087616 + }, + { + "bits_per_byte": 1.9919094256511316, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.3806864023208618, + "step": 1520, + "unclipped_grad_norm": 0.7394455671310425 + }, + { + "bits_per_byte": 2.0335464254250177, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.409546971321106, + "step": 1530, + "unclipped_grad_norm": 0.7591404318809509 + }, + { + "bits_per_byte": 2.01788568523745, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3986917734146118, + "step": 1540, + "unclipped_grad_norm": 0.7885791063308716 + }, + { + "bits_per_byte": 2.069445911983469, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.4344305992126465, + "step": 1550, + "unclipped_grad_norm": 0.8274276852607727 + }, + { + "bits_per_byte": 1.9439874918841151, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.347469449043274, + "step": 1560, + "unclipped_grad_norm": 0.7710230946540833 + }, + { + "bits_per_byte": 1.9399820159455203, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.3446930646896362, + "step": 1570, + "unclipped_grad_norm": 0.7320961952209473 + }, + { + "bits_per_byte": 1.9740135989329513, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.3682819604873657, + "step": 1580, + "unclipped_grad_norm": 0.735421895980835 + }, + { + "bits_per_byte": 1.9706288083813202, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3659358024597168, + "step": 1590, + "unclipped_grad_norm": 0.8058050870895386 + }, + { + "bits_per_byte": 1.9762392264177668, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.3698246479034424, + "step": 1600, + "unclipped_grad_norm": 0.778346061706543 + }, + { + "bits_per_byte": 1.9268554401184164, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.3355944156646729, + "step": 1610, + "unclipped_grad_norm": 0.7580106854438782 + }, + { + "bits_per_byte": 1.9415167891218201, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.3457568883895874, + "step": 1620, + "unclipped_grad_norm": 0.7155437469482422 + }, + { + "bits_per_byte": 1.9722715866623617, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.3670744895935059, + "step": 1630, + "unclipped_grad_norm": 0.7651764750480652 + }, + { + "bits_per_byte": 1.9322120117621895, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.3393073081970215, + "step": 1640, + "unclipped_grad_norm": 0.7987560629844666 + }, + { + "bits_per_byte": 1.9279643842511798, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.3363630771636963, + "step": 1650, + "unclipped_grad_norm": 0.7138199806213379 + }, + { + "bits_per_byte": 1.939719054472351, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.344510793685913, + "step": 1660, + "unclipped_grad_norm": 0.7974153161048889 + }, + { + "bits_per_byte": 1.8931189793598662, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.3122100830078125, + "step": 1670, + "unclipped_grad_norm": 0.7638903260231018 + }, + { + "bits_per_byte": 1.9213323892686083, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.331766128540039, + "step": 1680, + "unclipped_grad_norm": 0.6874826550483704 + }, + { + "bits_per_byte": 1.949427131148255, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.3512399196624756, + "step": 1690, + "unclipped_grad_norm": 0.7064375281333923 + }, + { + "bits_per_byte": 1.8891049042887282, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.3094277381896973, + "step": 1700, + "unclipped_grad_norm": 0.7620424628257751 + }, + { + "bits_per_byte": 1.9439364130368089, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3474340438842773, + "step": 1710, + "unclipped_grad_norm": 0.723478376865387 + }, + { + "bits_per_byte": 1.861835507150572, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.290526032447815, + "step": 1720, + "unclipped_grad_norm": 1.052738904953003 + }, + { + "bits_per_byte": 1.9479998471287445, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.3502506017684937, + "step": 1730, + "unclipped_grad_norm": 0.8339725732803345 + }, + { + "bits_per_byte": 1.945108474802439, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.348246455192566, + "step": 1740, + "unclipped_grad_norm": 0.6603237986564636 + }, + { + "bits_per_byte": 1.9152669051379787, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.327561855316162, + "step": 1750, + "unclipped_grad_norm": 0.7067477703094482 + }, + { + "bits_per_byte": 1.9564861590529188, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.3561328649520874, + "step": 1760, + "unclipped_grad_norm": 0.7016445994377136 + }, + { + "bits_per_byte": 1.9338995055324553, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.3404769897460938, + "step": 1770, + "unclipped_grad_norm": 0.7062034010887146 + }, + { + "bits_per_byte": 2.007134189818762, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.3912394046783447, + "step": 1780, + "unclipped_grad_norm": 0.7647057771682739 + }, + { + "bits_per_byte": 1.8814325862510874, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3041096925735474, + "step": 1790, + "unclipped_grad_norm": 0.7370473146438599 + }, + { + "bits_per_byte": 1.999015404819481, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.385611891746521, + "step": 1800, + "unclipped_grad_norm": 0.7158631086349487 + }, + { + "bits_per_byte": 1.8851541188331078, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.3066892623901367, + "step": 1810, + "unclipped_grad_norm": 0.7807532548904419 + }, + { + "bits_per_byte": 1.9386429590259024, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.3437649011611938, + "step": 1820, + "unclipped_grad_norm": 0.7723674774169922 + }, + { + "bits_per_byte": 1.9592741698060558, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.3580653667449951, + "step": 1830, + "unclipped_grad_norm": 0.7110446095466614 + }, + { + "bits_per_byte": 1.9470986580382232, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.3496259450912476, + "step": 1840, + "unclipped_grad_norm": 0.7288746237754822 + }, + { + "bits_per_byte": 1.885808168854339, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.3071426153182983, + "step": 1850, + "unclipped_grad_norm": 0.8063361048698425 + }, + { + "bits_per_byte": 1.9015588559683076, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.3180601596832275, + "step": 1860, + "unclipped_grad_norm": 0.6838510036468506 + }, + { + "bits_per_byte": 1.991481360833134, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.38038969039917, + "step": 1870, + "unclipped_grad_norm": 0.7435280084609985 + }, + { + "bits_per_byte": 1.8848841060712531, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.306502103805542, + "step": 1880, + "unclipped_grad_norm": 0.9143924713134766 + }, + { + "bits_per_byte": 1.9101691673837542, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3240283727645874, + "step": 1890, + "unclipped_grad_norm": 0.7090442180633545 + }, + { + "bits_per_byte": 1.9052858919951556, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.3206435441970825, + "step": 1900, + "unclipped_grad_norm": 0.7039555907249451 + }, + { + "bits_per_byte": 1.946532663134234, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.349233627319336, + "step": 1910, + "unclipped_grad_norm": 0.7154620289802551 + }, + { + "bits_per_byte": 1.881784118789451, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.3043533563613892, + "step": 1920, + "unclipped_grad_norm": 0.7035974860191345 + }, + { + "bits_per_byte": 1.924546745013234, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.3339941501617432, + "step": 1930, + "unclipped_grad_norm": 0.6795576810836792 + }, + { + "bits_per_byte": 1.9292521903408406, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.3372557163238525, + "step": 1940, + "unclipped_grad_norm": 0.6555379629135132 + }, + { + "bits_per_byte": 1.8725747918010494, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.297969937324524, + "step": 1950, + "unclipped_grad_norm": 0.7438015937805176 + }, + { + "bits_per_byte": 1.8593075341655396, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.288773775100708, + "step": 1960, + "unclipped_grad_norm": 0.6981571316719055 + }, + { + "bits_per_byte": 1.8687198006819594, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.2952978610992432, + "step": 1970, + "unclipped_grad_norm": 0.7336557507514954 + }, + { + "bits_per_byte": 1.8925715585821699, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.3118306398391724, + "step": 1980, + "unclipped_grad_norm": 0.6622102856636047 + }, + { + "bits_per_byte": 1.8584992157064846, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.2882134914398193, + "step": 1990, + "unclipped_grad_norm": 0.6662092804908752 + }, + { + "bits_per_byte": 1.9555655359228503, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.355494737625122, + "step": 2000, + "unclipped_grad_norm": 0.7562788724899292 + }, + { + "bits_per_byte": 1.9488332750548256, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.3508282899856567, + "step": 2010, + "unclipped_grad_norm": 0.729736864566803 + }, + { + "bits_per_byte": 1.8402425693867939, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.2755589485168457, + "step": 2020, + "unclipped_grad_norm": 0.6693859100341797 + }, + { + "bits_per_byte": 1.8725505422472777, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.2979531288146973, + "step": 2030, + "unclipped_grad_norm": 0.6776761412620544 + }, + { + "bits_per_byte": 1.8869475539163052, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.3079323768615723, + "step": 2040, + "unclipped_grad_norm": 0.7430518865585327 + }, + { + "bits_per_byte": 1.8346459099624164, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.2716796398162842, + "step": 2050, + "unclipped_grad_norm": 0.6765186190605164 + }, + { + "bits_per_byte": 1.827109630201608, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.266455888748169, + "step": 2060, + "unclipped_grad_norm": 0.6557005643844604 + }, + { + "bits_per_byte": 1.9416727773861526, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.34586501121521, + "step": 2070, + "unclipped_grad_norm": 0.7226375937461853 + }, + { + "bits_per_byte": 1.9190603264680597, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.3301912546157837, + "step": 2080, + "unclipped_grad_norm": 0.6966984868049622 + }, + { + "bits_per_byte": 1.837759139908335, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.2738375663757324, + "step": 2090, + "unclipped_grad_norm": 0.6771838068962097 + }, + { + "bits_per_byte": 1.7848431739255644, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.237159013748169, + "step": 2100, + "unclipped_grad_norm": 0.7056940197944641 + }, + { + "bits_per_byte": 1.829667012219938, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.268228530883789, + "step": 2110, + "unclipped_grad_norm": 0.6758897304534912 + }, + { + "bits_per_byte": 1.8421940565261343, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.2769116163253784, + "step": 2120, + "unclipped_grad_norm": 0.7475795149803162 + }, + { + "bits_per_byte": 1.875291945702031, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.2998533248901367, + "step": 2130, + "unclipped_grad_norm": 0.7510239481925964 + }, + { + "bits_per_byte": 1.9128335226109208, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.325875163078308, + "step": 2140, + "unclipped_grad_norm": 0.8174769282341003 + }, + { + "bits_per_byte": 1.8074826261198427, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2528514862060547, + "step": 2150, + "unclipped_grad_norm": 0.7480066418647766 + }, + { + "bits_per_byte": 1.8939334891943524, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.312774658203125, + "step": 2160, + "unclipped_grad_norm": 0.7128459215164185 + }, + { + "bits_per_byte": 1.8217678490658094, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.2627532482147217, + "step": 2170, + "unclipped_grad_norm": 0.6698381304740906 + }, + { + "bits_per_byte": 1.8911325797424008, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.310833215713501, + "step": 2180, + "unclipped_grad_norm": 0.6556115746498108 + }, + { + "bits_per_byte": 1.8777533614011794, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.3015594482421875, + "step": 2190, + "unclipped_grad_norm": 0.7796351909637451 + }, + { + "bits_per_byte": 1.8403968378246178, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.2756658792495728, + "step": 2200, + "unclipped_grad_norm": 0.6594662666320801 + }, + { + "bits_per_byte": 1.8317486902259816, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.2696714401245117, + "step": 2210, + "unclipped_grad_norm": 0.6690157055854797 + }, + { + "bits_per_byte": 1.8290461548503234, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2677981853485107, + "step": 2220, + "unclipped_grad_norm": 0.6996455788612366 + }, + { + "bits_per_byte": 1.7924436032151445, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.2424272298812866, + "step": 2230, + "unclipped_grad_norm": 0.6348102688789368 + }, + { + "bits_per_byte": 1.870566894352226, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.2965781688690186, + "step": 2240, + "unclipped_grad_norm": 0.6588371396064758 + }, + { + "bits_per_byte": 1.8608777357679172, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2898621559143066, + "step": 2250, + "unclipped_grad_norm": 0.7125651240348816 + }, + { + "bits_per_byte": 1.909179063262737, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.3233420848846436, + "step": 2260, + "unclipped_grad_norm": 0.6912793517112732 + }, + { + "bits_per_byte": 1.795292495826687, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.2444019317626953, + "step": 2270, + "unclipped_grad_norm": 0.6560991406440735 + }, + { + "bits_per_byte": 1.8575213223536786, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.2875356674194336, + "step": 2280, + "unclipped_grad_norm": 0.684260368347168 + }, + { + "bits_per_byte": 1.809362568476424, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.2541545629501343, + "step": 2290, + "unclipped_grad_norm": 0.6960922479629517 + }, + { + "bits_per_byte": 1.8043076544022645, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.2506507635116577, + "step": 2300, + "unclipped_grad_norm": 0.6702517867088318 + }, + { + "bits_per_byte": 1.8117378208674904, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.2558009624481201, + "step": 2310, + "unclipped_grad_norm": 0.7420644760131836 + }, + { + "bits_per_byte": 1.855573962797959, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.28618586063385, + "step": 2320, + "unclipped_grad_norm": 0.7159541845321655 + }, + { + "bits_per_byte": 1.7973022850846696, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.2457950115203857, + "step": 2330, + "unclipped_grad_norm": 0.7151830196380615 + }, + { + "bits_per_byte": 1.815665732630546, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.2585235834121704, + "step": 2340, + "unclipped_grad_norm": 0.6740409731864929 + }, + { + "bits_per_byte": 1.8689483656249561, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2954562902450562, + "step": 2350, + "unclipped_grad_norm": 0.7084233164787292 + }, + { + "bits_per_byte": 1.8101318468737333, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.254687786102295, + "step": 2360, + "unclipped_grad_norm": 0.6939548850059509 + }, + { + "bits_per_byte": 1.7417291871460685, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.2072746753692627, + "step": 2370, + "unclipped_grad_norm": 0.7165483832359314 + }, + { + "bits_per_byte": 1.845390010127118, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.2791268825531006, + "step": 2380, + "unclipped_grad_norm": 0.6941385269165039 + }, + { + "bits_per_byte": 1.8367289638296667, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.2731235027313232, + "step": 2390, + "unclipped_grad_norm": 0.662189781665802 + }, + { + "bits_per_byte": 1.8676318384326014, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.294543743133545, + "step": 2400, + "unclipped_grad_norm": 0.7116144895553589 + }, + { + "bits_per_byte": 1.8038388296960124, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2503257989883423, + "step": 2410, + "unclipped_grad_norm": 0.7248636484146118 + }, + { + "bits_per_byte": 1.83553162061436, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.2722935676574707, + "step": 2420, + "unclipped_grad_norm": 0.7576844692230225 + }, + { + "bits_per_byte": 1.8324259579050792, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.2701408863067627, + "step": 2430, + "unclipped_grad_norm": 0.6794077754020691 + }, + { + "bits_per_byte": 1.7812184675759797, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.2346465587615967, + "step": 2440, + "unclipped_grad_norm": 0.6916407942771912 + }, + { + "bits_per_byte": 1.9030011024784432, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.3190598487854004, + "step": 2450, + "unclipped_grad_norm": 0.8297364115715027 + }, + { + "bits_per_byte": 1.779484194524679, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.2334444522857666, + "step": 2460, + "unclipped_grad_norm": 0.6565462350845337 + }, + { + "bits_per_byte": 1.7687934089817452, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.226034164428711, + "step": 2470, + "unclipped_grad_norm": 0.703306257724762 + }, + { + "bits_per_byte": 1.8705173633487775, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.296543836593628, + "step": 2480, + "unclipped_grad_norm": 0.6981465816497803 + }, + { + "bits_per_byte": 1.859620714572761, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.28899085521698, + "step": 2490, + "unclipped_grad_norm": 0.71137934923172 + }, + { + "bits_per_byte": 1.8141043021433625, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2574412822723389, + "step": 2500, + "unclipped_grad_norm": 0.6674412488937378 + }, + { + "bits_per_byte": 1.7615077079432442, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.2209841012954712, + "step": 2510, + "unclipped_grad_norm": 0.6648856401443481 + }, + { + "bits_per_byte": 1.7879491806001466, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.239311933517456, + "step": 2520, + "unclipped_grad_norm": 0.7041890621185303 + }, + { + "bits_per_byte": 1.8098955427114478, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.2545239925384521, + "step": 2530, + "unclipped_grad_norm": 0.7114644646644592 + }, + { + "bits_per_byte": 1.803409389016808, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.250028133392334, + "step": 2540, + "unclipped_grad_norm": 0.6633747220039368 + }, + { + "bits_per_byte": 1.7619464356855952, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.2212882041931152, + "step": 2550, + "unclipped_grad_norm": 0.6549026966094971 + }, + { + "bits_per_byte": 1.7259772961535438, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.1963562965393066, + "step": 2560, + "unclipped_grad_norm": 0.6431301832199097 + }, + { + "bits_per_byte": 1.7770465124313497, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2317547798156738, + "step": 2570, + "unclipped_grad_norm": 0.6401205658912659 + }, + { + "bits_per_byte": 1.7548688336546379, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.216382384300232, + "step": 2580, + "unclipped_grad_norm": 0.692520260810852 + }, + { + "bits_per_byte": 1.7537937701040947, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.21563720703125, + "step": 2590, + "unclipped_grad_norm": 0.7020792365074158 + }, + { + "bits_per_byte": 1.7696894385927404, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.2266552448272705, + "step": 2600, + "unclipped_grad_norm": 0.6673828959465027 + }, + { + "bits_per_byte": 1.8014697686803771, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.2486836910247803, + "step": 2610, + "unclipped_grad_norm": 0.6997250318527222 + }, + { + "bits_per_byte": 1.7237446173800428, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.1948087215423584, + "step": 2620, + "unclipped_grad_norm": 0.6837529540061951 + }, + { + "bits_per_byte": 1.7271527975721912, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.1971710920333862, + "step": 2630, + "unclipped_grad_norm": 0.6684131622314453 + }, + { + "bits_per_byte": 1.7091694316847217, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.1847059726715088, + "step": 2640, + "unclipped_grad_norm": 0.6933920383453369 + }, + { + "bits_per_byte": 1.790166208952419, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2408486604690552, + "step": 2650, + "unclipped_grad_norm": 0.6608225107192993 + }, + { + "bits_per_byte": 1.6945797994467278, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.174593210220337, + "step": 2660, + "unclipped_grad_norm": 0.6910310387611389 + }, + { + "bits_per_byte": 1.7624080371205109, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.2216081619262695, + "step": 2670, + "unclipped_grad_norm": 0.6782522201538086 + }, + { + "bits_per_byte": 1.8550206946101333, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.2858023643493652, + "step": 2680, + "unclipped_grad_norm": 0.6924003958702087 + }, + { + "bits_per_byte": 1.7754747629851144, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.2306653261184692, + "step": 2690, + "unclipped_grad_norm": 0.7133976221084595 + }, + { + "bits_per_byte": 1.7010434234141047, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.179073452949524, + "step": 2700, + "unclipped_grad_norm": 0.6410738825798035 + }, + { + "bits_per_byte": 1.7752486057992298, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.23050856590271, + "step": 2710, + "unclipped_grad_norm": 0.664742112159729 + }, + { + "bits_per_byte": 1.8416703693542569, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.2765486240386963, + "step": 2720, + "unclipped_grad_norm": 0.7220624685287476 + }, + { + "bits_per_byte": 1.7200920498410186, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.192276954650879, + "step": 2730, + "unclipped_grad_norm": 0.6437366008758545 + }, + { + "bits_per_byte": 1.795299891080674, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.244407057762146, + "step": 2740, + "unclipped_grad_norm": 0.6574398279190063 + }, + { + "bits_per_byte": 1.747453801662688, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.21124267578125, + "step": 2750, + "unclipped_grad_norm": 0.6696984767913818 + }, + { + "bits_per_byte": 1.8755141472869457, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.3000073432922363, + "step": 2760, + "unclipped_grad_norm": 0.7663789987564087 + }, + { + "bits_per_byte": 1.797370734179713, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.245842456817627, + "step": 2770, + "unclipped_grad_norm": 0.7259417176246643 + }, + { + "bits_per_byte": 1.8076852216825592, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.2529919147491455, + "step": 2780, + "unclipped_grad_norm": 0.6801440715789795 + }, + { + "bits_per_byte": 1.768307386010407, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2256972789764404, + "step": 2790, + "unclipped_grad_norm": 0.6557941436767578 + }, + { + "bits_per_byte": 1.781051644404643, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.2345309257507324, + "step": 2800, + "unclipped_grad_norm": 0.7006475925445557 + }, + { + "bits_per_byte": 1.7159199227137376, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.1893850564956665, + "step": 2810, + "unclipped_grad_norm": 0.6332662105560303 + }, + { + "bits_per_byte": 1.787192800901652, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2387876510620117, + "step": 2820, + "unclipped_grad_norm": 0.6763468384742737 + }, + { + "bits_per_byte": 1.7353988497331103, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.2028868198394775, + "step": 2830, + "unclipped_grad_norm": 0.6498988270759583 + }, + { + "bits_per_byte": 1.7725247445748649, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.2286205291748047, + "step": 2840, + "unclipped_grad_norm": 0.6563114523887634 + }, + { + "bits_per_byte": 1.7105934480338656, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.1856930255889893, + "step": 2850, + "unclipped_grad_norm": 0.6982893943786621 + }, + { + "bits_per_byte": 1.744808708492418, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.209409236907959, + "step": 2860, + "unclipped_grad_norm": 0.6822196841239929 + }, + { + "bits_per_byte": 1.7307295207621851, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.1996502876281738, + "step": 2870, + "unclipped_grad_norm": 0.647637128829956 + }, + { + "bits_per_byte": 1.7478215005702327, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2114975452423096, + "step": 2880, + "unclipped_grad_norm": 0.6412912607192993 + }, + { + "bits_per_byte": 1.7139610403204104, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.188027262687683, + "step": 2890, + "unclipped_grad_norm": 0.6399510502815247 + }, + { + "bits_per_byte": 1.7422705885309846, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.2076499462127686, + "step": 2900, + "unclipped_grad_norm": 0.6834374070167542 + }, + { + "bits_per_byte": 1.7005489732928742, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.1787307262420654, + "step": 2910, + "unclipped_grad_norm": 0.6549072861671448 + }, + { + "bits_per_byte": 1.7520421268050568, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.2144230604171753, + "step": 2920, + "unclipped_grad_norm": 0.6255326271057129 + }, + { + "bits_per_byte": 1.6984851814825204, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.177300214767456, + "step": 2930, + "unclipped_grad_norm": 0.6335311532020569 + }, + { + "bits_per_byte": 1.7199554956162335, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.1921823024749756, + "step": 2940, + "unclipped_grad_norm": 0.6447077989578247 + }, + { + "bits_per_byte": 1.7777138051166974, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2322173118591309, + "step": 2950, + "unclipped_grad_norm": 0.6793509125709534 + }, + { + "bits_per_byte": 1.775813912772616, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.2309004068374634, + "step": 2960, + "unclipped_grad_norm": 0.6853662729263306 + }, + { + "bits_per_byte": 1.7504342610021404, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.213308572769165, + "step": 2970, + "unclipped_grad_norm": 0.6683195233345032 + }, + { + "bits_per_byte": 1.7755738249920114, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.2307339906692505, + "step": 2980, + "unclipped_grad_norm": 0.6556857824325562 + }, + { + "bits_per_byte": 1.7081996215165063, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.184033751487732, + "step": 2990, + "unclipped_grad_norm": 0.7018770575523376 + }, + { + "bits_per_byte": 1.798498940369373, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.24662446975708, + "step": 3000, + "unclipped_grad_norm": 0.7035220861434937 + }, + { + "bits_per_byte": 1.689298212238731, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.1709322929382324, + "step": 3010, + "unclipped_grad_norm": 0.7039667367935181 + }, + { + "bits_per_byte": 1.7468270968829438, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.210808277130127, + "step": 3020, + "unclipped_grad_norm": 0.6851271986961365 + }, + { + "bits_per_byte": 1.6906014967669694, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.1718356609344482, + "step": 3030, + "unclipped_grad_norm": 0.6524337530136108 + }, + { + "bits_per_byte": 1.6821963606540022, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.1660096645355225, + "step": 3040, + "unclipped_grad_norm": 0.6792376041412354 + }, + { + "bits_per_byte": 1.8287329744431022, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.2675811052322388, + "step": 3050, + "unclipped_grad_norm": 0.8266787528991699 + }, + { + "bits_per_byte": 1.7654899662239725, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.2237443923950195, + "step": 3060, + "unclipped_grad_norm": 0.6917386651039124 + }, + { + "bits_per_byte": 1.6754147407651798, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.161309003829956, + "step": 3070, + "unclipped_grad_norm": 0.6908915638923645 + }, + { + "bits_per_byte": 1.7633876502998254, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.2222871780395508, + "step": 3080, + "unclipped_grad_norm": 0.6716691255569458 + }, + { + "bits_per_byte": 1.7571090796647768, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.2179352045059204, + "step": 3090, + "unclipped_grad_norm": 0.6809936165809631 + }, + { + "bits_per_byte": 1.7084952596933394, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.1842386722564697, + "step": 3100, + "unclipped_grad_norm": 0.6592872738838196 + }, + { + "bits_per_byte": 1.8477355095195849, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.2807526588439941, + "step": 3110, + "unclipped_grad_norm": 0.7689393758773804 + }, + { + "bits_per_byte": 1.7366233662072534, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.203735589981079, + "step": 3120, + "unclipped_grad_norm": 0.7119982838630676 + }, + { + "bits_per_byte": 1.6887815763555392, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.1705741882324219, + "step": 3130, + "unclipped_grad_norm": 0.6932662129402161 + }, + { + "bits_per_byte": 1.713512681549611, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.1877164840698242, + "step": 3140, + "unclipped_grad_norm": 0.6585788130760193 + }, + { + "bits_per_byte": 1.7004705492040808, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.1786763668060303, + "step": 3150, + "unclipped_grad_norm": 0.6470773220062256 + }, + { + "bits_per_byte": 1.7027161266763964, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.1802328824996948, + "step": 3160, + "unclipped_grad_norm": 0.6105231642723083 + }, + { + "bits_per_byte": 1.7746963695073095, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.2301257848739624, + "step": 3170, + "unclipped_grad_norm": 0.7074170708656311 + }, + { + "bits_per_byte": 1.7592339253161868, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.2194080352783203, + "step": 3180, + "unclipped_grad_norm": 0.7336324453353882 + }, + { + "bits_per_byte": 1.7384247124923906, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.204984188079834, + "step": 3190, + "unclipped_grad_norm": 0.6899617314338684 + }, + { + "bits_per_byte": 1.7108154776361295, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.1858469247817993, + "step": 3200, + "unclipped_grad_norm": 0.6462433338165283 + }, + { + "bits_per_byte": 1.6925779213906846, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.1732056140899658, + "step": 3210, + "unclipped_grad_norm": 0.7260644435882568 + }, + { + "bits_per_byte": 1.666516530392189, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.1551412343978882, + "step": 3220, + "unclipped_grad_norm": 0.6502906084060669 + }, + { + "bits_per_byte": 1.7285224674036626, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.1981204748153687, + "step": 3230, + "unclipped_grad_norm": 0.6560307741165161 + }, + { + "bits_per_byte": 1.7414053438144939, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.2070502042770386, + "step": 3240, + "unclipped_grad_norm": 0.7204704880714417 + }, + { + "bits_per_byte": 1.722378215218938, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.1938616037368774, + "step": 3250, + "unclipped_grad_norm": 0.6699979901313782 + }, + { + "bits_per_byte": 1.710248278853584, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.1854537725448608, + "step": 3260, + "unclipped_grad_norm": 0.6620029807090759 + }, + { + "bits_per_byte": 1.7492042410831696, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.2124559879302979, + "step": 3270, + "unclipped_grad_norm": 0.6819154620170593 + }, + { + "bits_per_byte": 1.7582780457426914, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.218745470046997, + "step": 3280, + "unclipped_grad_norm": 0.7104907631874084 + }, + { + "bits_per_byte": 1.7038568875995694, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.1810235977172852, + "step": 3290, + "unclipped_grad_norm": 0.7336503267288208 + }, + { + "bits_per_byte": 1.7143715629080198, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.1883118152618408, + "step": 3300, + "unclipped_grad_norm": 0.6713031530380249 + }, + { + "bits_per_byte": 1.6971223909570836, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.1763556003570557, + "step": 3310, + "unclipped_grad_norm": 0.6613041162490845 + }, + { + "bits_per_byte": 1.6966908864860688, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.1760565042495728, + "step": 3320, + "unclipped_grad_norm": 0.6607292294502258 + }, + { + "bits_per_byte": 1.709701546006491, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.185074806213379, + "step": 3330, + "unclipped_grad_norm": 0.6867441534996033 + }, + { + "bits_per_byte": 1.6499943291064498, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.1436889171600342, + "step": 3340, + "unclipped_grad_norm": 0.7373800873756409 + }, + { + "bits_per_byte": 1.6846101371588618, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.1676827669143677, + "step": 3350, + "unclipped_grad_norm": 0.6591839790344238 + }, + { + "bits_per_byte": 1.7024478337410505, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.1800469160079956, + "step": 3360, + "unclipped_grad_norm": 0.6913484930992126 + }, + { + "bits_per_byte": 1.6528703949768286, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.145682454109192, + "step": 3370, + "unclipped_grad_norm": 0.6433871388435364 + }, + { + "bits_per_byte": 1.6860035405961522, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.168648600578308, + "step": 3380, + "unclipped_grad_norm": 0.6708442568778992 + }, + { + "bits_per_byte": 1.7146281610231071, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.1884896755218506, + "step": 3390, + "unclipped_grad_norm": 0.6880245208740234 + }, + { + "bits_per_byte": 1.6922919142422999, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.1730073690414429, + "step": 3400, + "unclipped_grad_norm": 0.6594834327697754 + }, + { + "bits_per_byte": 1.681003145022316, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.1651825904846191, + "step": 3410, + "unclipped_grad_norm": 0.635221540927887 + }, + { + "bits_per_byte": 1.6628152917629706, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.1525757312774658, + "step": 3420, + "unclipped_grad_norm": 0.671238124370575 + }, + { + "bits_per_byte": 1.7036319342922408, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1808676719665527, + "step": 3430, + "unclipped_grad_norm": 0.635124683380127 + }, + { + "bits_per_byte": 1.7127858828670648, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.1872127056121826, + "step": 3440, + "unclipped_grad_norm": 0.6800798177719116 + }, + { + "bits_per_byte": 1.781907774040638, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.2351243495941162, + "step": 3450, + "unclipped_grad_norm": 0.7077510952949524 + }, + { + "bits_per_byte": 1.7745056407475026, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.2299935817718506, + "step": 3460, + "unclipped_grad_norm": 0.6950393915176392 + }, + { + "bits_per_byte": 1.7279900091165914, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.1977514028549194, + "step": 3470, + "unclipped_grad_norm": 0.7210932970046997 + }, + { + "bits_per_byte": 1.7044161751801752, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.1814112663269043, + "step": 3480, + "unclipped_grad_norm": 0.6897231340408325 + }, + { + "bits_per_byte": 1.6279048774296307, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.1283776760101318, + "step": 3490, + "unclipped_grad_norm": 0.6812744736671448 + }, + { + "bits_per_byte": 1.6515733018240213, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1447833776474, + "step": 3500, + "unclipped_grad_norm": 0.665158748626709 + }, + { + "bits_per_byte": 1.6923600193720416, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.173054575920105, + "step": 3510, + "unclipped_grad_norm": 0.73247230052948 + }, + { + "bits_per_byte": 1.6625494065847366, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.1523914337158203, + "step": 3520, + "unclipped_grad_norm": 0.6351336240768433 + }, + { + "bits_per_byte": 1.6642508309497224, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.1535707712173462, + "step": 3530, + "unclipped_grad_norm": 0.6755833029747009 + }, + { + "bits_per_byte": 1.6822169985721056, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.1660239696502686, + "step": 3540, + "unclipped_grad_norm": 0.6904852986335754 + }, + { + "bits_per_byte": 1.7293584750695068, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.198699951171875, + "step": 3550, + "unclipped_grad_norm": 0.6900108456611633 + }, + { + "bits_per_byte": 1.666494000664926, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.155125617980957, + "step": 3560, + "unclipped_grad_norm": 0.6760779619216919 + }, + { + "bits_per_byte": 1.7160418584131993, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.189469575881958, + "step": 3570, + "unclipped_grad_norm": 0.7369740605354309 + }, + { + "bits_per_byte": 1.76083714758753, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.2205193042755127, + "step": 3580, + "unclipped_grad_norm": 0.7343559861183167 + }, + { + "bits_per_byte": 1.7074802180879471, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.183535099029541, + "step": 3590, + "unclipped_grad_norm": 0.6606658101081848 + }, + { + "bits_per_byte": 1.6828887628063758, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.166489601135254, + "step": 3600, + "unclipped_grad_norm": 0.6863994598388672 + }, + { + "bits_per_byte": 1.672810923431117, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.1595041751861572, + "step": 3610, + "unclipped_grad_norm": 0.6441814303398132 + }, + { + "bits_per_byte": 1.6656375270636292, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.1545319557189941, + "step": 3620, + "unclipped_grad_norm": 0.6579524874687195 + }, + { + "bits_per_byte": 1.681767779888052, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.165712594985962, + "step": 3630, + "unclipped_grad_norm": 0.674933135509491 + }, + { + "bits_per_byte": 1.6501129971355453, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.1437711715698242, + "step": 3640, + "unclipped_grad_norm": 0.6882205009460449 + }, + { + "bits_per_byte": 1.708073214268122, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.183946132659912, + "step": 3650, + "unclipped_grad_norm": 0.6684221029281616 + }, + { + "bits_per_byte": 1.6236231973537503, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.1254098415374756, + "step": 3660, + "unclipped_grad_norm": 0.6868197321891785 + }, + { + "bits_per_byte": 1.6306316623590604, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.1302677392959595, + "step": 3670, + "unclipped_grad_norm": 0.6331377029418945 + }, + { + "bits_per_byte": 1.7072803742476446, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.183396577835083, + "step": 3680, + "unclipped_grad_norm": 0.7000526785850525 + }, + { + "bits_per_byte": 1.6512415472905069, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.1445534229278564, + "step": 3690, + "unclipped_grad_norm": 0.6353453993797302 + }, + { + "bits_per_byte": 1.6105368654669483, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.116339087486267, + "step": 3700, + "unclipped_grad_norm": 0.6407413482666016 + }, + { + "bits_per_byte": 1.6698801670777639, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.1574727296829224, + "step": 3710, + "unclipped_grad_norm": 0.6686496734619141 + }, + { + "bits_per_byte": 1.6712761502548172, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.158440351486206, + "step": 3720, + "unclipped_grad_norm": 0.7011558413505554 + }, + { + "bits_per_byte": 1.7032078250752132, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.1805737018585205, + "step": 3730, + "unclipped_grad_norm": 0.6945915818214417 + }, + { + "bits_per_byte": 1.6424568454670856, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.138464331626892, + "step": 3740, + "unclipped_grad_norm": 0.6610867381095886 + }, + { + "bits_per_byte": 1.6320493153501228, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1312503814697266, + "step": 3750, + "unclipped_grad_norm": 0.676323413848877 + }, + { + "bits_per_byte": 1.6283203875141152, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.1286656856536865, + "step": 3760, + "unclipped_grad_norm": 0.668434739112854 + }, + { + "bits_per_byte": 1.6832143259644592, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.1667152643203735, + "step": 3770, + "unclipped_grad_norm": 0.6737239956855774 + }, + { + "bits_per_byte": 1.6297148228473108, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.1296322345733643, + "step": 3780, + "unclipped_grad_norm": 0.6598157286643982 + }, + { + "bits_per_byte": 1.6801834757083038, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.1646144390106201, + "step": 3790, + "unclipped_grad_norm": 0.6808153390884399 + }, + { + "bits_per_byte": 1.6228561547309022, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.124878168106079, + "step": 3800, + "unclipped_grad_norm": 0.6611790657043457 + }, + { + "bits_per_byte": 1.6072483851998005, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.1140596866607666, + "step": 3810, + "unclipped_grad_norm": 0.6496701240539551 + }, + { + "bits_per_byte": 1.7155450005348567, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1891251802444458, + "step": 3820, + "unclipped_grad_norm": 0.6694932579994202 + }, + { + "bits_per_byte": 1.6360461921561744, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.1340208053588867, + "step": 3830, + "unclipped_grad_norm": 0.6810566782951355 + }, + { + "bits_per_byte": 1.637058310056502, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.134722352027893, + "step": 3840, + "unclipped_grad_norm": 0.6738945841789246 + }, + { + "bits_per_byte": 1.6676970193077114, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1559594869613647, + "step": 3850, + "unclipped_grad_norm": 0.679168701171875 + }, + { + "bits_per_byte": 1.6166854172179446, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.120600938796997, + "step": 3860, + "unclipped_grad_norm": 0.6719157695770264 + }, + { + "bits_per_byte": 1.675943759399234, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.1616756916046143, + "step": 3870, + "unclipped_grad_norm": 0.7009814977645874 + }, + { + "bits_per_byte": 1.6451007347587996, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1402969360351562, + "step": 3880, + "unclipped_grad_norm": 0.7374706268310547 + }, + { + "bits_per_byte": 1.6731898012109676, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.1597667932510376, + "step": 3890, + "unclipped_grad_norm": 0.6934090256690979 + }, + { + "bits_per_byte": 1.6662349947927266, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.1549460887908936, + "step": 3900, + "unclipped_grad_norm": 0.7259919047355652 + }, + { + "bits_per_byte": 1.6866646419060687, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1691068410873413, + "step": 3910, + "unclipped_grad_norm": 0.6637568473815918 + }, + { + "bits_per_byte": 1.6381385330865712, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.1354711055755615, + "step": 3920, + "unclipped_grad_norm": 0.6496707201004028 + }, + { + "bits_per_byte": 1.6210266032910234, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.123610019683838, + "step": 3930, + "unclipped_grad_norm": 0.6914623975753784 + }, + { + "bits_per_byte": 1.626813819492557, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.1276214122772217, + "step": 3940, + "unclipped_grad_norm": 0.6604109406471252 + }, + { + "bits_per_byte": 1.7024247880658347, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1800309419631958, + "step": 3950, + "unclipped_grad_norm": 0.7089223861694336 + }, + { + "bits_per_byte": 1.7181938773234458, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.190961241722107, + "step": 3960, + "unclipped_grad_norm": 0.7124776244163513 + }, + { + "bits_per_byte": 1.6756278272695988, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1614567041397095, + "step": 3970, + "unclipped_grad_norm": 0.6807390451431274 + }, + { + "bits_per_byte": 1.6658442502099664, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.1546752452850342, + "step": 3980, + "unclipped_grad_norm": 0.6813681125640869 + }, + { + "bits_per_byte": 1.7078774980114402, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1838104724884033, + "step": 3990, + "unclipped_grad_norm": 0.6958358883857727 + }, + { + "bits_per_byte": 1.6926191972268918, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.173234224319458, + "step": 4000, + "unclipped_grad_norm": 0.6829394698143005 + }, + { + "bits_per_byte": 1.7165567744698826, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.189826488494873, + "step": 4010, + "unclipped_grad_norm": 0.7041646838188171 + }, + { + "bits_per_byte": 1.5945082543891853, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.1052289009094238, + "step": 4020, + "unclipped_grad_norm": 0.6747278571128845 + }, + { + "bits_per_byte": 1.6334055705348267, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.132190465927124, + "step": 4030, + "unclipped_grad_norm": 0.6892349720001221 + }, + { + "bits_per_byte": 1.6223340154028827, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.124516248703003, + "step": 4040, + "unclipped_grad_norm": 0.6960596442222595 + }, + { + "bits_per_byte": 1.6336955332841814, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.1323914527893066, + "step": 4050, + "unclipped_grad_norm": 0.7008145451545715 + }, + { + "bits_per_byte": 1.6438612557940313, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1394377946853638, + "step": 4060, + "unclipped_grad_norm": 0.6903786063194275 + }, + { + "bits_per_byte": 1.6182131391055588, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.1216598749160767, + "step": 4070, + "unclipped_grad_norm": 0.6921495199203491 + }, + { + "bits_per_byte": 1.6396062330290344, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.136488437652588, + "step": 4080, + "unclipped_grad_norm": 0.6660868525505066 + }, + { + "bits_per_byte": 1.599222298849335, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.1084964275360107, + "step": 4090, + "unclipped_grad_norm": 0.6874752044677734 + }, + { + "bits_per_byte": 1.5682403122270034, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.0870213508605957, + "step": 4100, + "unclipped_grad_norm": 0.6370568871498108 + }, + { + "bits_per_byte": 1.6773290796519338, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.1626359224319458, + "step": 4110, + "unclipped_grad_norm": 0.698296070098877 + }, + { + "bits_per_byte": 1.629915182635566, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.129771113395691, + "step": 4120, + "unclipped_grad_norm": 0.6765598058700562 + }, + { + "bits_per_byte": 1.5933643977782967, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.1044360399246216, + "step": 4130, + "unclipped_grad_norm": 0.6698799729347229 + }, + { + "bits_per_byte": 1.599178099308063, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.1084657907485962, + "step": 4140, + "unclipped_grad_norm": 0.668297290802002 + }, + { + "bits_per_byte": 1.6625531902030557, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1523940563201904, + "step": 4150, + "unclipped_grad_norm": 0.6791092157363892 + }, + { + "bits_per_byte": 1.6405001988482193, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.1371080875396729, + "step": 4160, + "unclipped_grad_norm": 0.7006524801254272 + }, + { + "bits_per_byte": 1.5892620956072663, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.1015925407409668, + "step": 4170, + "unclipped_grad_norm": 0.6852906346321106 + }, + { + "bits_per_byte": 1.5676975349808804, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.0866451263427734, + "step": 4180, + "unclipped_grad_norm": 0.670124351978302 + }, + { + "bits_per_byte": 1.7105721221851586, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.185678243637085, + "step": 4190, + "unclipped_grad_norm": 0.6962432265281677 + }, + { + "bits_per_byte": 1.6040746173607783, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1118597984313965, + "step": 4200, + "unclipped_grad_norm": 0.6914252638816833 + }, + { + "bits_per_byte": 1.6136564587709488, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.1185014247894287, + "step": 4210, + "unclipped_grad_norm": 0.6895549893379211 + }, + { + "bits_per_byte": 1.6649409973276348, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.1540491580963135, + "step": 4220, + "unclipped_grad_norm": 0.683404803276062 + }, + { + "bits_per_byte": 1.6312709218723176, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.1307108402252197, + "step": 4230, + "unclipped_grad_norm": 0.669493556022644 + }, + { + "bits_per_byte": 1.6767269683912631, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.1622185707092285, + "step": 4240, + "unclipped_grad_norm": 0.6939648389816284 + }, + { + "bits_per_byte": 1.7002624501965367, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1785321235656738, + "step": 4250, + "unclipped_grad_norm": 0.6851627230644226 + }, + { + "bits_per_byte": 1.6856095283430255, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1683754920959473, + "step": 4260, + "unclipped_grad_norm": 0.7326967120170593 + }, + { + "bits_per_byte": 1.6446709501142933, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.1399990320205688, + "step": 4270, + "unclipped_grad_norm": 0.6812265515327454 + }, + { + "bits_per_byte": 1.684671190999918, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1677250862121582, + "step": 4280, + "unclipped_grad_norm": 0.7294073104858398 + }, + { + "bits_per_byte": 1.6301456393877223, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.129930853843689, + "step": 4290, + "unclipped_grad_norm": 0.6867120862007141 + }, + { + "bits_per_byte": 1.5698593569022259, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.0881435871124268, + "step": 4300, + "unclipped_grad_norm": 0.6935779452323914 + }, + { + "bits_per_byte": 1.698711510651056, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.1774570941925049, + "step": 4310, + "unclipped_grad_norm": 0.7877510190010071 + }, + { + "bits_per_byte": 1.5795127430951554, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.094834804534912, + "step": 4320, + "unclipped_grad_norm": 0.6770227551460266 + }, + { + "bits_per_byte": 1.5926814546717205, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.1039626598358154, + "step": 4330, + "unclipped_grad_norm": 0.6699842214584351 + }, + { + "bits_per_byte": 1.594952657559015, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.105536937713623, + "step": 4340, + "unclipped_grad_norm": 0.6909552812576294 + }, + { + "bits_per_byte": 1.516730648380338, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.051317572593689, + "step": 4350, + "unclipped_grad_norm": 0.6726857423782349 + }, + { + "bits_per_byte": 1.6686382803558835, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.1566119194030762, + "step": 4360, + "unclipped_grad_norm": 0.721929669380188 + }, + { + "bits_per_byte": 1.6220110319845622, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.1242923736572266, + "step": 4370, + "unclipped_grad_norm": 0.7167156338691711 + }, + { + "bits_per_byte": 1.583184916656378, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.0973801612854004, + "step": 4380, + "unclipped_grad_norm": 0.7065753936767578 + }, + { + "bits_per_byte": 1.6116569884720178, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1171154975891113, + "step": 4390, + "unclipped_grad_norm": 0.7058514356613159 + }, + { + "bits_per_byte": 1.5752468854231543, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.0918779373168945, + "step": 4400, + "unclipped_grad_norm": 0.7191004753112793 + }, + { + "bits_per_byte": 1.5840026941612306, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.0979470014572144, + "step": 4410, + "unclipped_grad_norm": 0.7205251455307007 + }, + { + "bits_per_byte": 1.5742160214138825, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.0911633968353271, + "step": 4420, + "unclipped_grad_norm": 0.6976714730262756 + }, + { + "bits_per_byte": 1.590350745787228, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.1023471355438232, + "step": 4430, + "unclipped_grad_norm": 0.6934608221054077 + }, + { + "bits_per_byte": 1.601144204972727, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.1098285913467407, + "step": 4440, + "unclipped_grad_norm": 0.7232753038406372 + }, + { + "bits_per_byte": 1.5745089078683019, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.0913664102554321, + "step": 4450, + "unclipped_grad_norm": 0.6953805685043335 + }, + { + "bits_per_byte": 1.54706890394049, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.0723464488983154, + "step": 4460, + "unclipped_grad_norm": 0.6721370816230774 + }, + { + "bits_per_byte": 1.6580974636845018, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1493055820465088, + "step": 4470, + "unclipped_grad_norm": 0.7164402008056641 + }, + { + "bits_per_byte": 1.5745249022548322, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.0913774967193604, + "step": 4480, + "unclipped_grad_norm": 0.7170845866203308 + }, + { + "bits_per_byte": 1.510441414820936, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.0469582080841064, + "step": 4490, + "unclipped_grad_norm": 0.6605475544929504 + }, + { + "bits_per_byte": 1.6096272492265349, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.115708589553833, + "step": 4500, + "unclipped_grad_norm": 0.727374255657196 + }, + { + "bits_per_byte": 1.5895809514419659, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.101813554763794, + "step": 4510, + "unclipped_grad_norm": 0.6892575025558472 + }, + { + "bits_per_byte": 1.5823504568343916, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.0968017578125, + "step": 4520, + "unclipped_grad_norm": 0.6960499286651611 + }, + { + "bits_per_byte": 1.5984029735006244, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.1079285144805908, + "step": 4530, + "unclipped_grad_norm": 0.7077929973602295 + }, + { + "bits_per_byte": 1.5489815230007353, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.0736721754074097, + "step": 4540, + "unclipped_grad_norm": 0.6761224865913391 + }, + { + "bits_per_byte": 1.6196634688002849, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.1226651668548584, + "step": 4550, + "unclipped_grad_norm": 0.6921520829200745 + }, + { + "bits_per_byte": 1.6215786676002932, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.123992681503296, + "step": 4560, + "unclipped_grad_norm": 0.7218050956726074 + }, + { + "bits_per_byte": 1.5759939780585022, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.0923957824707031, + "step": 4570, + "unclipped_grad_norm": 0.7079866528511047 + }, + { + "bits_per_byte": 1.5385423480760136, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.0664362907409668, + "step": 4580, + "unclipped_grad_norm": 0.6904507279396057 + }, + { + "bits_per_byte": 1.641131203194235, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.1375454664230347, + "step": 4590, + "unclipped_grad_norm": 0.7530466318130493 + }, + { + "bits_per_byte": 1.6115324730327931, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.1170291900634766, + "step": 4600, + "unclipped_grad_norm": 0.6856536269187927 + }, + { + "bits_per_byte": 1.5943348958771157, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.1051087379455566, + "step": 4610, + "unclipped_grad_norm": 0.7065469622612 + }, + { + "bits_per_byte": 1.5315935610505527, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.061619758605957, + "step": 4620, + "unclipped_grad_norm": 0.6934234499931335 + }, + { + "bits_per_byte": 1.6085114257877369, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.1149351596832275, + "step": 4630, + "unclipped_grad_norm": 0.7632301449775696 + }, + { + "bits_per_byte": 1.585077757711774, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.0986921787261963, + "step": 4640, + "unclipped_grad_norm": 0.7001867294311523 + }, + { + "bits_per_byte": 1.5438934162749591, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.0701453685760498, + "step": 4650, + "unclipped_grad_norm": 0.7094289064407349 + }, + { + "bits_per_byte": 1.5896282466709533, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.1018463373184204, + "step": 4660, + "unclipped_grad_norm": 0.7139912843704224 + }, + { + "bits_per_byte": 1.6060599850823385, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.1132359504699707, + "step": 4670, + "unclipped_grad_norm": 0.7127988934516907 + }, + { + "bits_per_byte": 1.5783554718374995, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.094032645225525, + "step": 4680, + "unclipped_grad_norm": 0.7106269001960754 + }, + { + "bits_per_byte": 1.547733100938122, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.0728068351745605, + "step": 4690, + "unclipped_grad_norm": 0.7213415503501892 + }, + { + "bits_per_byte": 1.655831076311432, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.1477346420288086, + "step": 4700, + "unclipped_grad_norm": 0.751556396484375 + }, + { + "bits_per_byte": 1.5704428940366033, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.0885480642318726, + "step": 4710, + "unclipped_grad_norm": 0.7309710383415222 + }, + { + "bits_per_byte": 1.52847155998944, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.0594557523727417, + "step": 4720, + "unclipped_grad_norm": 0.7122988104820251 + }, + { + "bits_per_byte": 1.6560367675618637, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.1478772163391113, + "step": 4730, + "unclipped_grad_norm": 0.7403090596199036 + }, + { + "bits_per_byte": 1.6088426643732987, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.1151647567749023, + "step": 4740, + "unclipped_grad_norm": 0.7658607363700867 + }, + { + "bits_per_byte": 1.6419087367587857, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.1380844116210938, + "step": 4750, + "unclipped_grad_norm": 0.7641808390617371 + }, + { + "bits_per_byte": 1.5047314188296397, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.043000340461731, + "step": 4760, + "unclipped_grad_norm": 0.6704828143119812 + }, + { + "bits_per_byte": 1.6192441750974813, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.1223745346069336, + "step": 4770, + "unclipped_grad_norm": 0.7162955403327942 + }, + { + "bits_per_byte": 1.5679252400106227, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.0868029594421387, + "step": 4780, + "unclipped_grad_norm": 0.6884462833404541 + }, + { + "bits_per_byte": 1.5919591275380969, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.1034619808197021, + "step": 4790, + "unclipped_grad_norm": 0.719971776008606 + }, + { + "bits_per_byte": 1.564771938107053, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.084617257118225, + "step": 4800, + "unclipped_grad_norm": 0.6843602657318115 + }, + { + "bits_per_byte": 1.5458985620013685, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.0715352296829224, + "step": 4810, + "unclipped_grad_norm": 0.726342499256134 + }, + { + "bits_per_byte": 1.6161643097858303, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.1202397346496582, + "step": 4820, + "unclipped_grad_norm": 0.7903850078582764 + }, + { + "bits_per_byte": 1.6060498381059376, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.1132289171218872, + "step": 4830, + "unclipped_grad_norm": 0.7433529496192932 + }, + { + "bits_per_byte": 1.5974013465419996, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.107234239578247, + "step": 4840, + "unclipped_grad_norm": 0.7061898112297058 + }, + { + "bits_per_byte": 1.5356375611029407, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.064422845840454, + "step": 4850, + "unclipped_grad_norm": 0.7154237031936646 + }, + { + "bits_per_byte": 1.5590610822025026, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.0806587934494019, + "step": 4860, + "unclipped_grad_norm": 0.7582815885543823 + }, + { + "bits_per_byte": 1.5518556970619546, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.0756644010543823, + "step": 4870, + "unclipped_grad_norm": 0.7426844239234924 + }, + { + "bits_per_byte": 1.7203285259859549, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.1924408674240112, + "step": 4880, + "unclipped_grad_norm": 0.7642485499382019 + }, + { + "bits_per_byte": 1.4999091972820975, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.0396578311920166, + "step": 4890, + "unclipped_grad_norm": 0.695369303226471 + }, + { + "bits_per_byte": 1.600197440479727, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.1091723442077637, + "step": 4900, + "unclipped_grad_norm": 0.7042683959007263 + }, + { + "bits_per_byte": 1.5906699455872293, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.1025683879852295, + "step": 4910, + "unclipped_grad_norm": 0.7306523323059082 + }, + { + "bits_per_byte": 1.5111625380760036, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.0474580526351929, + "step": 4920, + "unclipped_grad_norm": 0.689345121383667 + }, + { + "bits_per_byte": 1.543948450723235, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.070183515548706, + "step": 4930, + "unclipped_grad_norm": 0.7031807899475098 + }, + { + "bits_per_byte": 1.560199091403262, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.0814476013183594, + "step": 4940, + "unclipped_grad_norm": 0.7517480254173279 + }, + { + "bits_per_byte": 1.584777132038066, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.0984838008880615, + "step": 4950, + "unclipped_grad_norm": 0.7657203674316406 + }, + { + "bits_per_byte": 1.5972649642998653, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.1071397066116333, + "step": 4960, + "unclipped_grad_norm": 0.7337971925735474 + }, + { + "bits_per_byte": 1.5564491816838493, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.0788483619689941, + "step": 4970, + "unclipped_grad_norm": 0.7312259078025818 + }, + { + "bits_per_byte": 1.6103607552324648, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.116217017173767, + "step": 4980, + "unclipped_grad_norm": 0.7422297596931458 + }, + { + "bits_per_byte": 1.5729031058571958, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.0902533531188965, + "step": 4990, + "unclipped_grad_norm": 0.7554135918617249 + }, + { + "bits_per_byte": 1.5715537299785263, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.089318037033081, + "step": 5000, + "unclipped_grad_norm": 0.7184606790542603 + }, + { + "bits_per_byte": 1.624125042728968, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.1257576942443848, + "step": 5010, + "unclipped_grad_norm": 0.7311696410179138 + }, + { + "bits_per_byte": 1.5834690319956033, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.0975770950317383, + "step": 5020, + "unclipped_grad_norm": 0.776550829410553 + }, + { + "bits_per_byte": 1.5160738466366928, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.0508623123168945, + "step": 5030, + "unclipped_grad_norm": 0.696950376033783 + }, + { + "bits_per_byte": 1.5757399596831778, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.0922197103500366, + "step": 5040, + "unclipped_grad_norm": 0.6912526488304138 + }, + { + "bits_per_byte": 1.5527629055452559, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.0762932300567627, + "step": 5050, + "unclipped_grad_norm": 0.733496367931366 + }, + { + "bits_per_byte": 1.6209263374055705, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.1235405206680298, + "step": 5060, + "unclipped_grad_norm": 0.7645672559738159 + }, + { + "bits_per_byte": 1.5676156712390696, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.0865883827209473, + "step": 5070, + "unclipped_grad_norm": 0.7433090806007385 + }, + { + "bits_per_byte": 1.617925928078618, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.1214607954025269, + "step": 5080, + "unclipped_grad_norm": 0.7580333352088928 + }, + { + "bits_per_byte": 1.561349999302836, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.0822453498840332, + "step": 5090, + "unclipped_grad_norm": 0.7248643636703491 + }, + { + "bits_per_byte": 1.5453986084353104, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.0711886882781982, + "step": 5100, + "unclipped_grad_norm": 0.7515865564346313 + }, + { + "bits_per_byte": 1.5794236560820083, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.0947730541229248, + "step": 5110, + "unclipped_grad_norm": 0.7437917590141296 + }, + { + "bits_per_byte": 1.5020624200708999, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0411503314971924, + "step": 5120, + "unclipped_grad_norm": 0.7223066687583923 + }, + { + "bits_per_byte": 1.6246576729986901, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.1261268854141235, + "step": 5130, + "unclipped_grad_norm": 0.7789642214775085 + }, + { + "bits_per_byte": 1.5969373373499716, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.106912612915039, + "step": 5140, + "unclipped_grad_norm": 0.7606791853904724 + }, + { + "bits_per_byte": 1.5321538805270638, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.0620081424713135, + "step": 5150, + "unclipped_grad_norm": 0.7157384753227234 + }, + { + "bits_per_byte": 1.555317535841172, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.07806396484375, + "step": 5160, + "unclipped_grad_norm": 0.7261564135551453 + }, + { + "bits_per_byte": 1.6137937009263372, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.1185965538024902, + "step": 5170, + "unclipped_grad_norm": 0.7376936674118042 + }, + { + "bits_per_byte": 1.575485597342552, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.092043399810791, + "step": 5180, + "unclipped_grad_norm": 0.7733915448188782 + }, + { + "bits_per_byte": 1.473890114015714, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.0216227769851685, + "step": 5190, + "unclipped_grad_norm": 0.7125696539878845 + }, + { + "bits_per_byte": 1.561634458607363, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.0824425220489502, + "step": 5200, + "unclipped_grad_norm": 0.73981773853302 + }, + { + "bits_per_byte": 1.5148080543263425, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.0499849319458008, + "step": 5210, + "unclipped_grad_norm": 0.7270516157150269 + }, + { + "bits_per_byte": 1.5298503449014074, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.0604114532470703, + "step": 5220, + "unclipped_grad_norm": 0.7458752989768982 + }, + { + "bits_per_byte": 1.6147979076247252, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.1192926168441772, + "step": 5230, + "unclipped_grad_norm": 0.7549944519996643 + }, + { + "bits_per_byte": 1.5781487486911625, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0938893556594849, + "step": 5240, + "unclipped_grad_norm": 0.833949089050293 + }, + { + "bits_per_byte": 1.5764992630867372, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0927460193634033, + "step": 5250, + "unclipped_grad_norm": 0.7506707906723022 + }, + { + "bits_per_byte": 1.5091309070213612, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.0460498332977295, + "step": 5260, + "unclipped_grad_norm": 0.7540333271026611 + }, + { + "bits_per_byte": 1.5509861527791922, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.0750616788864136, + "step": 5270, + "unclipped_grad_norm": 0.7425879240036011 + }, + { + "bits_per_byte": 1.4986365256657126, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.0387756824493408, + "step": 5280, + "unclipped_grad_norm": 0.7508636713027954 + }, + { + "bits_per_byte": 1.512912117583231, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.048670768737793, + "step": 5290, + "unclipped_grad_norm": 0.7426420450210571 + }, + { + "bits_per_byte": 1.5053122042416034, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.043402910232544, + "step": 5300, + "unclipped_grad_norm": 0.7597728967666626 + }, + { + "bits_per_byte": 1.494929267643714, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.0362060070037842, + "step": 5310, + "unclipped_grad_norm": 0.7301760911941528 + }, + { + "bits_per_byte": 1.5199307295649422, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0535356998443604, + "step": 5320, + "unclipped_grad_norm": 0.7844780087471008 + }, + { + "bits_per_byte": 1.5587018104448502, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.0804097652435303, + "step": 5330, + "unclipped_grad_norm": 0.7579611539840698 + }, + { + "bits_per_byte": 1.6127731558761174, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.117889165878296, + "step": 5340, + "unclipped_grad_norm": 0.7894383668899536 + }, + { + "bits_per_byte": 1.4851792272009996, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.0294477939605713, + "step": 5350, + "unclipped_grad_norm": 0.7475207448005676 + }, + { + "bits_per_byte": 1.5184396399819617, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.052502155303955, + "step": 5360, + "unclipped_grad_norm": 0.7459679245948792 + }, + { + "bits_per_byte": 1.594675421525824, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.1053447723388672, + "step": 5370, + "unclipped_grad_norm": 0.7826220989227295 + }, + { + "bits_per_byte": 1.5164852291375568, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.0511474609375, + "step": 5380, + "unclipped_grad_norm": 0.7348969578742981 + }, + { + "bits_per_byte": 1.5035959893686435, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.0422133207321167, + "step": 5390, + "unclipped_grad_norm": 0.7585330009460449 + }, + { + "bits_per_byte": 1.5553901125198362, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.0781142711639404, + "step": 5400, + "unclipped_grad_norm": 0.7648258805274963 + }, + { + "bits_per_byte": 1.450857681464214, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.0056579113006592, + "step": 5410, + "unclipped_grad_norm": 0.7363677620887756 + }, + { + "bits_per_byte": 1.6100176498439935, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.1159791946411133, + "step": 5420, + "unclipped_grad_norm": 0.8077604174613953 + }, + { + "bits_per_byte": 1.5422606129876675, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.0690135955810547, + "step": 5430, + "unclipped_grad_norm": 0.7490110993385315 + }, + { + "bits_per_byte": 1.5428438061567433, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.0694178342819214, + "step": 5440, + "unclipped_grad_norm": 0.7584846615791321 + }, + { + "bits_per_byte": 1.5455248437010436, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.0712761878967285, + "step": 5450, + "unclipped_grad_norm": 0.7466675043106079 + }, + { + "bits_per_byte": 1.5805709523459142, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.095568299293518, + "step": 5460, + "unclipped_grad_norm": 0.7483034729957581 + }, + { + "bits_per_byte": 1.53896817711955, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0667314529418945, + "step": 5470, + "unclipped_grad_norm": 0.7806885838508606 + }, + { + "bits_per_byte": 1.4896041688250488, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.0325149297714233, + "step": 5480, + "unclipped_grad_norm": 0.7554174065589905 + }, + { + "bits_per_byte": 1.542476967162453, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.0691635608673096, + "step": 5490, + "unclipped_grad_norm": 0.7489543557167053 + }, + { + "bits_per_byte": 1.5705183944203323, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.0886003971099854, + "step": 5500, + "unclipped_grad_norm": 0.7442025542259216 + }, + { + "bits_per_byte": 1.5572850173670425, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.079427719116211, + "step": 5510, + "unclipped_grad_norm": 0.7655790448188782 + }, + { + "bits_per_byte": 1.5436130845540528, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.069951057434082, + "step": 5520, + "unclipped_grad_norm": 0.7647954225540161 + }, + { + "bits_per_byte": 1.56602311189208, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.085484504699707, + "step": 5530, + "unclipped_grad_norm": 0.8068528771400452 + }, + { + "bits_per_byte": 1.5970931536316533, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.107020616531372, + "step": 5540, + "unclipped_grad_norm": 0.8010297417640686 + }, + { + "bits_per_byte": 1.522473149092647, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.0552979707717896, + "step": 5550, + "unclipped_grad_norm": 0.7579601407051086 + }, + { + "bits_per_byte": 1.5147924039051142, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.0499740839004517, + "step": 5560, + "unclipped_grad_norm": 0.7632036805152893 + }, + { + "bits_per_byte": 1.5799624777271615, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.0951465368270874, + "step": 5570, + "unclipped_grad_norm": 0.7855972051620483 + }, + { + "bits_per_byte": 1.4707880629421015, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.019472599029541, + "step": 5580, + "unclipped_grad_norm": 0.7356786131858826 + }, + { + "bits_per_byte": 1.5234341881456686, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.0559641122817993, + "step": 5590, + "unclipped_grad_norm": 0.7662619948387146 + }, + { + "bits_per_byte": 1.5279655870306017, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.0591050386428833, + "step": 5600, + "unclipped_grad_norm": 0.7386499643325806 + }, + { + "bits_per_byte": 1.540744585920312, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.0679627656936646, + "step": 5610, + "unclipped_grad_norm": 0.7514070272445679 + }, + { + "bits_per_byte": 1.477657050017562, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.0242338180541992, + "step": 5620, + "unclipped_grad_norm": 0.7735412120819092 + }, + { + "bits_per_byte": 1.487297365529026, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0309159755706787, + "step": 5630, + "unclipped_grad_norm": 0.7478345036506653 + }, + { + "bits_per_byte": 1.475474590178113, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.0227210521697998, + "step": 5640, + "unclipped_grad_norm": 0.7715311646461487 + }, + { + "bits_per_byte": 1.5788741715125016, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.09439218044281, + "step": 5650, + "unclipped_grad_norm": 0.7889861464500427 + }, + { + "bits_per_byte": 1.5272317370593702, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.0585963726043701, + "step": 5660, + "unclipped_grad_norm": 0.7796398997306824 + }, + { + "bits_per_byte": 1.5325131522847162, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.062257170677185, + "step": 5670, + "unclipped_grad_norm": 0.799603283405304 + }, + { + "bits_per_byte": 1.5454839118301382, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0712478160858154, + "step": 5680, + "unclipped_grad_norm": 0.785685658454895 + }, + { + "bits_per_byte": 1.5748112533685188, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.0915759801864624, + "step": 5690, + "unclipped_grad_norm": 0.7953619956970215 + }, + { + "bits_per_byte": 1.50339992914666, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.0420774221420288, + "step": 5700, + "unclipped_grad_norm": 0.7669439315795898 + }, + { + "bits_per_byte": 1.543373168756099, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.0697847604751587, + "step": 5710, + "unclipped_grad_norm": 0.7705186009407043 + }, + { + "bits_per_byte": 1.567441280831095, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0864675045013428, + "step": 5720, + "unclipped_grad_norm": 0.8056720495223999 + }, + { + "bits_per_byte": 1.4733966957903886, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.0212807655334473, + "step": 5730, + "unclipped_grad_norm": 0.7471829056739807 + }, + { + "bits_per_byte": 1.554708029326514, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.077641487121582, + "step": 5740, + "unclipped_grad_norm": 0.7884363532066345 + }, + { + "bits_per_byte": 1.5014171411648625, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.0407030582427979, + "step": 5750, + "unclipped_grad_norm": 0.7782815098762512 + }, + { + "bits_per_byte": 1.4985412472771347, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.0387096405029297, + "step": 5760, + "unclipped_grad_norm": 0.82447350025177 + }, + { + "bits_per_byte": 1.4545344985570101, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.0082064867019653, + "step": 5770, + "unclipped_grad_norm": 0.7557410597801208 + }, + { + "bits_per_byte": 1.5468133377213078, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.072169303894043, + "step": 5780, + "unclipped_grad_norm": 0.7550814747810364 + }, + { + "bits_per_byte": 1.486215594655099, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 1.0301661491394043, + "step": 5790, + "unclipped_grad_norm": 0.7433685064315796 + }, + { + "bits_per_byte": 1.5176457680655788, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.0519518852233887, + "step": 5800, + "unclipped_grad_norm": 0.7983713746070862 + }, + { + "bits_per_byte": 1.5336184847818115, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.063023328781128, + "step": 5810, + "unclipped_grad_norm": 0.7758554220199585 + }, + { + "bits_per_byte": 1.5150988769889515, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.0501865148544312, + "step": 5820, + "unclipped_grad_norm": 0.7797936201095581 + }, + { + "bits_per_byte": 1.503488844177156, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.0421390533447266, + "step": 5830, + "unclipped_grad_norm": 0.8076127171516418 + }, + { + "bits_per_byte": 1.461105955646478, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.0127614736557007, + "step": 5840, + "unclipped_grad_norm": 0.7823375463485718 + }, + { + "bits_per_byte": 1.4646016750079154, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.0151845216751099, + "step": 5850, + "unclipped_grad_norm": 0.7706903219223022 + }, + { + "bits_per_byte": 1.491532094341221, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.0338512659072876, + "step": 5860, + "unclipped_grad_norm": 0.8551949262619019 + }, + { + "bits_per_byte": 1.54194416491008, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0687942504882812, + "step": 5870, + "unclipped_grad_norm": 0.8119983673095703 + }, + { + "bits_per_byte": 1.5542440201344863, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.077319860458374, + "step": 5880, + "unclipped_grad_norm": 0.7998167872428894 + }, + { + "bits_per_byte": 1.5364713329943236, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.0650007724761963, + "step": 5890, + "unclipped_grad_norm": 0.7873921394348145 + }, + { + "bits_per_byte": 1.5121715602886157, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.0481574535369873, + "step": 5900, + "unclipped_grad_norm": 0.8678092360496521 + }, + { + "bits_per_byte": 1.4927588465898254, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.0347015857696533, + "step": 5910, + "unclipped_grad_norm": 0.8054438233375549 + }, + { + "bits_per_byte": 1.5103413209181338, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.046888828277588, + "step": 5920, + "unclipped_grad_norm": 0.7993885278701782 + }, + { + "bits_per_byte": 1.5646998773763416, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.0845673084259033, + "step": 5930, + "unclipped_grad_norm": 0.8135746121406555 + }, + { + "bits_per_byte": 1.5440261868814251, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.070237398147583, + "step": 5940, + "unclipped_grad_norm": 0.8457834720611572 + }, + { + "bits_per_byte": 1.506306779911543, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.0440922975540161, + "step": 5950, + "unclipped_grad_norm": 0.8144971132278442 + }, + { + "bits_per_byte": 1.5151527075586717, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.0502238273620605, + "step": 5960, + "unclipped_grad_norm": 0.7949123978614807 + }, + { + "bits_per_byte": 1.5146529259752644, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.049877405166626, + "step": 5970, + "unclipped_grad_norm": 0.807485044002533 + }, + { + "bits_per_byte": 1.47750312554504, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.0241271257400513, + "step": 5980, + "unclipped_grad_norm": 0.8315603733062744 + }, + { + "bits_per_byte": 1.4492998626126987, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0045781135559082, + "step": 5990, + "unclipped_grad_norm": 0.7972877621650696 + }, + { + "bits_per_byte": 1.537403650944651, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.065647006034851, + "step": 6000, + "unclipped_grad_norm": 0.8044735789299011 + }, + { + "bits_per_byte": 1.5050372039828739, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0432122945785522, + "step": 6010, + "unclipped_grad_norm": 0.8286860585212708 + }, + { + "bits_per_byte": 1.4865072772309622, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0303683280944824, + "step": 6020, + "unclipped_grad_norm": 0.7835216522216797 + }, + { + "bits_per_byte": 1.5443446987508231, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.070458173751831, + "step": 6030, + "unclipped_grad_norm": 0.8111705183982849 + }, + { + "bits_per_byte": 1.4931416799706458, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.0349669456481934, + "step": 6040, + "unclipped_grad_norm": 0.856879472732544 + }, + { + "bits_per_byte": 1.5039554331089469, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.0424624681472778, + "step": 6050, + "unclipped_grad_norm": 0.7954339385032654 + }, + { + "bits_per_byte": 1.5332958453287928, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.0627996921539307, + "step": 6060, + "unclipped_grad_norm": 0.8386449813842773 + }, + { + "bits_per_byte": 1.5026495688409454, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 1.0415573120117188, + "step": 6070, + "unclipped_grad_norm": 0.8040284514427185 + }, + { + "bits_per_byte": 1.4853049465187804, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.029534935951233, + "step": 6080, + "unclipped_grad_norm": 0.8260617256164551 + }, + { + "bits_per_byte": 1.4899703198887357, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.032768726348877, + "step": 6090, + "unclipped_grad_norm": 0.8013850450515747 + }, + { + "bits_per_byte": 1.490422634260505, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.0330822467803955, + "step": 6100, + "unclipped_grad_norm": 0.7864043712615967 + }, + { + "bits_per_byte": 1.468088451271508, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.0176013708114624, + "step": 6110, + "unclipped_grad_norm": 0.8051435947418213 + }, + { + "bits_per_byte": 1.4773643355457935, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0240309238433838, + "step": 6120, + "unclipped_grad_norm": 0.7863143682479858 + }, + { + "bits_per_byte": 1.4991493779305856, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.0391311645507812, + "step": 6130, + "unclipped_grad_norm": 0.8004176020622253 + }, + { + "bits_per_byte": 1.5327358698175835, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.0624115467071533, + "step": 6140, + "unclipped_grad_norm": 0.8052223324775696 + }, + { + "bits_per_byte": 1.4716404089597777, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.0200634002685547, + "step": 6150, + "unclipped_grad_norm": 0.7689356803894043 + }, + { + "bits_per_byte": 1.5533985534228447, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 1.0767338275909424, + "step": 6160, + "unclipped_grad_norm": 0.8380405902862549 + }, + { + "bits_per_byte": 1.4273700108358802, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 0.989377498626709, + "step": 6170, + "unclipped_grad_norm": 0.7539782524108887 + }, + { + "bits_per_byte": 1.4939972936586885, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0355600118637085, + "step": 6180, + "unclipped_grad_norm": 0.7956816554069519 + }, + { + "bits_per_byte": 1.4872425030634007, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.030877947807312, + "step": 6190, + "unclipped_grad_norm": 0.8438931703567505 + }, + { + "bits_per_byte": 1.4718997587972786, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.0202431678771973, + "step": 6200, + "unclipped_grad_norm": 0.8038849234580994 + }, + { + "bits_per_byte": 1.4795144626468804, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.0255212783813477, + "step": 6210, + "unclipped_grad_norm": 0.7972837090492249 + }, + { + "bits_per_byte": 1.4707916745777696, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0194751024246216, + "step": 6220, + "unclipped_grad_norm": 0.7892265319824219 + }, + { + "bits_per_byte": 1.4445531414488852, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.0012879371643066, + "step": 6230, + "unclipped_grad_norm": 0.772609531879425 + }, + { + "bits_per_byte": 1.4751142865245555, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.022471308708191, + "step": 6240, + "unclipped_grad_norm": 0.8259294033050537 + }, + { + "bits_per_byte": 1.4569049354338521, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.0098495483398438, + "step": 6250, + "unclipped_grad_norm": 0.7973186373710632 + }, + { + "bits_per_byte": 1.5251835956702449, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 1.0571767091751099, + "step": 6260, + "unclipped_grad_norm": 0.8520693182945251 + }, + { + "bits_per_byte": 1.4872426750460517, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.0308780670166016, + "step": 6270, + "unclipped_grad_norm": 0.7952439785003662 + }, + { + "bits_per_byte": 1.5963835532141935, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.1065287590026855, + "step": 6280, + "unclipped_grad_norm": 0.8804905414581299 + }, + { + "bits_per_byte": 1.4705715367846652, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 1.0193225145339966, + "step": 6290, + "unclipped_grad_norm": 0.8222136497497559 + }, + { + "bits_per_byte": 1.5002524746532195, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.03989577293396, + "step": 6300, + "unclipped_grad_norm": 0.7908111214637756 + }, + { + "bits_per_byte": 1.5415828293606173, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.068543791770935, + "step": 6310, + "unclipped_grad_norm": 0.8476980328559875 + }, + { + "bits_per_byte": 1.4906214462049023, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 1.0332200527191162, + "step": 6320, + "unclipped_grad_norm": 0.804865300655365 + }, + { + "bits_per_byte": 1.4334268958339667, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 0.9935758113861084, + "step": 6330, + "unclipped_grad_norm": 0.7845658659934998 + }, + { + "bits_per_byte": 1.5053029171784569, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 1.0433964729309082, + "step": 6340, + "unclipped_grad_norm": 0.8431986570358276 + }, + { + "bits_per_byte": 1.506793318830834, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.0444295406341553, + "step": 6350, + "unclipped_grad_norm": 0.8601893782615662 + }, + { + "bits_per_byte": 1.4418732217918155, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 0.9994303584098816, + "step": 6360, + "unclipped_grad_norm": 0.7917057871818542 + }, + { + "bits_per_byte": 1.4950085516457616, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 1.036260962486267, + "step": 6370, + "unclipped_grad_norm": 0.8345564603805542 + }, + { + "bits_per_byte": 1.4521970823491337, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.0065863132476807, + "step": 6380, + "unclipped_grad_norm": 0.8242836594581604 + }, + { + "bits_per_byte": 1.4966232967547126, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0373802185058594, + "step": 6390, + "unclipped_grad_norm": 0.8000714182853699 + }, + { + "bits_per_byte": 1.4794289872694015, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.025462031364441, + "step": 6400, + "unclipped_grad_norm": 0.7891771197319031 + }, + { + "bits_per_byte": 1.4626636025153423, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.013841152191162, + "step": 6410, + "unclipped_grad_norm": 0.8274503350257874 + }, + { + "bits_per_byte": 1.4468915895526668, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0029088258743286, + "step": 6420, + "unclipped_grad_norm": 0.8203821182250977 + }, + { + "bits_per_byte": 1.4724366886332725, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.0206153392791748, + "step": 6430, + "unclipped_grad_norm": 0.8160783052444458 + }, + { + "bits_per_byte": 1.488760249957265, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.0319299697875977, + "step": 6440, + "unclipped_grad_norm": 0.8628734946250916 + }, + { + "bits_per_byte": 1.470941471466671, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0195789337158203, + "step": 6450, + "unclipped_grad_norm": 0.8452542424201965 + }, + { + "bits_per_byte": 1.4462543938312202, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.002467155456543, + "step": 6460, + "unclipped_grad_norm": 0.8988414406776428 + }, + { + "bits_per_byte": 1.4752357062760646, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.0225554704666138, + "step": 6470, + "unclipped_grad_norm": 0.806004524230957 + }, + { + "bits_per_byte": 1.5128484840024117, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 1.0486266613006592, + "step": 6480, + "unclipped_grad_norm": 0.8720566630363464 + }, + { + "bits_per_byte": 1.4845225974400054, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0289926528930664, + "step": 6490, + "unclipped_grad_norm": 0.8538320660591125 + }, + { + "bits_per_byte": 1.4937064709960794, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.0353584289550781, + "step": 6500, + "unclipped_grad_norm": 0.8489036560058594 + }, + { + "bits_per_byte": 1.4326988932728644, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 0.9930711984634399, + "step": 6510, + "unclipped_grad_norm": 0.8175879120826721 + }, + { + "bits_per_byte": 1.4006631689486966, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 0.9708657264709473, + "step": 6520, + "unclipped_grad_norm": 0.8166611790657043 + }, + { + "bits_per_byte": 1.5059882680421453, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.043871521949768, + "step": 6530, + "unclipped_grad_norm": 0.8702302575111389 + }, + { + "bits_per_byte": 1.4737977593322007, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.0215587615966797, + "step": 6540, + "unclipped_grad_norm": 0.81386798620224 + }, + { + "bits_per_byte": 1.5069716648397786, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 1.0445531606674194, + "step": 6550, + "unclipped_grad_norm": 0.8570835590362549 + }, + { + "bits_per_byte": 1.4602217928383923, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.0121486186981201, + "step": 6560, + "unclipped_grad_norm": 0.8049841523170471 + }, + { + "bits_per_byte": 1.4110844576777284, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 0.9780892133712769, + "step": 6570, + "unclipped_grad_norm": 0.9000046849250793 + }, + { + "bits_per_byte": 1.4991215167411458, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.039111852645874, + "step": 6580, + "unclipped_grad_norm": 0.833981990814209 + }, + { + "bits_per_byte": 1.4741881599496593, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.02182936668396, + "step": 6590, + "unclipped_grad_norm": 0.820466935634613 + }, + { + "bits_per_byte": 1.3980581477360776, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9690600633621216, + "step": 6600, + "unclipped_grad_norm": 0.80569988489151 + }, + { + "bits_per_byte": 1.514899033148649, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.0500479936599731, + "step": 6610, + "unclipped_grad_norm": 0.8121935725212097 + }, + { + "bits_per_byte": 1.5173026626771076, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 1.0517140626907349, + "step": 6620, + "unclipped_grad_norm": 0.8675882816314697 + }, + { + "bits_per_byte": 1.5035885941146563, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.042208194732666, + "step": 6630, + "unclipped_grad_norm": 0.8735457062721252 + }, + { + "bits_per_byte": 1.4762426646968663, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.0232534408569336, + "step": 6640, + "unclipped_grad_norm": 0.8865591287612915 + }, + { + "bits_per_byte": 1.4605107236918418, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.0123488903045654, + "step": 6650, + "unclipped_grad_norm": 0.8401181697845459 + }, + { + "bits_per_byte": 1.5170605111046929, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.0515462160110474, + "step": 6660, + "unclipped_grad_norm": 0.875600278377533 + }, + { + "bits_per_byte": 1.4813287076308321, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.0267788171768188, + "step": 6670, + "unclipped_grad_norm": 0.8744917511940002 + }, + { + "bits_per_byte": 1.5059530115987183, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.0438470840454102, + "step": 6680, + "unclipped_grad_norm": 0.8782076835632324 + }, + { + "bits_per_byte": 1.4718156592810068, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.020184874534607, + "step": 6690, + "unclipped_grad_norm": 0.8556012511253357 + }, + { + "bits_per_byte": 1.5236689444640963, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.0561268329620361, + "step": 6700, + "unclipped_grad_norm": 0.8650729656219482 + }, + { + "bits_per_byte": 1.459142773686879, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.0114006996154785, + "step": 6710, + "unclipped_grad_norm": 0.8319787979125977 + }, + { + "bits_per_byte": 1.4521162505032281, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.0065302848815918, + "step": 6720, + "unclipped_grad_norm": 0.8662752509117126 + }, + { + "bits_per_byte": 1.4211395953430734, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 0.9850589036941528, + "step": 6730, + "unclipped_grad_norm": 0.8533402681350708 + }, + { + "bits_per_byte": 1.4698897975566452, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 1.0188499689102173, + "step": 6740, + "unclipped_grad_norm": 0.8741611242294312 + }, + { + "bits_per_byte": 1.4398852743304924, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 0.9980524182319641, + "step": 6750, + "unclipped_grad_norm": 0.8483414053916931 + }, + { + "bits_per_byte": 1.4154785284159477, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 0.9811349511146545, + "step": 6760, + "unclipped_grad_norm": 0.8301479816436768 + }, + { + "bits_per_byte": 1.4384663314695487, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 0.9970688819885254, + "step": 6770, + "unclipped_grad_norm": 0.8384212255477905 + }, + { + "bits_per_byte": 1.4502099948010647, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 1.005208969116211, + "step": 6780, + "unclipped_grad_norm": 0.8327454328536987 + }, + { + "bits_per_byte": 1.5315159968750136, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.0615659952163696, + "step": 6790, + "unclipped_grad_norm": 0.8572182655334473 + }, + { + "bits_per_byte": 1.5111083635409819, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0474205017089844, + "step": 6800, + "unclipped_grad_norm": 0.9349188804626465 + }, + { + "bits_per_byte": 1.4879230384128648, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0313496589660645, + "step": 6810, + "unclipped_grad_norm": 0.8842780590057373 + }, + { + "bits_per_byte": 1.4287211065410585, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 0.9903140068054199, + "step": 6820, + "unclipped_grad_norm": 0.8380818367004395 + }, + { + "bits_per_byte": 1.4440282503984518, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.0009241104125977, + "step": 6830, + "unclipped_grad_norm": 0.8713801503181458 + }, + { + "bits_per_byte": 1.497092809391568, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 1.037705659866333, + "step": 6840, + "unclipped_grad_norm": 0.8698394298553467 + }, + { + "bits_per_byte": 1.44771745024211, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.0034812688827515, + "step": 6850, + "unclipped_grad_norm": 0.8601061105728149 + }, + { + "bits_per_byte": 1.458025918352176, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.0106265544891357, + "step": 6860, + "unclipped_grad_norm": 0.8748350739479065 + }, + { + "bits_per_byte": 1.4797149944177865, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 1.0256602764129639, + "step": 6870, + "unclipped_grad_norm": 0.9106220006942749 + }, + { + "bits_per_byte": 1.5139932005265546, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 1.0494201183319092, + "step": 6880, + "unclipped_grad_norm": 0.8633177280426025 + }, + { + "bits_per_byte": 1.397942403412047, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 0.9689798355102539, + "step": 6890, + "unclipped_grad_norm": 0.816047728061676 + }, + { + "bits_per_byte": 1.4317385421504465, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 0.9924055337905884, + "step": 6900, + "unclipped_grad_norm": 0.8447687029838562 + }, + { + "bits_per_byte": 1.600019610418735, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.1090490818023682, + "step": 6910, + "unclipped_grad_norm": 0.9243294596672058 + }, + { + "bits_per_byte": 1.4113978100676003, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 0.9783064126968384, + "step": 6920, + "unclipped_grad_norm": 0.8489204049110413 + }, + { + "bits_per_byte": 1.4319384719820745, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.9925441145896912, + "step": 6930, + "unclipped_grad_norm": 0.8538230657577515 + }, + { + "bits_per_byte": 1.4550657529655253, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 1.0085747241973877, + "step": 6940, + "unclipped_grad_norm": 0.8362781405448914 + }, + { + "bits_per_byte": 1.4427452598230155, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 1.0000348091125488, + "step": 6950, + "unclipped_grad_norm": 0.8803022503852844 + }, + { + "bits_per_byte": 1.4975286134288543, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 1.0380077362060547, + "step": 6960, + "unclipped_grad_norm": 0.8924822807312012 + }, + { + "bits_per_byte": 1.4602926496905477, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 1.012197732925415, + "step": 6970, + "unclipped_grad_norm": 0.8632159233093262 + }, + { + "bits_per_byte": 1.5205742886444709, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.0539817810058594, + "step": 6980, + "unclipped_grad_norm": 0.9233362078666687 + }, + { + "bits_per_byte": 1.458933298818128, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.0112555027008057, + "step": 6990, + "unclipped_grad_norm": 0.8967857360839844 + }, + { + "bits_per_byte": 1.404953620139771, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 0.9738396406173706, + "step": 7000, + "unclipped_grad_norm": 0.8754119873046875 + }, + { + "bits_per_byte": 1.4564221801328803, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 1.0095149278640747, + "step": 7010, + "unclipped_grad_norm": 0.8857821822166443 + }, + { + "bits_per_byte": 1.4183886468511973, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 0.9831520915031433, + "step": 7020, + "unclipped_grad_norm": 0.8428258299827576 + }, + { + "bits_per_byte": 1.4404831720162168, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 0.9984668493270874, + "step": 7030, + "unclipped_grad_norm": 0.8319308757781982 + }, + { + "bits_per_byte": 1.5218872042011575, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.05489182472229, + "step": 7040, + "unclipped_grad_norm": 0.9351308941841125 + }, + { + "bits_per_byte": 1.4687751779964031, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.0180773735046387, + "step": 7050, + "unclipped_grad_norm": 0.848258376121521 + }, + { + "bits_per_byte": 1.4903865179038238, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 1.0330572128295898, + "step": 7060, + "unclipped_grad_norm": 0.9581419825553894 + }, + { + "bits_per_byte": 1.4309001267274903, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 0.9918243885040283, + "step": 7070, + "unclipped_grad_norm": 0.8295577764511108 + }, + { + "bits_per_byte": 1.3550836388864316, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 0.939272403717041, + "step": 7080, + "unclipped_grad_norm": 0.8738391399383545 + }, + { + "bits_per_byte": 1.4267666956966536, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 0.9889593124389648, + "step": 7090, + "unclipped_grad_norm": 0.8696648478507996 + }, + { + "bits_per_byte": 1.397788134974223, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 0.9688729047775269, + "step": 7100, + "unclipped_grad_norm": 0.8454760909080505 + }, + { + "bits_per_byte": 1.4677217842598684, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 1.0173472166061401, + "step": 7110, + "unclipped_grad_norm": 0.8988236784934998 + }, + { + "bits_per_byte": 1.4718662221803605, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.0202199220657349, + "step": 7120, + "unclipped_grad_norm": 0.9128355383872986 + }, + { + "bits_per_byte": 1.4053619069529193, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 0.9741226434707642, + "step": 7130, + "unclipped_grad_norm": 0.8448015451431274 + }, + { + "bits_per_byte": 1.514108084937331, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.049499750137329, + "step": 7140, + "unclipped_grad_norm": 0.8931875228881836 + }, + { + "bits_per_byte": 1.545076656912895, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 1.0709655284881592, + "step": 7150, + "unclipped_grad_norm": 0.8927509784698486 + }, + { + "bits_per_byte": 1.460541680568997, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0123703479766846, + "step": 7160, + "unclipped_grad_norm": 0.9125353693962097 + }, + { + "bits_per_byte": 1.4573925062490483, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.0101875066757202, + "step": 7170, + "unclipped_grad_norm": 0.8619760274887085 + }, + { + "bits_per_byte": 1.3817432715359534, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 0.957751452922821, + "step": 7180, + "unclipped_grad_norm": 0.7999554872512817 + }, + { + "bits_per_byte": 1.3799526741664954, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 0.9565103054046631, + "step": 7190, + "unclipped_grad_norm": 0.8222221732139587 + }, + { + "bits_per_byte": 1.4825253629155357, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 1.0276082754135132, + "step": 7200, + "unclipped_grad_norm": 0.8906659483909607 + }, + { + "bits_per_byte": 1.480764948501304, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 1.0263880491256714, + "step": 7210, + "unclipped_grad_norm": 0.8587933778762817 + }, + { + "bits_per_byte": 1.4444729975335833, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 1.001232385635376, + "step": 7220, + "unclipped_grad_norm": 0.8614253997802734 + }, + { + "bits_per_byte": 1.4312375566884832, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 0.992058277130127, + "step": 7230, + "unclipped_grad_norm": 0.8867919445037842 + }, + { + "bits_per_byte": 1.3955073010584804, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 0.9672919511795044, + "step": 7240, + "unclipped_grad_norm": 0.8536276817321777 + }, + { + "bits_per_byte": 1.4414722442413288, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 0.999152421951294, + "step": 7250, + "unclipped_grad_norm": 0.9005672335624695 + }, + { + "bits_per_byte": 1.4225180362897387, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 0.9860143661499023, + "step": 7260, + "unclipped_grad_norm": 0.8858867287635803 + }, + { + "bits_per_byte": 1.499904725733175, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 1.0396547317504883, + "step": 7270, + "unclipped_grad_norm": 0.9363247752189636 + }, + { + "bits_per_byte": 1.4469973588829474, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.0029821395874023, + "step": 7280, + "unclipped_grad_norm": 0.8984721302986145 + }, + { + "bits_per_byte": 1.442100582856256, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 0.9995879530906677, + "step": 7290, + "unclipped_grad_norm": 0.8785810470581055 + }, + { + "bits_per_byte": 1.437434263581721, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 0.9963535070419312, + "step": 7300, + "unclipped_grad_norm": 0.9088795185089111 + }, + { + "bits_per_byte": 1.4833225025022847, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 1.028160810470581, + "step": 7310, + "unclipped_grad_norm": 0.8880798816680908 + }, + { + "bits_per_byte": 1.474129685848366, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 1.0217888355255127, + "step": 7320, + "unclipped_grad_norm": 0.9047746062278748 + }, + { + "bits_per_byte": 1.464644670670631, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 1.0152143239974976, + "step": 7330, + "unclipped_grad_norm": 0.944588303565979 + }, + { + "bits_per_byte": 1.4384054496111434, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.9970266819000244, + "step": 7340, + "unclipped_grad_norm": 0.8976200819015503 + }, + { + "bits_per_byte": 1.4702452856959785, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 1.0190963745117188, + "step": 7350, + "unclipped_grad_norm": 0.8795632123947144 + }, + { + "bits_per_byte": 1.4627798627873256, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 1.0139217376708984, + "step": 7360, + "unclipped_grad_norm": 0.874075174331665 + }, + { + "bits_per_byte": 1.4722351249664611, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 1.0204756259918213, + "step": 7370, + "unclipped_grad_norm": 0.8843042254447937 + }, + { + "bits_per_byte": 1.5082159593187712, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 1.0454156398773193, + "step": 7380, + "unclipped_grad_norm": 0.9491526484489441 + }, + { + "bits_per_byte": 1.484452944466406, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 1.0289443731307983, + "step": 7390, + "unclipped_grad_norm": 0.8915677070617676 + }, + { + "bits_per_byte": 1.4290258597983874, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.9905252456665039, + "step": 7400, + "unclipped_grad_norm": 0.8781417608261108 + }, + { + "bits_per_byte": 1.441836847461158, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 0.9994051456451416, + "step": 7410, + "unclipped_grad_norm": 0.868361234664917 + }, + { + "bits_per_byte": 1.5182549306149349, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0523741245269775, + "step": 7420, + "unclipped_grad_norm": 0.9446384310722351 + }, + { + "bits_per_byte": 1.4176792184163882, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 0.9826603531837463, + "step": 7430, + "unclipped_grad_norm": 0.9316976070404053 + }, + { + "bits_per_byte": 1.4405904891903554, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 0.9985412359237671, + "step": 7440, + "unclipped_grad_norm": 0.8922200798988342 + }, + { + "bits_per_byte": 1.383831742856706, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 0.959199070930481, + "step": 7450, + "unclipped_grad_norm": 0.9256144165992737 + }, + { + "bits_per_byte": 1.4154423260679412, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 0.9811098575592041, + "step": 7460, + "unclipped_grad_norm": 0.876299262046814 + }, + { + "bits_per_byte": 1.461966900796697, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 1.013358235359192, + "step": 7470, + "unclipped_grad_norm": 0.8973875045776367 + }, + { + "bits_per_byte": 1.3943412586856305, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 0.9664837121963501, + "step": 7480, + "unclipped_grad_norm": 0.8939456939697266 + }, + { + "bits_per_byte": 1.4122259065315048, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 0.9788804054260254, + "step": 7490, + "unclipped_grad_norm": 0.8852412104606628 + }, + { + "bits_per_byte": 1.479005909948279, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.025168776512146, + "step": 7500, + "unclipped_grad_norm": 0.9357314109802246 + }, + { + "bits_per_byte": 1.4640246732142708, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 1.014784574508667, + "step": 7510, + "unclipped_grad_norm": 0.9328818917274475 + }, + { + "bits_per_byte": 1.4351207529623144, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 0.994749903678894, + "step": 7520, + "unclipped_grad_norm": 0.8785576224327087 + }, + { + "bits_per_byte": 1.451345424262061, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.0059959888458252, + "step": 7530, + "unclipped_grad_norm": 0.9018154740333557 + }, + { + "bits_per_byte": 1.4939732160875676, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.0355433225631714, + "step": 7540, + "unclipped_grad_norm": 0.9328163266181946 + }, + { + "bits_per_byte": 1.427676913876345, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 0.9895902276039124, + "step": 7550, + "unclipped_grad_norm": 0.9355925917625427 + }, + { + "bits_per_byte": 1.4685663910582558, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 1.017932653427124, + "step": 7560, + "unclipped_grad_norm": 0.9279981851577759 + }, + { + "bits_per_byte": 1.4795531587433246, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 1.0255481004714966, + "step": 7570, + "unclipped_grad_norm": 0.9160473942756653 + }, + { + "bits_per_byte": 1.4271792820760734, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 0.9892452955245972, + "step": 7580, + "unclipped_grad_norm": 0.8842087388038635 + }, + { + "bits_per_byte": 1.4122523058684122, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 0.9788987040519714, + "step": 7590, + "unclipped_grad_norm": 0.8663433194160461 + }, + { + "bits_per_byte": 1.3991989086592505, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 0.9698507785797119, + "step": 7600, + "unclipped_grad_norm": 0.8966830372810364 + }, + { + "bits_per_byte": 1.3740469619185174, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 0.9524167776107788, + "step": 7610, + "unclipped_grad_norm": 0.8679200410842896 + }, + { + "bits_per_byte": 1.4654550529214967, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 1.0157760381698608, + "step": 7620, + "unclipped_grad_norm": 0.9321436285972595 + }, + { + "bits_per_byte": 1.4257648967553778, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 0.9882649183273315, + "step": 7630, + "unclipped_grad_norm": 0.9096561670303345 + }, + { + "bits_per_byte": 1.429847420921559, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.991094708442688, + "step": 7640, + "unclipped_grad_norm": 0.9261285662651062 + }, + { + "bits_per_byte": 1.513285835883556, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 1.0489298105239868, + "step": 7650, + "unclipped_grad_norm": 0.9924286007881165 + }, + { + "bits_per_byte": 1.4343550862006733, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 0.994219183921814, + "step": 7660, + "unclipped_grad_norm": 0.8935940861701965 + }, + { + "bits_per_byte": 1.423741176902675, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 0.9868621826171875, + "step": 7670, + "unclipped_grad_norm": 0.9237322211265564 + }, + { + "bits_per_byte": 1.402396238121441, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 0.9720669984817505, + "step": 7680, + "unclipped_grad_norm": 0.9117143154144287 + }, + { + "bits_per_byte": 1.4328639106263672, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 0.9931855797767639, + "step": 7690, + "unclipped_grad_norm": 0.895114004611969 + }, + { + "bits_per_byte": 1.4630005165283826, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 1.014074683189392, + "step": 7700, + "unclipped_grad_norm": 0.9410009384155273 + }, + { + "bits_per_byte": 1.3690389131167178, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 0.9489454627037048, + "step": 7710, + "unclipped_grad_norm": 0.872177004814148 + }, + { + "bits_per_byte": 1.3723180203293934, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 0.9512183666229248, + "step": 7720, + "unclipped_grad_norm": 0.8478551506996155 + }, + { + "bits_per_byte": 1.439988291938359, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 0.998123824596405, + "step": 7730, + "unclipped_grad_norm": 0.904336154460907 + }, + { + "bits_per_byte": 1.4136017677384074, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 0.9798340797424316, + "step": 7740, + "unclipped_grad_norm": 0.8677012920379639 + }, + { + "bits_per_byte": 1.4310562869744738, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9919326305389404, + "step": 7750, + "unclipped_grad_norm": 0.9314078688621521 + }, + { + "bits_per_byte": 1.414364166829682, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 0.9803625345230103, + "step": 7760, + "unclipped_grad_norm": 0.9044833183288574 + }, + { + "bits_per_byte": 1.402848896458512, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 0.9723807573318481, + "step": 7770, + "unclipped_grad_norm": 0.8981534838676453 + }, + { + "bits_per_byte": 1.5398874243884115, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 1.0673686265945435, + "step": 7780, + "unclipped_grad_norm": 0.9842527508735657 + }, + { + "bits_per_byte": 1.425984002652577, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 0.9884167909622192, + "step": 7790, + "unclipped_grad_norm": 0.8993093371391296 + }, + { + "bits_per_byte": 1.4644575535464923, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 1.0150846242904663, + "step": 7800, + "unclipped_grad_norm": 0.978873074054718 + }, + { + "bits_per_byte": 1.39211666309672, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9649417400360107, + "step": 7810, + "unclipped_grad_norm": 0.9312298893928528 + }, + { + "bits_per_byte": 1.4169101120017298, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 0.9821272492408752, + "step": 7820, + "unclipped_grad_norm": 0.9480841159820557 + }, + { + "bits_per_byte": 1.453633137483838, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 1.0075817108154297, + "step": 7830, + "unclipped_grad_norm": 0.9303885698318481 + }, + { + "bits_per_byte": 1.4100764673610215, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.9773905277252197, + "step": 7840, + "unclipped_grad_norm": 0.8638455271720886 + }, + { + "bits_per_byte": 1.3817875570685507, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.9577821493148804, + "step": 7850, + "unclipped_grad_norm": 0.9019877910614014 + }, + { + "bits_per_byte": 1.3966441063806836, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 0.9680799245834351, + "step": 7860, + "unclipped_grad_norm": 0.8802809119224548 + }, + { + "bits_per_byte": 1.446224640832621, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0024465322494507, + "step": 7870, + "unclipped_grad_norm": 0.9661257863044739 + }, + { + "bits_per_byte": 1.4321896526436597, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9927182197570801, + "step": 7880, + "unclipped_grad_norm": 0.9163897037506104 + }, + { + "bits_per_byte": 1.502017016651072, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 1.041118860244751, + "step": 7890, + "unclipped_grad_norm": 0.9412688612937927 + }, + { + "bits_per_byte": 1.4600511860487364, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 1.0120303630828857, + "step": 7900, + "unclipped_grad_norm": 0.9177646636962891 + }, + { + "bits_per_byte": 1.4703694571699015, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.0191824436187744, + "step": 7910, + "unclipped_grad_norm": 1.0214102268218994 + }, + { + "bits_per_byte": 1.378860498342191, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 0.9557532668113708, + "step": 7920, + "unclipped_grad_norm": 0.9210410714149475 + }, + { + "bits_per_byte": 1.382480905125504, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.9582627415657043, + "step": 7930, + "unclipped_grad_norm": 0.8971716165542603 + }, + { + "bits_per_byte": 1.4463304101629015, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 1.0025198459625244, + "step": 7940, + "unclipped_grad_norm": 0.9843243360519409 + }, + { + "bits_per_byte": 1.4058984068322857, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 0.9744945168495178, + "step": 7950, + "unclipped_grad_norm": 0.9143235087394714 + }, + { + "bits_per_byte": 1.4271902029744032, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 0.9892528653144836, + "step": 7960, + "unclipped_grad_norm": 0.9196386933326721 + }, + { + "bits_per_byte": 1.4201626478948473, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 0.9843817353248596, + "step": 7970, + "unclipped_grad_norm": 0.9190718531608582 + }, + { + "bits_per_byte": 1.4561760729594955, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 1.0093443393707275, + "step": 7980, + "unclipped_grad_norm": 0.9694462418556213 + }, + { + "bits_per_byte": 1.4533447225783411, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 1.007381796836853, + "step": 7990, + "unclipped_grad_norm": 0.9130844473838806 + }, + { + "bits_per_byte": 1.3815890890894549, + "learning_rate": 3e-05, + "loss_nats": 0.9576445817947388, + "step": 8000, + "unclipped_grad_norm": 0.8965908885002136 + } + ], + "variant": "uniform_group_6_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073001.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073001.json new file mode 100644 index 0000000..9d284d5 --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073001.json @@ -0,0 +1,14045 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "13c48a7f312c32bd2ec4a201490f5e15615d1b9403674df1e10fa36d7f203c66", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.0002677410375326872, + 0.0002473295899108052, + 0.00023034414334688336, + 0.00021438254043459892, + 0.00020044577831868082, + 0.00018819731485564262, + 0.00017973134526982903, + 0.000171217747265473, + 0.0001640921109355986, + 0.00016061548376455903, + 0.00015463090676348656, + 0.00015024171443656087, + 0.0001478576596127823, + 0.00014403100067283958, + 0.00013951722939964384, + 0.00013661097909789532, + 0.00013379570737015456, + 0.0001311724045081064, + 0.0001286817860091105, + 0.00012687484559137374, + 0.0001249351626029238, + 0.00012369138130452484, + 0.00012227990373503417, + 0.00012101328320568427, + 0.00011953213106608018, + 0.0001181897969217971, + 0.0001171672047348693, + 0.00011609455395955592, + 0.00011492414341773838, + 0.0001140913664130494, + 0.00011346775136189535, + 0.00011285000800853595 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00021242368711682502, + "first_to_last_ratio": 1.8345658968465297, + "imbalance_abs_log_ratio": 0.6068078850437921, + "last_quartile_mean": 0.0001157896194854402, + "mean": 0.0001511171253696375, + "normalized": [ + 1.7717451736709768, + 1.636674793183293, + 1.5242755762025908, + 1.4186515254985965, + 1.3264266232459345, + 1.2453738409548603, + 1.1893512719369177, + 1.1330135273991528, + 1.085860457802012, + 1.0628542818802846, + 1.023252039669589, + 0.9942070699734702, + 0.9784308644775869, + 0.9531083940389613, + 0.9232390376562555, + 0.9040072643238835, + 0.8853775311228679, + 0.8680181295617843, + 0.8515367513400655, + 0.8395795332993111, + 0.8267439067367663, + 0.8185133286645813, + 0.8091730400239778, + 0.800791325997512, + 0.7909899739933552, + 0.7821072339266707, + 0.7753403490721149, + 0.7682422073314642, + 0.7604971517068638, + 0.7549863467424895, + 0.7508596466770359, + 0.7467718018887739 + ], + "population_cv": 0.2714186609524326 + }, + "activation_output_rms_by_block": [ + 0.0034885562490671873, + 0.004831995815038681, + 0.005952076520770788, + 0.006925336085259914, + 0.00385080324485898, + 0.005542940925806761, + 0.006729288958013058, + 0.007927131839096546, + 0.004081308841705322, + 0.005811207927763462, + 0.007534808944910765, + 0.008581922389566898, + 0.00499920267611742, + 0.007102519739419222, + 0.008542952127754688, + 0.010447698645293713, + 0.005041283555328846, + 0.007636935915797949, + 0.009387043304741383, + 0.011020516976714134, + 0.005807018838822842, + 0.008367395959794521, + 0.01024632342159748, + 0.012378672137856483, + 0.006453148555010557, + 0.009127611294388771, + 0.011037957854568958, + 0.012611051090061665, + 0.007010402157902718, + 0.009794782847166061, + 0.012445803731679916, + 0.014087481424212456 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.0056560162047389895, + "first_to_last_ratio": 0.5480088979804595, + "imbalance_abs_log_ratio": 0.6014637549753238, + "last_quartile_mean": 0.010321029869373888, + "mean": 0.007962599374877755, + "normalized": [ + 0.4381177659237371, + 0.606836484864952, + 0.747504205668035, + 0.8697330807712821, + 0.4836113263475722, + 0.6961220406611074, + 0.845112084785299, + 0.9955457339856744, + 0.512559862622497, + 0.7298129234151854, + 0.9462750278110585, + 1.077778999737589, + 0.6278355146046978, + 0.8919850673170736, + 1.072884836415099, + 1.3120964842531853, + 0.6331203314377778, + 0.9591008609440675, + 1.1788918244911069, + 1.384035094304029, + 0.7292868277593074, + 1.0508372411895934, + 1.2868063479276561, + 1.554601902603762, + 0.8104324041933391, + 1.1463105029730196, + 1.3862254440922965, + 1.5837857082010076, + 0.8804162846646224, + 1.2300986632667847, + 1.5630327667805075, + 1.7692063559870779 + ], + "population_cv": 0.3451071044649734 + }, + "bits_per_byte": 7.959341947457508, + "branch_output_rms_by_sublayer": [ + 0.0022265110164880753, + 0.0027073738165199757, + 0.0021613112185150385, + 0.002722575329244137, + 0.0025601957459002733, + 0.0026850115973502398, + 0.0025323182344436646, + 0.0026861224323511124, + 0.0027253651060163975, + 0.00271838391199708, + 0.002904455177485943, + 0.0026645169127732515, + 0.0029208383057266474, + 0.0026859452482312918, + 0.00299624796025455, + 0.0027359756641089916, + 0.0030937574338167906, + 0.0026767742820084095, + 0.0033804215490818024, + 0.0027451682835817337, + 0.0034370317589491606, + 0.0026999011170119047, + 0.0038969130255281925, + 0.002663193503394723, + 0.004255065228790045, + 0.002626648638397455, + 0.004119740333408117, + 0.0026531771291047335, + 0.003970506135374308, + 0.0026517712976783514, + 0.004385598469525576, + 0.002676903735846281, + 0.0041999719105660915, + 0.002670086920261383, + 0.005232699681073427, + 0.0026355870068073273, + 0.004664790350943804, + 0.0026610493659973145, + 0.004898969549685717, + 0.0026388936676084995, + 0.005131533369421959, + 0.0026469926815479994, + 0.005069608334451914, + 0.00264862016774714, + 0.0051318020559847355, + 0.002641494618728757, + 0.0058372304774820805, + 0.002674198243767023, + 0.0059690922498703, + 0.002557571977376938, + 0.00554921617731452, + 0.00252733426168561, + 0.0058550527319312096, + 0.0025468147359788418, + 0.006255472078919411, + 0.0026856842450797558, + 0.006502739619463682, + 0.002551089273765683, + 0.006122888531535864, + 0.0025778315030038357, + 0.007071305997669697, + 0.0026001501828432083, + 0.006241494789719582, + 0.0025662274565547705 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0056993408498569815, + 0.005586029321860134, + 0.005830344684528519, + 0.00540288712216694, + 0.00538025811909819, + 0.0052226633069546126, + 0.005297605878918106, + 0.0052023753769565, + 0.0048107019260181325, + 0.005248672753719287, + 0.004962353375385338, + 0.005501319898161136, + 0.005588445553926757, + 0.005229606380765812, + 0.004884302670286647, + 0.005031817046498285, + 0.005000470961685071, + 0.0053937604587735305, + 0.004980398173225761, + 0.004658922297714218, + 0.00496598185096497, + 0.004578385934052906, + 0.004596161734371138, + 0.004990442293511754, + 0.005201771295872185, + 0.00459813521112035, + 0.004706502430387731, + 0.004781878207244352, + 0.0048378722082161795, + 0.004473110720597102, + 0.005084432926394394, + 0.004685705425265166 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.005452688082542498, + "first_to_last_ratio": 1.1368823875795573, + "imbalance_abs_log_ratio": 0.12828976841498158, + "last_quartile_mean": 0.004796176053137182, + "mean": 0.005075395512328069, + "normalized": [ + 1.122935313319594, + 1.1006096585560166, + 1.1487468652180286, + 1.064525337787646, + 1.060066768398564, + 1.0290160233362764, + 1.0437818818356701, + 1.025018713185997, + 0.947847692723257, + 1.0341406380981206, + 0.9777274230809889, + 1.0839194472230789, + 1.1010857262951226, + 1.0303840100861434, + 0.9623491722808084, + 0.9914137793352396, + 0.9852376922229987, + 1.0627271206100406, + 0.9812827711906272, + 0.9179427074003902, + 0.9784423379227619, + 0.9020747098294245, + 0.9055770576316904, + 0.9832617539638114, + 1.024899691706223, + 0.905965889742295, + 0.9273173724009683, + 0.9421685848185097, + 0.9532010257062827, + 0.8813324419214177, + 1.00178063247374, + 0.9232197596982638 + ], + "population_cv": 0.06921897980134169 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.02802751213312149, + 0.014061377383768559, + 0.01413465291261673, + 0.014165290631353855, + 0.014225807972252369, + 0.014281988143920898, + 0.01433455292135477, + 0.014350161887705326, + 0.01440478302538395, + 0.00966146495193243, + 0.009703479707241058, + 0.009741888381540775, + 0.00977881159633398, + 0.009820535778999329, + 0.009858808480203152, + 0.00991273857653141, + 0.009950309991836548, + 0.0074927168898284435, + 0.00752287870272994, + 0.007546746172010899, + 0.007571155205368996, + 0.00766373984515667, + 0.007693085819482803, + 0.007762390188872814, + 0.007788818795233965, + 0.006325984839349985, + 0.006353048607707024, + 0.006375366821885109, + 0.006389216054230928, + 0.006445344537496567, + 0.0064707002602517605, + 0.006555322092026472, + 0.00656962301582098, + 0.005531130358576775, + 0.005553483963012695, + 0.005611324217170477, + 0.0056249531917274, + 0.0056777093559503555, + 0.005688921548426151, + 0.00574630219489336, + 0.005768724251538515, + 0.004946626722812653, + 0.004966245498508215, + 0.0050392793491482735, + 0.005057469941675663, + 0.005084683652967215, + 0.005098348017781973, + 0.005203457549214363, + 0.005216081161051989, + 0.004667261149734259, + 0.0046758754178881645, + 0.004717131145298481, + 0.004727107938379049, + 0.004768412560224533, + 0.004777040798217058, + 0.004826710093766451, + 0.004837613552808762, + 0.0044052014127373695, + 0.0044186655431985855, + 0.004504534415900707, + 0.004510571248829365, + 0.004574235528707504, + 0.0045849340967834, + 0.004644096828997135 + ], + "loss_nats": 5.516995429992676, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0022265110164880753, + 0.0034885562490671873, + 0.0040087271481752396, + 0.004831995815038681, + 0.005379301495850086, + 0.005952076520770788, + 0.006410556845366955, + 0.006925336085259914, + 0.0027253651060163975, + 0.00385080324485898, + 0.004838948138058186, + 0.005542940925806761, + 0.006175771821290255, + 0.006729288958013058, + 0.00742313964292407, + 0.007927131839096546, + 0.0030937574338167906, + 0.004081308841705322, + 0.005150026176124811, + 0.005811207927763462, + 0.007019025273621082, + 0.007534808944910765, + 0.008143599145114422, + 0.008581922389566898, + 0.004255065228790045, + 0.00499920267611742, + 0.0066224560141563416, + 0.007102519739419222, + 0.008129785768687725, + 0.008542952127754688, + 0.010099578648805618, + 0.010447698645293713, + 0.0041999719105660915, + 0.005041283555328846, + 0.00719039561226964, + 0.007636935915797949, + 0.009030314162373543, + 0.009387043304741383, + 0.010736005380749702, + 0.011020516976714134, + 0.005131533369421959, + 0.005807018838822842, + 0.00787449348717928, + 0.008367395959794521, + 0.009912566281855106, + 0.01024632342159748, + 0.012113703414797783, + 0.012378672137856483, + 0.0059690922498703, + 0.006453148555010557, + 0.00876442901790142, + 0.009127611294388771, + 0.010806876234710217, + 0.011037957854568958, + 0.012353328987956047, + 0.012611051090061665, + 0.006502739619463682, + 0.007010402157902718, + 0.009470963850617409, + 0.009794782847166061, + 0.012123330496251583, + 0.012445803731679916, + 0.01384514570236206, + 0.014087481424212456 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.974941061343998e-05, + 6.65338448015973e-05, + 6.307093281066045e-05, + 6.0219437727937475e-05, + 5.5739485105732456e-05, + 5.2711369789903983e-05, + 4.9725767894415185e-05, + 4.6466440835502e-05, + 4.218258254695684e-05, + 3.949318124796264e-05, + 3.673651735880412e-05, + 3.430777360335924e-05, + 3.1624895200366154e-05, + 2.949452755274251e-05, + 2.7691108698491007e-05, + 2.608137219795026e-05, + 2.4780991225270554e-05, + 2.359667996643111e-05, + 2.261923873447813e-05, + 2.1716778064728715e-05, + 2.1001040295232087e-05, + 2.030819268838968e-05, + 1.9718510884558782e-05, + 1.923036325024441e-05, + 1.8957081920234486e-05, + 1.85366425284883e-05, + 1.818706550693605e-05, + 1.7875350749818608e-05, + 1.7483942428953014e-05, + 1.7282449334743433e-05, + 1.7089892935473472e-05, + 1.6942854927037843e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.80270861973986e-05, + "first_to_last_ratio": 3.2609727471827696, + "imbalance_abs_log_ratio": 1.1820255395759312, + "last_quartile_mean": 1.779441004146065e-05, + "mean": 3.272361635708876e-05, + "normalized": [ + 2.131470123971506, + 2.0332057458308515, + 1.9273827233033705, + 1.8402439715344123, + 1.7033412351950483, + 1.6108051510781562, + 1.5195682332843183, + 1.419966556521379, + 1.2890562609782896, + 1.2068709282312382, + 1.1226301200309137, + 1.0484102132534416, + 0.9664242134874989, + 0.9013223731414776, + 0.8462117510582664, + 0.7970198621492022, + 0.7572815594356633, + 0.7210902275878649, + 0.6912206306189088, + 0.6636423623767461, + 0.6417701535815351, + 0.6205974445727914, + 0.6025773762100489, + 0.5876600874548094, + 0.5793088915775626, + 0.5664606969538927, + 0.5557779833522671, + 0.5462523015414327, + 0.534291266532513, + 0.5281338451763025, + 0.5222495200097703, + 0.5177561899685208 + ], + "population_cv": 0.501910780095511 + }, + "activation_output_rms_by_block": [ + 0.0058675240725278854, + 0.010148076340556145, + 0.01483201701194048, + 0.019581714645028114, + 0.006728556007146835, + 0.011936783790588379, + 0.017140226438641548, + 0.023016884922981262, + 0.007510852534323931, + 0.013793004676699638, + 0.02042997069656849, + 0.02688039094209671, + 0.009191237390041351, + 0.016781959682703018, + 0.024295387789607048, + 0.031403254717588425, + 0.00985476840287447, + 0.018672063946723938, + 0.02787620574235916, + 0.03780702129006386, + 0.012548788450658321, + 0.02336551994085312, + 0.03465462848544121, + 0.046962957829236984, + 0.017280612140893936, + 0.02920808270573616, + 0.04429623484611511, + 0.05986764654517174, + 0.019072027876973152, + 0.03784291073679924, + 0.056897345930337906, + 0.07636407017707825 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013656472903676331, + "first_to_last_ratio": 0.320547269628702, + "imbalance_abs_log_ratio": 1.1377255263555355, + "last_quartile_mean": 0.042603616369888186, + "mean": 0.02537839770957362, + "normalized": [ + 0.23120151790806123, + 0.3998706481271643, + 0.5844347299492956, + 0.7715898721865017, + 0.2651292679761492, + 0.4703521446543257, + 0.6753864698154547, + 0.9069479163492851, + 0.29595456026329725, + 0.5434939129942169, + 0.8050142065849015, + 1.0591839268070296, + 0.36216775760331377, + 0.6612694731461426, + 0.9573255202176132, + 1.237401000526602, + 0.3883132621551165, + 0.7357463682461002, + 1.098422605767711, + 1.4897323985036979, + 0.494467325883402, + 0.9206853879525588, + 1.365516802204115, + 1.8505091758224324, + 0.6809181705894334, + 1.150903340706882, + 1.7454307144617338, + 2.359000250145326, + 0.7515063833119188, + 1.4911465715790075, + 2.241959739990765, + 3.009018577570445 + ], + "population_cv": 0.6532868931140454 + }, + "bits_per_byte": 5.151190650043275, + "branch_output_rms_by_sublayer": [ + 0.0029148487374186516, + 0.005083057563751936, + 0.0026889806613326073, + 0.0052253082394599915, + 0.002842964604496956, + 0.005695616360753775, + 0.0029088223818689585, + 0.005877713672816753, + 0.003076757537201047, + 0.0059105874970555305, + 0.003284021280705929, + 0.005919222719967365, + 0.00385852693580091, + 0.00588945671916008, + 0.0033993269316852093, + 0.0066772932186722755, + 0.003750445321202278, + 0.006290686782449484, + 0.004310164600610733, + 0.006426629144698381, + 0.004354329779744148, + 0.0068703917786479, + 0.004852214362472296, + 0.006513176020234823, + 0.005523395258933306, + 0.006732374895364046, + 0.0054849484004080296, + 0.007042272947728634, + 0.005391012877225876, + 0.006981347221881151, + 0.005734470672905445, + 0.006853354629129171, + 0.005626704078167677, + 0.007042378187179565, + 0.007186792325228453, + 0.007168001960963011, + 0.006888850592076778, + 0.007524309679865837, + 0.007285580970346928, + 0.007718907669186592, + 0.007807946298271418, + 0.00789446011185646, + 0.008222765289247036, + 0.00821562111377716, + 0.008884192444384098, + 0.008158302865922451, + 0.009264851920306683, + 0.008294272236526012, + 0.01269012875854969, + 0.007940465584397316, + 0.008947747759521008, + 0.008016594685614109, + 0.012151533737778664, + 0.008417033590376377, + 0.012250049971044064, + 0.00902593694627285, + 0.01350546907633543, + 0.008439894765615463, + 0.01492574717849493, + 0.008805050514638424, + 0.015486398711800575, + 0.008856607601046562, + 0.014740203507244587, + 0.009200834669172764 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00039601879790061005, + 0.000333862432555857, + 0.0003161007854675207, + 0.00030988523092101834, + 0.00028427570869778146, + 0.0002825306549544828, + 0.0002943972531561356, + 0.0002622653083661585, + 0.0002403195999084796, + 0.00025988749521925844, + 0.00023729733365524382, + 0.00024328940067478417, + 0.00024965496925999223, + 0.00023540665147734743, + 0.0002103490628616318, + 0.00021047475283705516, + 0.00019861827276508777, + 0.00020527520450466737, + 0.00019782799213683124, + 0.00018762602752636965, + 0.00018818340238149357, + 0.00017792823449411867, + 0.0001758567146683672, + 0.00017180438220615693, + 0.0001881011724950432, + 0.00015104948613129673, + 0.00017350178049072384, + 0.00015950290593162857, + 0.00016172504907608438, + 0.00016959582981761887, + 0.00017033648988492637, + 0.00016261075710768377 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00030991702150244554, + "first_to_last_ratio": 1.8552025057483756, + "imbalance_abs_log_ratio": 0.6179938576203984, + "last_quartile_mean": 0.00016705293386687573, + "mean": 0.00022517372311035798, + "normalized": [ + 1.7587256293955784, + 1.4826882459647857, + 1.4038084955081516, + 1.3762051212749329, + 1.262472835511351, + 1.2547230247466048, + 1.3074227715809055, + 1.164724305942314, + 1.0672630739897595, + 1.15416440084302, + 1.0538411426405383, + 1.0804520052970288, + 1.108721594205004, + 1.0454445937369623, + 0.9341634536927713, + 0.9347216448248767, + 0.8820668327615858, + 0.9116303699613373, + 0.8785571842229364, + 0.8332501010094053, + 0.8357254113938712, + 0.7901820516016239, + 0.7809823998965436, + 0.7629859285221986, + 0.8353602271915829, + 0.6708131128482836, + 0.7705241006548991, + 0.7083548814150751, + 0.7182234536168445, + 0.7531777130784469, + 0.7564669959355969, + 0.7221568967351844 + ], + "population_cv": 0.2632870685857609 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6930894255638123, + "mean_weights": [ + 0.5025115609169006, + 0.49748843908309937 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929049491882324, + "mean_weights": [ + 0.5103171467781067, + 0.4896828532218933 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928620934486389, + "mean_weights": [ + 0.5053681135177612, + 0.4946318566799164 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927832365036011, + "mean_weights": [ + 0.5126656293869019, + 0.48733440041542053 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928107738494873, + "mean_weights": [ + 0.5030388832092285, + 0.4969611167907715 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927524209022522, + "mean_weights": [ + 0.5128893852233887, + 0.4871106743812561 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6925663948059082, + "mean_weights": [ + 0.5112941265106201, + 0.4887058734893799 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929243803024292, + "mean_weights": [ + 0.509519636631012, + 0.49048036336898804 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0983479022979736, + "mean_weights": [ + 0.33988624811172485, + 0.3283587694168091, + 0.3317549526691437 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098447322845459, + "mean_weights": [ + 0.3408041000366211, + 0.33043020963668823, + 0.3287656307220459 + ], + "sources": 3 + }, + { + "entropy_mean": 1.09804105758667, + "mean_weights": [ + 0.34727632999420166, + 0.3313482403755188, + 0.32137539982795715 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985231399536133, + "mean_weights": [ + 0.33737295866012573, + 0.3342229723930359, + 0.3284040689468384 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982680320739746, + "mean_weights": [ + 0.34455230832099915, + 0.33067673444747925, + 0.324770987033844 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985300540924072, + "mean_weights": [ + 0.3371764123439789, + 0.33328622579574585, + 0.32953736186027527 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0983145236968994, + "mean_weights": [ + 0.34229838848114014, + 0.3327796757221222, + 0.32492193579673767 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984891653060913, + "mean_weights": [ + 0.3379301428794861, + 0.33446988463401794, + 0.32760000228881836 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3861057758331299, + "mean_weights": [ + 0.2561870217323303, + 0.2513977885246277, + 0.24581778049468994, + 0.24659737944602966 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861830234527588, + "mean_weights": [ + 0.25369757413864136, + 0.25217315554618835, + 0.2471860945224762, + 0.2469431757926941 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3860965967178345, + "mean_weights": [ + 0.2570868730545044, + 0.24996671080589294, + 0.24625366926193237, + 0.2466927170753479 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862223625183105, + "mean_weights": [ + 0.25144249200820923, + 0.25284475088119507, + 0.24949932098388672, + 0.24621345102787018 + ], + "sources": 4 + }, + { + "entropy_mean": 1.386009931564331, + "mean_weights": [ + 0.25487321615219116, + 0.25375044345855713, + 0.24856814742088318, + 0.24280822277069092 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862274885177612, + "mean_weights": [ + 0.251717746257782, + 0.2530381679534912, + 0.24868176877498627, + 0.24656230211257935 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859820365905762, + "mean_weights": [ + 0.2546093463897705, + 0.25476738810539246, + 0.24875231087207794, + 0.2418709397315979 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862541913986206, + "mean_weights": [ + 0.2502170205116272, + 0.25215911865234375, + 0.2499476969242096, + 0.24767616391181946 + ], + "sources": 4 + }, + { + "entropy_mean": 1.609063982963562, + "mean_weights": [ + 0.20365189015865326, + 0.2064075767993927, + 0.1998407393693924, + 0.1955183446407318, + 0.19458144903182983 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609306812286377, + "mean_weights": [ + 0.2017529010772705, + 0.20363107323646545, + 0.20011015236377716, + 0.19684572517871857, + 0.1976601779460907 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6086302995681763, + "mean_weights": [ + 0.2043541818857193, + 0.20843032002449036, + 0.20422470569610596, + 0.1954159438610077, + 0.1875748485326767 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609345555305481, + "mean_weights": [ + 0.20028294622898102, + 0.20384901762008667, + 0.20015379786491394, + 0.19763919711112976, + 0.19807502627372742 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6086030006408691, + "mean_weights": [ + 0.20521211624145508, + 0.21035641431808472, + 0.20178914070129395, + 0.1938098669052124, + 0.18883244693279266 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093711853027344, + "mean_weights": [ + 0.20029732584953308, + 0.2033734768629074, + 0.20007534325122833, + 0.19802936911582947, + 0.1982244849205017 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6083765029907227, + "mean_weights": [ + 0.20369838178157806, + 0.21107208728790283, + 0.20391488075256348, + 0.1940811723470688, + 0.18723349273204803 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093382835388184, + "mean_weights": [ + 0.20086458325386047, + 0.20333075523376465, + 0.2011795938014984, + 0.19899898767471313, + 0.19562608003616333 + ], + "sources": 5 + }, + { + "entropy_mean": 1.790840983390808, + "mean_weights": [ + 0.1716446876525879, + 0.17735548317432404, + 0.16901014745235443, + 0.16098332405090332, + 0.15850363671779633, + 0.1625027358531952 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7916771173477173, + "mean_weights": [ + 0.16707950830459595, + 0.16981208324432373, + 0.16858437657356262, + 0.1652665138244629, + 0.16461315751075745, + 0.16464436054229736 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7898242473602295, + "mean_weights": [ + 0.17289358377456665, + 0.18102474510669708, + 0.17351795732975006, + 0.16231776773929596, + 0.1569076031446457, + 0.15333834290504456 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917211055755615, + "mean_weights": [ + 0.16617721319198608, + 0.16886237263679504, + 0.16764816641807556, + 0.16660192608833313, + 0.16607347130775452, + 0.16463688015937805 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7891050577163696, + "mean_weights": [ + 0.17248018085956573, + 0.18408390879631042, + 0.17476218938827515, + 0.16211476922035217, + 0.1555776298046112, + 0.15098130702972412 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917201519012451, + "mean_weights": [ + 0.16665437817573547, + 0.16895613074302673, + 0.1659712791442871, + 0.16513998806476593, + 0.16659502685070038, + 0.16668319702148438 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7891126871109009, + "mean_weights": [ + 0.17235681414604187, + 0.18359792232513428, + 0.1746135950088501, + 0.16401760280132294, + 0.15544399619102478, + 0.14997008442878723 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917444705963135, + "mean_weights": [ + 0.16582947969436646, + 0.1677287220954895, + 0.16632026433944702, + 0.16698063910007477, + 0.16699230670928955, + 0.1661486029624939 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9437191486358643, + "mean_weights": [ + 0.14780816435813904, + 0.15844857692718506, + 0.14988581836223602, + 0.14075975120067596, + 0.1353664994239807, + 0.13102254271507263, + 0.13670864701271057 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458949565887451, + "mean_weights": [ + 0.14285539090633392, + 0.14305751025676727, + 0.14160889387130737, + 0.14283885061740875, + 0.14389647543430328, + 0.14331260323524475, + 0.14243027567863464 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9426133632659912, + "mean_weights": [ + 0.1483447551727295, + 0.16144436597824097, + 0.15264388918876648, + 0.14237377047538757, + 0.13469436764717102, + 0.13087959587574005, + 0.12961925566196442 + ], + "sources": 7 + }, + { + "entropy_mean": 1.945899486541748, + "mean_weights": [ + 0.14265955984592438, + 0.14297081530094147, + 0.14172807335853577, + 0.14274433255195618, + 0.1432293802499771, + 0.14324454963207245, + 0.14342327415943146 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9411585330963135, + "mean_weights": [ + 0.14914065599441528, + 0.16433146595954895, + 0.15439847111701965, + 0.1425367295742035, + 0.13549207150936127, + 0.12850183248519897, + 0.12559874355793 + ], + "sources": 7 + }, + { + "entropy_mean": 1.945886254310608, + "mean_weights": [ + 0.14266645908355713, + 0.14309072494506836, + 0.14319351315498352, + 0.14221635460853577, + 0.14128589630126953, + 0.14307740330696106, + 0.14446964859962463 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9416486024856567, + "mean_weights": [ + 0.1489623785018921, + 0.16320645809173584, + 0.1538664698600769, + 0.14291483163833618, + 0.1364762783050537, + 0.129567950963974, + 0.12500563263893127 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0793356895446777, + "mean_weights": [ + 0.12416472285985947, + 0.12248896062374115, + 0.12276060879230499, + 0.12438765913248062, + 0.1256687194108963, + 0.1261385828256607, + 0.12748940289020538, + 0.12690132856369019 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1906580924987793, + "mean_weights": [ + 0.11858292669057846, + 0.1327415108680725, + 0.12362463027238846, + 0.11748363077640533, + 0.11156919598579407, + 0.10436765849590302, + 0.09861244261264801, + 0.09467194974422455, + 0.09834607690572739 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197183609008789, + "mean_weights": [ + 0.11082982271909714, + 0.11017665266990662, + 0.11016038060188293, + 0.11044234037399292, + 0.1113046407699585, + 0.11165352165699005, + 0.1125120222568512, + 0.11187836527824402, + 0.11104223132133484 + ], + "sources": 9 + }, + { + "entropy_mean": 2.189943790435791, + "mean_weights": [ + 0.11908848583698273, + 0.13332396745681763, + 0.1237700879573822, + 0.11780860275030136, + 0.11251186579465866, + 0.10542744398117065, + 0.09969248622655869, + 0.0944419801235199, + 0.0939350575208664 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197188138961792, + "mean_weights": [ + 0.11065463721752167, + 0.1101759597659111, + 0.11036042869091034, + 0.11101746559143066, + 0.11014606803655624, + 0.11144031584262848, + 0.11245380342006683, + 0.11143516004085541, + 0.11231614649295807 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1897106170654297, + "mean_weights": [ + 0.11972114443778992, + 0.13409414887428284, + 0.12356151640415192, + 0.11793002486228943, + 0.11148501187562943, + 0.10532795637845993, + 0.10018672049045563, + 0.09460655599832535, + 0.09308692067861557 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197195053100586, + "mean_weights": [ + 0.11115142703056335, + 0.11168935894966125, + 0.11139166355133057, + 0.11114688217639923, + 0.11217151582241058, + 0.11137430369853973, + 0.11164972931146622, + 0.10970751196146011, + 0.10971760749816895 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1886544227600098, + "mean_weights": [ + 0.1198912039399147, + 0.13586071133613586, + 0.12443617731332779, + 0.11758707463741302, + 0.11255505681037903, + 0.104681596159935, + 0.09946994483470917, + 0.09411460161209106, + 0.09140364825725555 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.02857121080160141, + 0.014366969466209412, + 0.015028351917862892, + 0.014920232817530632, + 0.0157560333609581, + 0.015600415877997875, + 0.016723914071917534, + 0.016733111813664436, + 0.017853006720542908, + 0.0120475385338068, + 0.012999274767935276, + 0.01321115531027317, + 0.014046707190573215, + 0.014213940128684044, + 0.015208248049020767, + 0.015383333899080753, + 0.016688203439116478, + 0.012680167332291603, + 0.0136343352496624, + 0.013726864010095596, + 0.014754028059542179, + 0.014983372762799263, + 0.016035236418247223, + 0.016218259930610657, + 0.017354078590869904, + 0.014127783477306366, + 0.015000199899077415, + 0.015246106311678886, + 0.016188079491257668, + 0.01640736125409603, + 0.01740950159728527, + 0.017550179734826088, + 0.018516210839152336, + 0.015639834105968475, + 0.016506247222423553, + 0.016805538907647133, + 0.01769549958407879, + 0.017855772748589516, + 0.01894219033420086, + 0.01905881054699421, + 0.020316828042268753, + 0.017556607723236084, + 0.018616946414113045, + 0.01869107224047184, + 0.019841432571411133, + 0.019845549017190933, + 0.021212533116340637, + 0.021093817427754402, + 0.022626768797636032, + 0.020589889958500862, + 0.021300721913576126, + 0.02186623588204384, + 0.0225946344435215, + 0.023382706567645073, + 0.024155739694833755, + 0.024957869201898575, + 0.02599317952990532, + 0.022718844935297966, + 0.024633634835481644, + 0.02416442520916462, + 0.026326298713684082, + 0.025567933917045593, + 0.027950936928391457, + 0.026888607069849968 + ], + "loss_nats": 3.570533275604248, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.193143367767334, + "mean_weights": [ + 0.10072997957468033, + 0.09430642426013947, + 0.10201297700405121, + 0.10992498695850372, + 0.11447066068649292, + 0.1184505969285965, + 0.11966084688901901, + 0.12088891863822937, + 0.11955460906028748 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.0029148487374186516, + 0.0058675240725278854, + 0.006538186222314835, + 0.010148076340556145, + 0.01062243152409792, + 0.01483201701194048, + 0.015128692612051964, + 0.019581714645028114, + 0.003076757537201047, + 0.006728556007146835, + 0.007619878742843866, + 0.011936783790588379, + 0.012705771252512932, + 0.017140226438641548, + 0.017775695770978928, + 0.023016884922981262, + 0.003750445321202278, + 0.007510852534323931, + 0.009126928634941578, + 0.013793004676699638, + 0.015184049494564533, + 0.02042997069656849, + 0.02182827889919281, + 0.02688039094209671, + 0.005523395258933306, + 0.009191237390041351, + 0.011900501325726509, + 0.016781959682703018, + 0.019034823402762413, + 0.024295387789607048, + 0.02624632976949215, + 0.031403254717588425, + 0.005626704078167677, + 0.00985476840287447, + 0.0137434471398592, + 0.018672063946723938, + 0.02210696041584015, + 0.02787620574235916, + 0.03176325187087059, + 0.03780702129006386, + 0.007807946298271418, + 0.012548788450658321, + 0.01767749898135662, + 0.02336551994085312, + 0.02850686013698578, + 0.03465462848544121, + 0.04043632000684738, + 0.046962957829236984, + 0.01269012875854969, + 0.017280612140893936, + 0.02355561964213848, + 0.02920808270573616, + 0.03787730261683464, + 0.04429623484611511, + 0.05274556949734688, + 0.05986764654517174, + 0.01350546907633543, + 0.019072027876973152, + 0.0314585380256176, + 0.03784291073679924, + 0.050032839179039, + 0.056897345930337906, + 0.06909433007240295, + 0.07636407017707825 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00014032237231731415, + 0.00014269929670263082, + 0.00014183347229845822, + 0.00013509647396858782, + 0.0001350848760921508, + 0.0001232736831298098, + 0.00011216397251700982, + 0.00010401678446214646, + 9.937267168425024e-05, + 9.035648690769449e-05, + 8.297963358927518e-05, + 7.7556207543239e-05, + 7.989619916770607e-05, + 7.340640149777755e-05, + 6.78515643812716e-05, + 6.222894444363192e-05, + 6.128090899437666e-05, + 5.597613926511258e-05, + 5.102259819977917e-05, + 4.699481723946519e-05, + 4.6880406443960965e-05, + 4.325482586864382e-05, + 3.944073978345841e-05, + 3.686518903123215e-05, + 3.118604945484549e-05, + 2.854893682524562e-05, + 2.7016852982342243e-05, + 2.6106961740879342e-05, + 2.809257603075821e-05, + 2.7874853913090192e-05, + 2.773736196104437e-05, + 2.769191087281797e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00012931136643601349, + "first_to_last_ratio": 4.6130013045220375, + "imbalance_abs_log_ratio": 1.5288786873681943, + "last_quartile_mean": 2.803193797262793e-05, + "mean": 7.10659427909377e-05, + "normalized": [ + 1.9745375464885553, + 2.007984290343191, + 1.9958008960171674, + 1.9010016424606593, + 1.9008384436627326, + 1.734637975499139, + 1.5783083726472835, + 1.4636657219639482, + 1.3983163774606562, + 1.2714456933823541, + 1.1676427600965664, + 1.0913273573446343, + 1.1242544041489084, + 1.0329336193248717, + 0.9547690738569081, + 0.8756507266314256, + 0.8623105046907389, + 0.7876647669303929, + 0.7179613214993547, + 0.6612846518861346, + 0.6596747274833753, + 0.6086575956065365, + 0.554987920155868, + 0.5187462177161629, + 0.43883255790454834, + 0.40172459133105554, + 0.3801659686950276, + 0.36736249060511295, + 0.3953029387564892, + 0.3922392755006802, + 0.3903045660372415, + 0.38966500387228004 + ], + "population_cv": 0.5552614744965129 + }, + "activation_output_rms_by_block": [ + 0.02147751860320568, + 0.03838019073009491, + 0.05048823729157448, + 0.05868127942085266, + 0.018261993303894997, + 0.026578109711408615, + 0.032689545303583145, + 0.04116985574364662, + 0.026149341836571693, + 0.029165346175432205, + 0.03554752469062805, + 0.04089665040373802, + 0.023141179233789444, + 0.03429487720131874, + 0.03853890299797058, + 0.04666254296898842, + 0.037595316767692566, + 0.035180509090423584, + 0.0481911227107048, + 0.05348655954003334, + 0.029506860300898552, + 0.03956221789121628, + 0.05266078561544418, + 0.0708107277750969, + 0.022752845659852028, + 0.04251012206077576, + 0.061593133956193924, + 0.08408339321613312, + 0.03222670406103134, + 0.05886174365878105, + 0.0927082747220993, + 0.12734989821910858 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.03596584126353264, + "first_to_last_ratio": 0.5511097145400238, + "imbalance_abs_log_ratio": 0.5958213707033574, + "last_quartile_mean": 0.06526076444424689, + "mean": 0.04535010346444324, + "normalized": [ + 0.47359359654041666, + 0.8463087798727276, + 1.113299274634727, + 1.2939613129408125, + 0.40268912105599314, + 0.5860650292065417, + 0.720826256310824, + 0.9078227522882248, + 0.5766104118610026, + 0.6431153172186101, + 0.7838466061824776, + 0.901798392495467, + 0.5102784220091867, + 0.7562248943534969, + 0.8498084911357925, + 1.028940165606771, + 0.8290017860084749, + 0.7757536676406235, + 1.062646367466153, + 1.179414278116686, + 0.6506459312498243, + 0.8723732664079819, + 1.1612054128329143, + 1.5614237314941535, + 0.5017154079414924, + 0.9373765176546032, + 1.3581696457315922, + 1.8540948485830628, + 0.7106202985027079, + 1.2979406696377578, + 2.04427923289717, + 2.808150114121731 + ], + "population_cv": 0.49805060611268404 + }, + "bits_per_byte": 3.0783718143110845, + "branch_output_rms_by_sublayer": [ + 0.01099894754588604, + 0.01711432635784149, + 0.01347433216869831, + 0.014373406767845154, + 0.013855574652552605, + 0.011604705825448036, + 0.01134195365011692, + 0.01094871573150158, + 0.015810651704669, + 0.008319049142301083, + 0.014106041751801968, + 0.00772661529481411, + 0.014131364412605762, + 0.007499936036765575, + 0.020124683156609535, + 0.008251973427832127, + 0.024223143234848976, + 0.008236965164542198, + 0.015907051041722298, + 0.008312772028148174, + 0.014329465106129646, + 0.007945015095174313, + 0.0158215519040823, + 0.009073426015675068, + 0.020143266767263412, + 0.008672122843563557, + 0.02034893073141575, + 0.009621595032513142, + 0.019871076568961143, + 0.009306521154940128, + 0.0206281915307045, + 0.01009637676179409, + 0.03402981907129288, + 0.01116005890071392, + 0.03202904388308525, + 0.010967749170958996, + 0.024206310510635376, + 0.012425323016941547, + 0.02113247476518154, + 0.013728998601436615, + 0.025038713589310646, + 0.014223441481590271, + 0.017763102427124977, + 0.016583407297730446, + 0.021287458017468452, + 0.01755274273455143, + 0.01132896076887846, + 0.02176096849143505, + 0.014728590846061707, + 0.016626309603452682, + 0.012032261118292809, + 0.0191692803055048, + 0.008020774461328983, + 0.02069026790559292, + 0.007941779680550098, + 0.023783253505825996, + 0.010982516221702099, + 0.02774551324546337, + 0.007478485815227032, + 0.02620389312505722, + 0.009052526205778122, + 0.031721483916044235, + 0.00956410076469183, + 0.031295549124479294 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0004472335087707588, + 0.00042784539914012557, + 0.00042397821782952415, + 0.00038775862355601047, + 0.0004323157128658365, + 0.0003573334406079681, + 0.00027671793109539035, + 0.00029373233863930356, + 0.00037119297165631927, + 0.00028101247110944553, + 0.0003063418412926746, + 0.00026586103897006664, + 0.0002624198572814148, + 0.00023119836984615815, + 0.0001925886330923581, + 0.0001740839523966621, + 0.00019272195127700555, + 0.00020446672647005584, + 0.00018302813520551577, + 0.00015285044026622867, + 0.0001584299612979944, + 0.00014756737641485025, + 0.00014639070573531482, + 0.00012448558518306078, + 0.00011966984297997159, + 0.00010711590740674922, + 9.39984431135086e-05, + 8.769848150961743e-05, + 0.0001072933742449643, + 9.570002003870382e-05, + 0.00011496225598768216, + 0.0001075344633234999 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00038086439656311467, + "first_to_last_ratio": 3.6534947112634804, + "imbalance_abs_log_ratio": 1.2956841646672694, + "last_quartile_mean": 0.00010424659857558713, + "mean": 0.00022736024933139813, + "normalized": [ + 1.9670699257497537, + 1.8817950824662504, + 1.8647860348338092, + 1.7054811678659676, + 1.9014568911546945, + 1.5716618963023845, + 1.217090198964591, + 1.2919247735832753, + 1.6326203579908638, + 1.235978901042831, + 1.3473852276004221, + 1.1693382627433264, + 1.1542028919000442, + 1.0168812293531821, + 0.8470637838351538, + 0.7656745315357172, + 0.8476501580366226, + 0.899307275881929, + 0.8050137864633308, + 0.6722830429493211, + 0.6968234850370366, + 0.6490465103236118, + 0.6438711523487869, + 0.5475257242597925, + 0.5263446153489301, + 0.471128562366318, + 0.41343393750636404, + 0.38572477716537407, + 0.4719091158656081, + 0.42091799388912693, + 0.5056392061750173, + 0.4729694994605618 + ], + "population_cv": 0.5007792018338078 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6579722166061401, + "mean_weights": [ + 0.3871001601219177, + 0.6128998398780823 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6694055199623108, + "mean_weights": [ + 0.43293511867523193, + 0.5670648813247681 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6891770362854004, + "mean_weights": [ + 0.5099499225616455, + 0.4900500774383545 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6754072308540344, + "mean_weights": [ + 0.4642259478569031, + 0.5357741117477417 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6834067106246948, + "mean_weights": [ + 0.46594539284706116, + 0.5340546369552612 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6751099824905396, + "mean_weights": [ + 0.45455044507980347, + 0.5454495549201965 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6826268434524536, + "mean_weights": [ + 0.4582672715187073, + 0.5417327284812927 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6737973690032959, + "mean_weights": [ + 0.443834125995636, + 0.556165874004364 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0911246538162231, + "mean_weights": [ + 0.30305415391921997, + 0.33151721954345703, + 0.3654285967350006 + ], + "sources": 3 + }, + { + "entropy_mean": 1.082936406135559, + "mean_weights": [ + 0.30230170488357544, + 0.31684446334838867, + 0.3808538317680359 + ], + "sources": 3 + }, + { + "entropy_mean": 1.09015691280365, + "mean_weights": [ + 0.30438169836997986, + 0.3149605393409729, + 0.38065776228904724 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0699574947357178, + "mean_weights": [ + 0.29616135358810425, + 0.2899787127971649, + 0.41385990381240845 + ], + "sources": 3 + }, + { + "entropy_mean": 1.084320068359375, + "mean_weights": [ + 0.28653275966644287, + 0.3149440288543701, + 0.3985231816768646 + ], + "sources": 3 + }, + { + "entropy_mean": 1.063678503036499, + "mean_weights": [ + 0.29684972763061523, + 0.28854992985725403, + 0.4146003723144531 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0910385847091675, + "mean_weights": [ + 0.3071846663951874, + 0.3137527108192444, + 0.3790626525878906 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0611608028411865, + "mean_weights": [ + 0.293984979391098, + 0.2840859591960907, + 0.4219290614128113 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3756752014160156, + "mean_weights": [ + 0.22521960735321045, + 0.22203180193901062, + 0.2683175802230835, + 0.2844310402870178 + ], + "sources": 4 + }, + { + "entropy_mean": 1.359569787979126, + "mean_weights": [ + 0.23506423830986023, + 0.2177267074584961, + 0.2517319321632385, + 0.29547712206840515 + ], + "sources": 4 + }, + { + "entropy_mean": 1.377488136291504, + "mean_weights": [ + 0.2321462780237198, + 0.22501498460769653, + 0.25978168845176697, + 0.2830570340156555 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3566675186157227, + "mean_weights": [ + 0.23594214022159576, + 0.21935097873210907, + 0.23278743028640747, + 0.3119194507598877 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3823401927947998, + "mean_weights": [ + 0.2448958158493042, + 0.23326033353805542, + 0.25029197335243225, + 0.2715519070625305 + ], + "sources": 4 + }, + { + "entropy_mean": 1.348025918006897, + "mean_weights": [ + 0.24705132842063904, + 0.21487052738666534, + 0.22015517950057983, + 0.3179229497909546 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3812311887741089, + "mean_weights": [ + 0.2542978525161743, + 0.22274735569953918, + 0.24763193726539612, + 0.2753228545188904 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3406031131744385, + "mean_weights": [ + 0.2467106580734253, + 0.21065476536750793, + 0.21252679824829102, + 0.33010774850845337 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6029343605041504, + "mean_weights": [ + 0.20290890336036682, + 0.18466229736804962, + 0.17943665385246277, + 0.20726214349269867, + 0.22573000192642212 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5750751495361328, + "mean_weights": [ + 0.19683444499969482, + 0.17654260993003845, + 0.16274544596672058, + 0.21074137091636658, + 0.25313615798950195 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6062473058700562, + "mean_weights": [ + 0.1997165083885193, + 0.19476813077926636, + 0.18442964553833008, + 0.1991252899169922, + 0.2219604253768921 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5746448040008545, + "mean_weights": [ + 0.19694097340106964, + 0.17951148748397827, + 0.1657242774963379, + 0.19501730799674988, + 0.2628059387207031 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6034471988677979, + "mean_weights": [ + 0.19473953545093536, + 0.1895807385444641, + 0.1837574541568756, + 0.20168806612491608, + 0.23023423552513123 + ], + "sources": 5 + }, + { + "entropy_mean": 1.572009563446045, + "mean_weights": [ + 0.19747325778007507, + 0.18249398469924927, + 0.1621016263961792, + 0.1875222772359848, + 0.27040886878967285 + ], + "sources": 5 + }, + { + "entropy_mean": 1.590608835220337, + "mean_weights": [ + 0.21454644203186035, + 0.18159104883670807, + 0.16405089199543, + 0.1901196837425232, + 0.2496919482946396 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5680205821990967, + "mean_weights": [ + 0.2022026777267456, + 0.18351733684539795, + 0.16088660061359406, + 0.17795470356941223, + 0.2754386365413666 + ], + "sources": 5 + }, + { + "entropy_mean": 1.784020185470581, + "mean_weights": [ + 0.15995945036411285, + 0.16581295430660248, + 0.15002475678920746, + 0.15491130948066711, + 0.1788797676563263, + 0.1904117465019226 + ], + "sources": 6 + }, + { + "entropy_mean": 1.769648790359497, + "mean_weights": [ + 0.1617799699306488, + 0.16202957928180695, + 0.141836017370224, + 0.14341230690479279, + 0.19649666547775269, + 0.19444546103477478 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7798104286193848, + "mean_weights": [ + 0.15804769098758698, + 0.16410765051841736, + 0.14577797055244446, + 0.14935636520385742, + 0.1779199093580246, + 0.204790398478508 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7738641500473022, + "mean_weights": [ + 0.15458105504512787, + 0.1678159385919571, + 0.15270966291427612, + 0.14449885487556458, + 0.17126679420471191, + 0.20912769436836243 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7794538736343384, + "mean_weights": [ + 0.15359632670879364, + 0.17179939150810242, + 0.1570495367050171, + 0.14789816737174988, + 0.1734778881072998, + 0.19617870450019836 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7795114517211914, + "mean_weights": [ + 0.16449342668056488, + 0.16738544404506683, + 0.15034916996955872, + 0.14670482277870178, + 0.16584224998950958, + 0.205224871635437 + ], + "sources": 6 + }, + { + "entropy_mean": 1.780828595161438, + "mean_weights": [ + 0.1501428186893463, + 0.17618349194526672, + 0.1605343073606491, + 0.1513833999633789, + 0.16873937845230103, + 0.19301661849021912 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7803771495819092, + "mean_weights": [ + 0.16790270805358887, + 0.16918238997459412, + 0.15100756287574768, + 0.1441158652305603, + 0.16803117096424103, + 0.1997603178024292 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9368391036987305, + "mean_weights": [ + 0.12787564098834991, + 0.15315480530261993, + 0.13808783888816833, + 0.1309826821088791, + 0.14271506667137146, + 0.15394961833953857, + 0.1532343327999115 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9413785934448242, + "mean_weights": [ + 0.14784708619117737, + 0.14482080936431885, + 0.1400838941335678, + 0.13297274708747864, + 0.13089412450790405, + 0.1445799469947815, + 0.15880140662193298 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9327088594436646, + "mean_weights": [ + 0.12323280423879623, + 0.16729964315891266, + 0.14944957196712494, + 0.1286679357290268, + 0.13248378038406372, + 0.14697910845279694, + 0.1518871784210205 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9385182857513428, + "mean_weights": [ + 0.14671167731285095, + 0.15210682153701782, + 0.13766315579414368, + 0.1314491331577301, + 0.13046692311763763, + 0.13822834193706512, + 0.1633739471435547 + ], + "sources": 7 + }, + { + "entropy_mean": 1.924707055091858, + "mean_weights": [ + 0.1185823380947113, + 0.17032256722450256, + 0.15510119497776031, + 0.12802176177501678, + 0.1305253505706787, + 0.14411872625350952, + 0.1533280462026596 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9383785724639893, + "mean_weights": [ + 0.14844639599323273, + 0.15185445547103882, + 0.13416922092437744, + 0.13076812028884888, + 0.13055755198001862, + 0.1383575201034546, + 0.16584673523902893 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9237637519836426, + "mean_weights": [ + 0.11661051958799362, + 0.17571279406547546, + 0.16371585428714752, + 0.12955446541309357, + 0.12852978706359863, + 0.14146803319454193, + 0.14440855383872986 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0491421222686768, + "mean_weights": [ + 0.11470498144626617, + 0.1497368961572647, + 0.13490065932273865, + 0.12435469031333923, + 0.11709396541118622, + 0.1388157606124878, + 0.1177397221326828, + 0.10265333950519562 + ], + "sources": 8 + }, + { + "entropy_mean": 2.138361930847168, + "mean_weights": [ + 0.08652530610561371, + 0.16489386558532715, + 0.15517112612724304, + 0.11844676733016968, + 0.11005911231040955, + 0.1110924631357193, + 0.09961430728435516, + 0.07884600758552551, + 0.0753510445356369 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1906728744506836, + "mean_weights": [ + 0.11011575162410736, + 0.11945432424545288, + 0.11334187537431717, + 0.1101863905787468, + 0.10481680184602737, + 0.1116793304681778, + 0.11565238237380981, + 0.10603460669517517, + 0.10871852189302444 + ], + "sources": 9 + }, + { + "entropy_mean": 2.117628574371338, + "mean_weights": [ + 0.08489099144935608, + 0.1764935851097107, + 0.16548463702201843, + 0.11571207642555237, + 0.11045145988464355, + 0.10794807970523834, + 0.10117396712303162, + 0.07742631435394287, + 0.06041890010237694 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1755805015563965, + "mean_weights": [ + 0.10780518501996994, + 0.13283979892730713, + 0.11966764181852341, + 0.11064392328262329, + 0.10227380692958832, + 0.11783622205257416, + 0.11054521799087524, + 0.10119100660085678, + 0.09719720482826233 + ], + "sources": 9 + }, + { + "entropy_mean": 2.09588623046875, + "mean_weights": [ + 0.0810154378414154, + 0.1841772347688675, + 0.17320731282234192, + 0.12209606170654297, + 0.11263981461524963, + 0.10979034006595612, + 0.09270387887954712, + 0.07204672694206238, + 0.05232318490743637 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1623687744140625, + "mean_weights": [ + 0.1018882468342781, + 0.14171633124351501, + 0.11894478648900986, + 0.11747735738754272, + 0.11976975202560425, + 0.12610143423080444, + 0.10464757680892944, + 0.08886905014514923, + 0.08058547228574753 + ], + "sources": 9 + }, + { + "entropy_mean": 2.099823236465454, + "mean_weights": [ + 0.08218918740749359, + 0.18377935886383057, + 0.1668383628129959, + 0.12124596536159515, + 0.11231734603643417, + 0.11366412043571472, + 0.09457215666770935, + 0.07361987978219986, + 0.05177363008260727 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.04389791935682297, + 0.017910990864038467, + 0.024006975814700127, + 0.027713732793927193, + 0.03076089918613434, + 0.03230839595198631, + 0.03558119758963585, + 0.03628998622298241, + 0.03896484524011612, + 0.025513041764497757, + 0.02685926854610443, + 0.02669278159737587, + 0.027765151113271713, + 0.028513295575976372, + 0.029727669432759285, + 0.030201857909560204, + 0.031855203211307526, + 0.022706419229507446, + 0.024480726569890976, + 0.023568065837025642, + 0.025195777416229248, + 0.024511540308594704, + 0.02623848058283329, + 0.025304824113845825, + 0.027537759393453598, + 0.02055380307137966, + 0.022043660283088684, + 0.021111827343702316, + 0.022863954305648804, + 0.021456727758049965, + 0.02333555743098259, + 0.022339651361107826, + 0.024513432756066322, + 0.018822720274329185, + 0.020240623503923416, + 0.019272731617093086, + 0.020459000021219254, + 0.019870083779096603, + 0.021075308322906494, + 0.020815087482333183, + 0.02200455404818058, + 0.01851768232882023, + 0.018790191039443016, + 0.01927866041660309, + 0.019968196749687195, + 0.020534450188279152, + 0.021694926545023918, + 0.02158089354634285, + 0.02334309183061123, + 0.020366568118333817, + 0.021458804607391357, + 0.02146214060485363, + 0.02281521074473858, + 0.022754240781068802, + 0.02429022081196308, + 0.024386556819081306, + 0.025360461324453354, + 0.022437898442149162, + 0.025937557220458984, + 0.02393130585551262, + 0.02763180062174797, + 0.024537326768040657, + 0.028428789228200912, + 0.025934023782610893 + ], + "loss_nats": 2.1337647438049316, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.169145345687866, + "mean_weights": [ + 0.12321731448173523, + 0.09184826165437698, + 0.09223933517932892, + 0.09703588485717773, + 0.09026999771595001, + 0.09880580008029938, + 0.13764077425003052, + 0.12600889801979065, + 0.14293372631072998 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.01099894754588604, + 0.02147751860320568, + 0.029096253216266632, + 0.03838019073009491, + 0.04300357773900032, + 0.05048823729157448, + 0.05208193510770798, + 0.05868127942085266, + 0.015810651704669, + 0.018261993303894997, + 0.023285243660211563, + 0.026578109711408615, + 0.029341328889131546, + 0.032689545303583145, + 0.037220392376184464, + 0.04116985574364662, + 0.024223143234848976, + 0.026149341836571693, + 0.025821728631854057, + 0.029165346175432205, + 0.0317302942276001, + 0.03554752469062805, + 0.036120567470788956, + 0.04089665040373802, + 0.020143266767263412, + 0.023141179233789444, + 0.03015838749706745, + 0.03429487720131874, + 0.034047871828079224, + 0.03853890299797058, + 0.041530173271894455, + 0.04666254296898842, + 0.03402981907129288, + 0.037595316767692566, + 0.030641408637166023, + 0.035180509090423584, + 0.04259808361530304, + 0.0481911227107048, + 0.04585010185837746, + 0.05348655954003334, + 0.025038713589310646, + 0.029506860300898552, + 0.03033880889415741, + 0.03956221789121628, + 0.0403439998626709, + 0.05266078561544418, + 0.053870342671871185, + 0.0708107277750969, + 0.014728590846061707, + 0.022752845659852028, + 0.028656987473368645, + 0.04251012206077576, + 0.044980332255363464, + 0.061593133956193924, + 0.06434623152017593, + 0.08408339321613312, + 0.010982516221702099, + 0.03222670406103134, + 0.03523196280002594, + 0.05886174365878105, + 0.06332802772521973, + 0.0927082747220993, + 0.0982721745967865, + 0.12734989821910858 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.992713653948158e-05, + 8.917884406400844e-05, + 8.380923100048676e-05, + 7.761221058899537e-05, + 7.656603702344e-05, + 7.363042095676064e-05, + 7.035063026705757e-05, + 6.534677231684327e-05, + 6.520663009723648e-05, + 6.236798799363896e-05, + 6.0153666709084064e-05, + 5.832234091940336e-05, + 6.603607471333817e-05, + 6.202541408129036e-05, + 5.885191058041528e-05, + 5.6633467465871945e-05, + 6.183729419717565e-05, + 5.828957728226669e-05, + 6.290266901487485e-05, + 6.699833465972915e-05, + 0.00011958443064941093, + 9.639097697800025e-05, + 7.382153125945479e-05, + 5.686042641173117e-05, + 4.306148548494093e-05, + 4.093023744644597e-05, + 3.9059948903741315e-05, + 3.786685920204036e-05, + 4.648702815757133e-05, + 3.8292448152787983e-05, + 3.712073157657869e-05, + 3.6008652386954054e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 7.70526603446342e-05, + "first_to_last_ratio": 1.9334012683862178, + "imbalance_abs_log_ratio": 0.6592807670873676, + "last_quartile_mean": 3.985342391388258e-05, + "mean": 6.317285647128301e-05, + "normalized": [ + 1.2652132736124524, + 1.4116639494455532, + 1.3266652116417212, + 1.2285689602191119, + 1.2120084685144044, + 1.1655388891624905, + 1.1136211689119584, + 1.034412182177459, + 1.0321937892246171, + 0.9872592673087385, + 0.9522074838649823, + 0.923218359548383, + 1.0453235519491937, + 0.9818364649932483, + 0.9316012266624046, + 0.8964841963670943, + 0.9788586055988385, + 0.9226997248219074, + 0.9957230451257022, + 1.0605557260211136, + 1.8929717180633012, + 1.525829008884812, + 1.1685640856371982, + 0.9000768619284877, + 0.6816453757242358, + 0.6479086071571254, + 0.6183027186921191, + 0.5994166057577878, + 0.7358702891439349, + 0.6061535015469007, + 0.5876057162850149, + 0.5700019660077077 + ], + "population_cv": 0.2929422816884334 + }, + "activation_output_rms_by_block": [ + 0.08211566507816315, + 0.1792730838060379, + 0.211857408285141, + 0.220450758934021, + 0.07067389786243439, + 0.08581872284412384, + 0.09455942362546921, + 0.1087210401892662, + 0.03784036636352539, + 0.061385419219732285, + 0.07992809265851974, + 0.10070297867059708, + 0.052349984645843506, + 0.08056230843067169, + 0.10181442648172379, + 0.12473750859498978, + 0.04709841310977936, + 0.0759245902299881, + 0.09948468208312988, + 0.12497665733098984, + 0.0473700612783432, + 0.07518213987350464, + 0.11003360897302628, + 0.15435311198234558, + 0.08784668892621994, + 0.122419573366642, + 0.1544882357120514, + 0.1959162950515747, + 0.06041311100125313, + 0.11818350851535797, + 0.19397477805614471, + 0.271341472864151 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.13168375007808208, + "first_to_last_ratio": 0.8745511271251216, + "imbalance_abs_log_ratio": 0.13404452182408486, + "last_quartile_mean": 0.15057295793667436, + "mean": 0.1134936879388988, + "normalized": [ + 0.7235262733058024, + 1.5795863810730384, + 1.86668890695664, + 1.9424054582903703, + 0.622712145017993, + 0.7561541474476163, + 0.8331690100367241, + 0.9579479014533211, + 0.3334138404586639, + 0.5408707773491345, + 0.7042514355648603, + 0.8873002587140545, + 0.4612589968353745, + 0.7098395504959161, + 0.8970932950609312, + 1.0990700087404357, + 0.41498707077996766, + 0.6689763246645343, + 0.8765657710998792, + 1.1011771632469383, + 0.41738057982436566, + 0.6624345480250865, + 0.9695130272994977, + 1.3600149469584963, + 0.7740226837417924, + 1.078646536118807, + 1.3612055306126158, + 1.7262307588158512, + 0.5323037086765346, + 1.0413223031309373, + 1.7091239308442796, + 2.3908067293595407 + ], + "population_cv": 0.49583241879714957 + }, + "bits_per_byte": 2.082092828197317, + "branch_output_rms_by_sublayer": [ + 0.04474508762359619, + 0.06587300449609756, + 0.11403985321521759, + 0.025835111737251282, + 0.05185890570282936, + 0.0093452837318182, + 0.028878280892968178, + 0.015730490908026695, + 0.0686812773346901, + 0.010891503654420376, + 0.021933114156126976, + 0.008122298866510391, + 0.02270776964724064, + 0.010885748080909252, + 0.03175743296742439, + 0.01868068240582943, + 0.03290550038218498, + 0.016094334423542023, + 0.025583041831851006, + 0.016362514346837997, + 0.02299588918685913, + 0.01966964453458786, + 0.0283373836427927, + 0.023146003484725952, + 0.045610349625349045, + 0.019498733803629875, + 0.03369466960430145, + 0.024007661268115044, + 0.024997565895318985, + 0.026509301736950874, + 0.025535905733704567, + 0.028978457674384117, + 0.03624705225229263, + 0.025891011580824852, + 0.03498883917927742, + 0.027694009244441986, + 0.032704923301935196, + 0.024267304688692093, + 0.037640441209077835, + 0.030298134312033653, + 0.03696197643876076, + 0.02701757103204727, + 0.031886376440525055, + 0.03578891605138779, + 0.04192064702510834, + 0.04901382327079773, + 0.04307623952627182, + 0.0664195567369461, + 0.06881473958492279, + 0.04626939445734024, + 0.03165752813220024, + 0.048946719616651535, + 0.022332612425088882, + 0.05671551451086998, + 0.014488600194454193, + 0.0774463340640068, + 0.018940383568406105, + 0.056074850261211395, + 0.011767655611038208, + 0.06862709671258926, + 0.013134603388607502, + 0.09076401591300964, + 0.015714097768068314, + 0.08702827244997025 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0002495691607128966, + 0.00021240167225128356, + 0.00014140046018161473, + 0.00013619327335749612, + 0.00015135584077317685, + 9.069026139108003e-05, + 0.0001052842865895884, + 0.0001530877937722825, + 0.00012502418193388078, + 0.00011099494215183414, + 0.00011979337091067383, + 0.00013387665677100084, + 0.0001549905694825217, + 0.00015368851888774519, + 0.0001502065077119804, + 0.00016519100012277706, + 0.00016374885070466274, + 0.00018997919926291845, + 0.00018457276255654968, + 0.00024019728245558452, + 0.0004537561922402595, + 0.0004691816118193811, + 0.0004430776618984136, + 0.0003935913777510191, + 0.00022084757766276802, + 0.00018095033893162916, + 0.00017396371286821383, + 0.0001763517726535442, + 0.0002200809259294041, + 0.00019820591407077653, + 0.0001973006272208746, + 0.00019304640106072995 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00015499784362867734, + "first_to_last_ratio": 0.794480165077119, + "imbalance_abs_log_ratio": 0.23006725860697416, + "last_quartile_mean": 0.00019509340879974256, + "mean": 0.00020164377206526756, + "normalized": [ + 1.237673537629105, + 1.0533510163781994, + 0.7012389162004299, + 0.6754152234040406, + 0.7506100447485498, + 0.44975483478718914, + 0.5221301184323721, + 0.7591992165407986, + 0.6200250107075623, + 0.5504506338827443, + 0.5940841598217048, + 0.6639265641572504, + 0.7686355392734605, + 0.7621783569789582, + 0.7449102254611758, + 0.8192219299949837, + 0.8120699639146847, + 0.9421525758872754, + 0.9153407549666727, + 1.191196137601701, + 2.250286173447444, + 2.3267845419372417, + 2.197328771230283, + 1.951914377120571, + 1.0952362941875766, + 0.897376284317175, + 0.8627279240338039, + 0.8745708872995248, + 1.0914342837138002, + 0.9829508347355341, + 0.978461299350286, + 0.9573635678579012 + ], + "population_cv": 0.4856726798113577 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.4470241665840149, + "mean_weights": [ + 0.8263151049613953, + 0.17368490993976593 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6415067911148071, + "mean_weights": [ + 0.5125402212142944, + 0.48745977878570557 + ], + "sources": 2 + }, + { + "entropy_mean": 0.502849817276001, + "mean_weights": [ + 0.790879487991333, + 0.20912054181098938 + ], + "sources": 2 + }, + { + "entropy_mean": 0.689715564250946, + "mean_weights": [ + 0.49880802631378174, + 0.5011919736862183 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6313446760177612, + "mean_weights": [ + 0.6256753206253052, + 0.37432461977005005 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6904382705688477, + "mean_weights": [ + 0.49753934144973755, + 0.5024607181549072 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6193819046020508, + "mean_weights": [ + 0.6637097597122192, + 0.33629029989242554 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6863826513290405, + "mean_weights": [ + 0.5121387243270874, + 0.4878612756729126 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0556713342666626, + "mean_weights": [ + 0.42627638578414917, + 0.25716903805732727, + 0.31655457615852356 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0944163799285889, + "mean_weights": [ + 0.35974377393722534, + 0.31755340099334717, + 0.3227027952671051 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0598217248916626, + "mean_weights": [ + 0.376791387796402, + 0.2430613934993744, + 0.38014721870422363 + ], + "sources": 3 + }, + { + "entropy_mean": 1.090745210647583, + "mean_weights": [ + 0.3455575704574585, + 0.2835947871208191, + 0.37084758281707764 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0520071983337402, + "mean_weights": [ + 0.32410189509391785, + 0.26038146018981934, + 0.4155166745185852 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0808686017990112, + "mean_weights": [ + 0.3871895968914032, + 0.25268471240997314, + 0.36012572050094604 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0486629009246826, + "mean_weights": [ + 0.4070909321308136, + 0.21351183950901031, + 0.3793972432613373 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0926542282104492, + "mean_weights": [ + 0.368913471698761, + 0.2951905131340027, + 0.33589601516723633 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3314954042434692, + "mean_weights": [ + 0.2802572250366211, + 0.1488448679447174, + 0.23578360676765442, + 0.3351143002510071 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3797216415405273, + "mean_weights": [ + 0.28753793239593506, + 0.23653249442577362, + 0.22328856587409973, + 0.252640962600708 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3570446968078613, + "mean_weights": [ + 0.2796326279640198, + 0.1950460821390152, + 0.2361772358417511, + 0.28914403915405273 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3725279569625854, + "mean_weights": [ + 0.26074060797691345, + 0.2200281322002411, + 0.2141101211309433, + 0.30512115359306335 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3559216260910034, + "mean_weights": [ + 0.3055962920188904, + 0.2065197229385376, + 0.21821975708007812, + 0.2696642279624939 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3629136085510254, + "mean_weights": [ + 0.2978547513484955, + 0.20104330778121948, + 0.20380084216594696, + 0.29730111360549927 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3432574272155762, + "mean_weights": [ + 0.35359156131744385, + 0.1976601779460907, + 0.18546369671821594, + 0.2632845640182495 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3645089864730835, + "mean_weights": [ + 0.24801740050315857, + 0.21878156065940857, + 0.21089327335357666, + 0.3223077654838562 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5993901491165161, + "mean_weights": [ + 0.221097931265831, + 0.17356757819652557, + 0.201701819896698, + 0.2097943127155304, + 0.19383837282657623 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5945298671722412, + "mean_weights": [ + 0.20218440890312195, + 0.182874396443367, + 0.15146246552467346, + 0.2149074375629425, + 0.24857130646705627 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6036434173583984, + "mean_weights": [ + 0.21030640602111816, + 0.18881945312023163, + 0.18687176704406738, + 0.1896277368068695, + 0.2243746668100357 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5864531993865967, + "mean_weights": [ + 0.20628216862678528, + 0.1875663697719574, + 0.14435511827468872, + 0.19175346195697784, + 0.27004286646842957 + ], + "sources": 5 + }, + { + "entropy_mean": 1.578747272491455, + "mean_weights": [ + 0.1874420940876007, + 0.17765584588050842, + 0.18386343121528625, + 0.19846069812774658, + 0.25257793068885803 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5874240398406982, + "mean_weights": [ + 0.1995781809091568, + 0.1994006484746933, + 0.14679205417633057, + 0.18447858095169067, + 0.26975053548812866 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5716018676757812, + "mean_weights": [ + 0.23373252153396606, + 0.1665368676185608, + 0.1456945538520813, + 0.17969666421413422, + 0.27433937788009644 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5717716217041016, + "mean_weights": [ + 0.20210981369018555, + 0.19559699296951294, + 0.13711118698120117, + 0.17601484060287476, + 0.2891671061515808 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7721970081329346, + "mean_weights": [ + 0.16085217893123627, + 0.13547661900520325, + 0.15191225707530975, + 0.159015953540802, + 0.2241518795490265, + 0.16859111189842224 + ], + "sources": 6 + }, + { + "entropy_mean": 1.760238528251648, + "mean_weights": [ + 0.17539653182029724, + 0.16050413250923157, + 0.10674943029880524, + 0.1337609589099884, + 0.19425225257873535, + 0.229336678981781 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7718207836151123, + "mean_weights": [ + 0.1517525017261505, + 0.140568807721138, + 0.14039498567581177, + 0.15412259101867676, + 0.19506502151489258, + 0.2180960774421692 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7590861320495605, + "mean_weights": [ + 0.1593845784664154, + 0.16133753955364227, + 0.11901883780956268, + 0.13751879334449768, + 0.17641760408878326, + 0.24632267653942108 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7548754215240479, + "mean_weights": [ + 0.14345301687717438, + 0.13339507579803467, + 0.15310579538345337, + 0.14481791853904724, + 0.17683108150959015, + 0.2483971118927002 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7640254497528076, + "mean_weights": [ + 0.1701541244983673, + 0.17921733856201172, + 0.1149544045329094, + 0.13397127389907837, + 0.1662806123495102, + 0.2354222536087036 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7433195114135742, + "mean_weights": [ + 0.14727026224136353, + 0.13531836867332458, + 0.14743109047412872, + 0.13777202367782593, + 0.1678760051727295, + 0.26433226466178894 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7791519165039062, + "mean_weights": [ + 0.17231981456279755, + 0.17694923281669617, + 0.1283765733242035, + 0.1468508392572403, + 0.17394042015075684, + 0.20156311988830566 + ], + "sources": 6 + }, + { + "entropy_mean": 1.8095097541809082, + "mean_weights": [ + 0.10068220645189285, + 0.09515271335840225, + 0.1109427735209465, + 0.10793591290712357, + 0.11995180696249008, + 0.1291356384754181, + 0.33619895577430725 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9032983779907227, + "mean_weights": [ + 0.14098325371742249, + 0.14394652843475342, + 0.10495204478502274, + 0.10883566737174988, + 0.12474660575389862, + 0.13783156871795654, + 0.23870430886745453 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7488462924957275, + "mean_weights": [ + 0.09754721075296402, + 0.1006798967719078, + 0.10256344079971313, + 0.09797176718711853, + 0.10534974932670593, + 0.10952232778072357, + 0.3863656222820282 + ], + "sources": 7 + }, + { + "entropy_mean": 1.913016676902771, + "mean_weights": [ + 0.14555314183235168, + 0.15097737312316895, + 0.0965791791677475, + 0.11318434774875641, + 0.13583506643772125, + 0.14158430695533752, + 0.21628659963607788 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7430827617645264, + "mean_weights": [ + 0.10184907913208008, + 0.10196703672409058, + 0.10727791488170624, + 0.09979915618896484, + 0.10172094404697418, + 0.10092388838529587, + 0.38646200299263 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9263007640838623, + "mean_weights": [ + 0.14746519923210144, + 0.1593504250049591, + 0.10653319954872131, + 0.11790864169597626, + 0.13947832584381104, + 0.1390301138162613, + 0.19023407995700836 + ], + "sources": 7 + }, + { + "entropy_mean": 1.795186996459961, + "mean_weights": [ + 0.10676933825016022, + 0.11987081170082092, + 0.11137323081493378, + 0.10783538967370987, + 0.10585478693246841, + 0.10099773108959198, + 0.3472987115383148 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0090737342834473, + "mean_weights": [ + 0.10895542055368423, + 0.1293213963508606, + 0.12196331471204758, + 0.1217975988984108, + 0.10339051485061646, + 0.09512487053871155, + 0.17939026653766632, + 0.14005661010742188 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1119484901428223, + "mean_weights": [ + 0.08712957799434662, + 0.10547330230474472, + 0.08657413721084595, + 0.0845983624458313, + 0.08441256731748581, + 0.08474937081336975, + 0.21037259697914124, + 0.1462702453136444, + 0.1104198470711708 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1433091163635254, + "mean_weights": [ + 0.09100314974784851, + 0.10062295198440552, + 0.1095413789153099, + 0.10181419551372528, + 0.09032904356718063, + 0.07545670866966248, + 0.16484607756137848, + 0.1340474784374237, + 0.13233903050422668 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0714099407196045, + "mean_weights": [ + 0.08840608596801758, + 0.10295165330171585, + 0.08186539262533188, + 0.08558160811662674, + 0.08149823546409607, + 0.062403563410043716, + 0.19065919518470764, + 0.1486435979604721, + 0.15799067914485931 + ], + "sources": 9 + }, + { + "entropy_mean": 2.117767810821533, + "mean_weights": [ + 0.09704384952783585, + 0.09514244645833969, + 0.10763552784919739, + 0.10719326138496399, + 0.08750765025615692, + 0.07916541397571564, + 0.16374100744724274, + 0.1349412202835083, + 0.12762963771820068 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0721163749694824, + "mean_weights": [ + 0.07688745856285095, + 0.1265699863433838, + 0.11175613105297089, + 0.1130957081913948, + 0.09152831882238388, + 0.08462467789649963, + 0.19756469130516052, + 0.12186874449253082, + 0.07610426098108292 + ], + "sources": 9 + }, + { + "entropy_mean": 2.142956495285034, + "mean_weights": [ + 0.10645000636577606, + 0.12463926523923874, + 0.1075366735458374, + 0.12534354627132416, + 0.11135344207286835, + 0.09434133023023605, + 0.12007111310958862, + 0.10086819529533386, + 0.10939642041921616 + ], + "sources": 9 + }, + { + "entropy_mean": 2.110433578491211, + "mean_weights": [ + 0.08767393231391907, + 0.11861253529787064, + 0.11157490313053131, + 0.10369857400655746, + 0.09385119378566742, + 0.08841145783662796, + 0.18325303494930267, + 0.12253503501415253, + 0.09038933366537094 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.055996399372816086, + 0.04779405519366264, + 0.05638476461172104, + 0.06255476176738739, + 0.09986891597509384, + 0.09359770268201828, + 0.11572936177253723, + 0.08922671526670456, + 0.11640548706054688, + 0.07563301920890808, + 0.08695963025093079, + 0.07699443399906158, + 0.08520372956991196, + 0.08319501578807831, + 0.08013332635164261, + 0.07385323941707611, + 0.0863872542977333, + 0.050723277032375336, + 0.06699021905660629, + 0.0597233884036541, + 0.06351321190595627, + 0.061697762459516525, + 0.06176900118589401, + 0.059726178646087646, + 0.06771399825811386, + 0.052387818694114685, + 0.052762869745492935, + 0.05403366684913635, + 0.05481218546628952, + 0.054102130234241486, + 0.05716872587800026, + 0.053420521318912506, + 0.06145264580845833, + 0.047365475445985794, + 0.04863803833723068, + 0.04772668331861496, + 0.05118095502257347, + 0.049068622291088104, + 0.05436447635293007, + 0.05214037373661995, + 0.05560959875583649, + 0.03437337279319763, + 0.04204346239566803, + 0.03716471791267395, + 0.04484523460268974, + 0.04414026439189911, + 0.046946510672569275, + 0.04997780919075012, + 0.04656887426972389, + 0.03895776346325874, + 0.03996332362294197, + 0.03980288282036781, + 0.04153545945882797, + 0.041592076420784, + 0.044184572994709015, + 0.044165194034576416, + 0.05373477563261986, + 0.051981449127197266, + 0.0502699576318264, + 0.05537257716059685, + 0.057096291333436966, + 0.05461600050330162, + 0.058509763330221176, + 0.05886553227901459 + ], + "loss_nats": 1.4431967735290527, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.9460878372192383, + "mean_weights": [ + 0.11653168499469757, + 0.08696287870407104, + 0.07659222185611725, + 0.05975610390305519, + 0.05996435508131981, + 0.05836858972907066, + 0.08710259944200516, + 0.14973820745944977, + 0.30498337745666504 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.04474508762359619, + 0.08211566507816315, + 0.168130025267601, + 0.1792730838060379, + 0.20899084210395813, + 0.211857408285141, + 0.21545885503292084, + 0.220450758934021, + 0.0686812773346901, + 0.07067389786243439, + 0.08430273830890656, + 0.08581872284412384, + 0.09278582781553268, + 0.09455942362546921, + 0.104335255920887, + 0.1087210401892662, + 0.03290550038218498, + 0.03784036636352539, + 0.054665278643369675, + 0.061385419219732285, + 0.07034551352262497, + 0.07992809265851974, + 0.08893903344869614, + 0.10070297867059708, + 0.045610349625349045, + 0.052349984645843506, + 0.06921831518411636, + 0.08056230843067169, + 0.0885518491268158, + 0.10181442648172379, + 0.1097707524895668, + 0.12473750859498978, + 0.03624705225229263, + 0.04709841310977936, + 0.06292294710874557, + 0.0759245902299881, + 0.08810343593358994, + 0.09948468208312988, + 0.11294817179441452, + 0.12497665733098984, + 0.03696197643876076, + 0.0473700612783432, + 0.06089227274060249, + 0.07518213987350464, + 0.09078802168369293, + 0.11003360897302628, + 0.12545886635780334, + 0.15435311198234558, + 0.06881473958492279, + 0.08784668892621994, + 0.09990327060222626, + 0.122419573366642, + 0.12555573880672455, + 0.1544882357120514, + 0.15512077510356903, + 0.1959162950515747, + 0.018940383568406105, + 0.06041311100125313, + 0.062181342393159866, + 0.11818350851535797, + 0.12034129351377487, + 0.19397477805614471, + 0.19681726396083832, + 0.271341472864151 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.283424929482862e-05, + 8.091042400337756e-05, + 7.604472921229899e-05, + 7.153357000788674e-05, + 7.648907194379717e-05, + 7.206430746009573e-05, + 6.806159944972023e-05, + 6.28807392786257e-05, + 6.066179776098579e-05, + 5.745151429437101e-05, + 5.5302389228018e-05, + 5.339186463970691e-05, + 5.882261029910296e-05, + 5.5242526286747307e-05, + 5.2224102546460927e-05, + 5.005050843465142e-05, + 5.959395639365539e-05, + 5.497936581377871e-05, + 5.621434320346452e-05, + 5.750052150688134e-05, + 0.00011040258686989546, + 8.961222192738205e-05, + 6.908736395416781e-05, + 5.273316492093727e-05, + 4.1642044379841536e-05, + 3.95435381506104e-05, + 3.777490564971231e-05, + 3.668162753456272e-05, + 5.148682612343691e-05, + 3.890264270012267e-05, + 3.686407580971718e-05, + 3.4813441743608564e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 7.260233633132884e-05, + "first_to_last_ratio": 1.8281462093054863, + "imbalance_abs_log_ratio": 0.6033024530421626, + "last_quartile_mean": 3.971363776145154e-05, + "mean": 5.911870721320156e-05, + "normalized": [ + 1.2320000339683392, + 1.3686094946492635, + 1.2863056855769293, + 1.209998888336869, + 1.2938217960002465, + 1.2189763757892755, + 1.1512700912803735, + 1.0636352221278622, + 1.0261015610882582, + 0.9717992324693755, + 0.9354465250496656, + 0.9031297732401055, + 0.9949914853004012, + 0.9344339362416119, + 0.8833769378299087, + 0.846610333581091, + 1.008038896702188, + 0.9299825453811935, + 0.950872335566085, + 0.9726281953276734, + 1.8674729552482823, + 1.5158014468112575, + 1.1686210205006005, + 0.8919877887512675, + 0.7043801588838299, + 0.6688836751453877, + 0.6389670449572508, + 0.6204741149409218, + 0.8709058189949661, + 0.6580428519829926, + 0.6235602493263792, + 0.5888735289501479 + ], + "population_cv": 0.2804013834416186 + }, + "activation_output_rms_by_block": [ + 0.12014345079660416, + 0.26731228828430176, + 0.3029491901397705, + 0.31159067153930664, + 0.06965517997741699, + 0.08030477166175842, + 0.0959664061665535, + 0.11720705777406693, + 0.04052923247218132, + 0.06900312006473541, + 0.09020354598760605, + 0.11894716322422028, + 0.04706801846623421, + 0.08083708584308624, + 0.11319102346897125, + 0.14463475346565247, + 0.05436912551522255, + 0.09273304045200348, + 0.1164766401052475, + 0.14852507412433624, + 0.0572793073952198, + 0.09725138545036316, + 0.14356882870197296, + 0.20402640104293823, + 0.10730373114347458, + 0.15423046052455902, + 0.1994396299123764, + 0.24681825935840607, + 0.07070667296648026, + 0.14180640876293182, + 0.22800633311271667, + 0.3292829096317291 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.17064112704247236, + "first_to_last_ratio": 0.9238861566740402, + "imbalance_abs_log_ratio": 0.07916642199379362, + "last_quartile_mean": 0.18469930067658424, + "mean": 0.13941772398538888, + "normalized": [ + 0.8617516292920928, + 1.9173479572246952, + 2.1729603774877284, + 2.234943037600884, + 0.49961495558998603, + 0.5760011666104551, + 0.6883372029270172, + 0.8406897948380677, + 0.2907035871309219, + 0.4949379325111276, + 0.6470019980892779, + 0.8531710303683201, + 0.33760426666531285, + 0.5798192907779649, + 0.8118840290409115, + 1.0374201308924706, + 0.3899728381803198, + 0.6651452756589399, + 0.8354507359297758, + 1.0653241917785277, + 0.4108466682559151, + 0.6975539599294793, + 1.0297745838758574, + 1.4634179587117662, + 0.7696563110922692, + 1.1062471550656108, + 1.4305184750633133, + 1.7703506577418584, + 0.5071569879729954, + 1.0171332934526551, + 1.6354185579490021, + 2.361843962294481 + ], + "population_cv": 0.5696823710981592 + }, + "bits_per_byte": 1.9488798823532094, + "branch_output_rms_by_sublayer": [ + 0.07303544133901596, + 0.09185322374105453, + 0.1779898852109909, + 0.022334590554237366, + 0.062374215573072433, + 0.007825793698430061, + 0.03649044781923294, + 0.018411319702863693, + 0.06719767302274704, + 0.011600266210734844, + 0.023174433037638664, + 0.007791830692440271, + 0.02815386839210987, + 0.012416418641805649, + 0.04434986039996147, + 0.02558070607483387, + 0.03437936678528786, + 0.021278981119394302, + 0.036810629069805145, + 0.020222196355462074, + 0.02944403700530529, + 0.024546733126044273, + 0.028734680265188217, + 0.03155883029103279, + 0.035293661057949066, + 0.024987563490867615, + 0.033774156123399734, + 0.030630556866526604, + 0.030397579073905945, + 0.03747265785932541, + 0.032739195972681046, + 0.04104550555348396, + 0.035251349210739136, + 0.036599673330783844, + 0.04529772326350212, + 0.03992510959506035, + 0.0332656130194664, + 0.035177286714315414, + 0.04443706199526787, + 0.043413370847702026, + 0.042124342173337936, + 0.036167487502098083, + 0.04090939089655876, + 0.05162384733557701, + 0.051316387951374054, + 0.07156099379062653, + 0.056411903351545334, + 0.09908116608858109, + 0.08093558251857758, + 0.06232227385044098, + 0.04824332147836685, + 0.06585292518138885, + 0.04406072571873665, + 0.07637913525104523, + 0.021837742999196053, + 0.09651224315166473, + 0.018632294610142708, + 0.06743000447750092, + 0.013925542123615742, + 0.08935486525297165, + 0.016003893688321114, + 0.10868491232395172, + 0.026686565950512886, + 0.11891410499811172 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.000262948161177326, + 0.00019323142799372565, + 0.00013525288752647198, + 0.00013819830489748063, + 0.00010828169751714791, + 7.170504275896834e-05, + 9.093051582807932e-05, + 0.00015553897073798643, + 0.00012394064620165063, + 0.00010900634493461773, + 0.00011368409114400034, + 0.00012945667145370724, + 0.0001177555017844673, + 0.00014306275804877984, + 0.00016152074126719868, + 0.00017119258041381125, + 0.00018413287193100227, + 0.00023154746582235088, + 0.000204144960288782, + 0.00025644736828085534, + 0.0005031558098617796, + 0.0005438972365887106, + 0.0005236852348503333, + 0.00046668486096972675, + 0.0002557702843491047, + 0.00022233808045469506, + 0.0002139560818005556, + 0.00021215358161764752, + 0.000282871258408856, + 0.0002467690520817826, + 0.00022896051552603945, + 0.00022810043195383136 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00014451087605464827, + "first_to_last_ratio": 0.6113888714758638, + "imbalance_abs_log_ratio": 0.49202207139883514, + "last_quartile_mean": 0.00023636491077406405, + "mean": 0.00021969754495223352, + "normalized": [ + 1.1968643583818652, + 0.8795338520316096, + 0.6156322209056936, + 0.6290389131454682, + 0.4928671257600554, + 0.3263807193410304, + 0.41388954003946365, + 0.7079686337496561, + 0.5641421538351633, + 0.49616551226513617, + 0.5174572668470986, + 0.5892495133791942, + 0.535989156410781, + 0.6511805039964577, + 0.7351959317629899, + 0.7792193601937236, + 0.8381198432191689, + 1.0539374296271438, + 0.9292091103392487, + 1.1672746199171773, + 2.29022044816739, + 2.475663698048899, + 2.3836644827514686, + 2.1242151844309096, + 1.1641927287112566, + 1.0120189577131398, + 0.9738665120134764, + 0.965662049904899, + 1.2875485635051482, + 1.1232217098076127, + 1.0421623763516332, + 1.0382475234460393 + ], + "population_cv": 0.5588051969510979 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.2486211508512497, + "mean_weights": [ + 0.928449273109436, + 0.07155071198940277 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6078275442123413, + "mean_weights": [ + 0.5862155556678772, + 0.4137844443321228 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5342782735824585, + "mean_weights": [ + 0.7634186148643494, + 0.23658135533332825 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6855527758598328, + "mean_weights": [ + 0.4585249423980713, + 0.5414750576019287 + ], + "sources": 2 + }, + { + "entropy_mean": 0.658619225025177, + "mean_weights": [ + 0.5537886023521423, + 0.44621139764785767 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6893052458763123, + "mean_weights": [ + 0.47804930806159973, + 0.5219507217407227 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6224605441093445, + "mean_weights": [ + 0.6530391573905945, + 0.3469608426094055 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6836730241775513, + "mean_weights": [ + 0.5152404308319092, + 0.4847595691680908 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0568022727966309, + "mean_weights": [ + 0.3889986574649811, + 0.24523048102855682, + 0.3657708764076233 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0907280445098877, + "mean_weights": [ + 0.35903704166412354, + 0.2848784923553467, + 0.3560844361782074 + ], + "sources": 3 + }, + { + "entropy_mean": 1.055159330368042, + "mean_weights": [ + 0.33855703473091125, + 0.25161677598953247, + 0.40982621908187866 + ], + "sources": 3 + }, + { + "entropy_mean": 1.084274172782898, + "mean_weights": [ + 0.3707119822502136, + 0.2698661684989929, + 0.35942181944847107 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0361523628234863, + "mean_weights": [ + 0.2864622473716736, + 0.2566634714603424, + 0.4568743109703064 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0700488090515137, + "mean_weights": [ + 0.40097421407699585, + 0.2287827581167221, + 0.37024301290512085 + ], + "sources": 3 + }, + { + "entropy_mean": 1.03178870677948, + "mean_weights": [ + 0.42037367820739746, + 0.1924310177564621, + 0.38719528913497925 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0842034816741943, + "mean_weights": [ + 0.39326632022857666, + 0.27339667081832886, + 0.33333703875541687 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3209505081176758, + "mean_weights": [ + 0.30210015177726746, + 0.13647639751434326, + 0.24396686255931854, + 0.31745660305023193 + ], + "sources": 4 + }, + { + "entropy_mean": 1.371370553970337, + "mean_weights": [ + 0.3008589744567871, + 0.22207970917224884, + 0.20789846777915955, + 0.2691628336906433 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3458349704742432, + "mean_weights": [ + 0.2885082960128784, + 0.18893417716026306, + 0.23869511485099792, + 0.2838624119758606 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3656761646270752, + "mean_weights": [ + 0.28106409311294556, + 0.20955704152584076, + 0.20403917133808136, + 0.3053396940231323 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3414140939712524, + "mean_weights": [ + 0.30585163831710815, + 0.20411336421966553, + 0.2163928598165512, + 0.2736421823501587 + ], + "sources": 4 + }, + { + "entropy_mean": 1.365321397781372, + "mean_weights": [ + 0.2955209016799927, + 0.19335070252418518, + 0.21881619095802307, + 0.2923122048377991 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3175382614135742, + "mean_weights": [ + 0.37765079736709595, + 0.17676019668579102, + 0.17624405026435852, + 0.26934492588043213 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3654873371124268, + "mean_weights": [ + 0.24775154888629913, + 0.2333429455757141, + 0.19744190573692322, + 0.32146361470222473 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5920979976654053, + "mean_weights": [ + 0.2174428254365921, + 0.17040714621543884, + 0.2146083265542984, + 0.20569837093353271, + 0.19184333086013794 + ], + "sources": 5 + }, + { + "entropy_mean": 1.587863802909851, + "mean_weights": [ + 0.20315341651439667, + 0.18720076978206635, + 0.14404666423797607, + 0.19615545868873596, + 0.26944369077682495 + ], + "sources": 5 + }, + { + "entropy_mean": 1.599494457244873, + "mean_weights": [ + 0.2126145362854004, + 0.18835213780403137, + 0.192312091588974, + 0.1875757873058319, + 0.21914544701576233 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5893950462341309, + "mean_weights": [ + 0.21140219271183014, + 0.19927793741226196, + 0.14736294746398926, + 0.18148277699947357, + 0.26047417521476746 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5570417642593384, + "mean_weights": [ + 0.1875864714384079, + 0.15970240533351898, + 0.19339191913604736, + 0.19908206164836884, + 0.2602371573448181 + ], + "sources": 5 + }, + { + "entropy_mean": 1.580305814743042, + "mean_weights": [ + 0.20317688584327698, + 0.19076292216777802, + 0.14548362791538239, + 0.17790120840072632, + 0.2826753556728363 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5573816299438477, + "mean_weights": [ + 0.2441616654396057, + 0.1415461301803589, + 0.15834765136241913, + 0.18082576990127563, + 0.27511876821517944 + ], + "sources": 5 + }, + { + "entropy_mean": 1.576451301574707, + "mean_weights": [ + 0.20345619320869446, + 0.1890067458152771, + 0.1430710405111313, + 0.17605465650558472, + 0.28841137886047363 + ], + "sources": 5 + }, + { + "entropy_mean": 1.759494423866272, + "mean_weights": [ + 0.15768522024154663, + 0.11713434010744095, + 0.14977338910102844, + 0.1549873948097229, + 0.2221473753452301, + 0.19827227294445038 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7580336332321167, + "mean_weights": [ + 0.17079707980155945, + 0.1651972085237503, + 0.10748225450515747, + 0.13355788588523865, + 0.18136337399482727, + 0.24160218238830566 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7561116218566895, + "mean_weights": [ + 0.14564168453216553, + 0.11846636235713959, + 0.1588398814201355, + 0.1484883576631546, + 0.18241314589977264, + 0.24615056812763214 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7579470872879028, + "mean_weights": [ + 0.16142059862613678, + 0.1640883982181549, + 0.11881326138973236, + 0.1345720887184143, + 0.1706210970878601, + 0.25048452615737915 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7134730815887451, + "mean_weights": [ + 0.13741520047187805, + 0.10915715247392654, + 0.1542142629623413, + 0.13406038284301758, + 0.16351932287216187, + 0.30163368582725525 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7681612968444824, + "mean_weights": [ + 0.175822451710701, + 0.19316132366657257, + 0.1212742030620575, + 0.13059334456920624, + 0.15986239910125732, + 0.21928629279136658 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7127671241760254, + "mean_weights": [ + 0.14985418319702148, + 0.11401912569999695, + 0.155704066157341, + 0.1306324154138565, + 0.15071439743041992, + 0.29907578229904175 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7775979042053223, + "mean_weights": [ + 0.1765751987695694, + 0.19438962638378143, + 0.13620537519454956, + 0.14063861966133118, + 0.1579708456993103, + 0.19422034919261932 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7804237604141235, + "mean_weights": [ + 0.10079847276210785, + 0.08881710469722748, + 0.10964279621839523, + 0.1017041951417923, + 0.10778775811195374, + 0.13341641426086426, + 0.35783326625823975 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8986074924468994, + "mean_weights": [ + 0.14076843857765198, + 0.15517404675483704, + 0.10426843166351318, + 0.1073455661535263, + 0.11623178422451019, + 0.1344693899154663, + 0.241742342710495 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7529070377349854, + "mean_weights": [ + 0.10170947015285492, + 0.09542374312877655, + 0.1095866709947586, + 0.10136895626783371, + 0.09919941425323486, + 0.11099758744239807, + 0.3817141652107239 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9109292030334473, + "mean_weights": [ + 0.14686265587806702, + 0.16674570739269257, + 0.10305891931056976, + 0.11107458174228668, + 0.12119877338409424, + 0.13405174016952515, + 0.2170076072216034 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7260308265686035, + "mean_weights": [ + 0.10430073738098145, + 0.09887559711933136, + 0.10967855155467987, + 0.10041452944278717, + 0.0915672704577446, + 0.09773178398609161, + 0.39743152260780334 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9246466159820557, + "mean_weights": [ + 0.14914585649967194, + 0.16779503226280212, + 0.10872989147901535, + 0.11952568590641022, + 0.1287989616394043, + 0.1340678334236145, + 0.19193677604198456 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7794851064682007, + "mean_weights": [ + 0.11010485887527466, + 0.11871978640556335, + 0.1121491938829422, + 0.10657300055027008, + 0.09759172797203064, + 0.09646660089492798, + 0.3583948314189911 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0037269592285156, + "mean_weights": [ + 0.10686184465885162, + 0.1148708164691925, + 0.11906561255455017, + 0.14062100648880005, + 0.11285024881362915, + 0.09308953583240509, + 0.17511320114135742, + 0.1375277191400528 + ], + "sources": 8 + }, + { + "entropy_mean": 2.088080644607544, + "mean_weights": [ + 0.09118194878101349, + 0.10171566158533096, + 0.08584169298410416, + 0.08255404233932495, + 0.0748370960354805, + 0.07946085929870605, + 0.20189812779426575, + 0.16033506393432617, + 0.12217549979686737 + ], + "sources": 9 + }, + { + "entropy_mean": 2.14388108253479, + "mean_weights": [ + 0.08876891434192657, + 0.09881433099508286, + 0.1152849867939949, + 0.10974849760532379, + 0.09509578347206116, + 0.08069240301847458, + 0.14142301678657532, + 0.12444892525672913, + 0.1457231342792511 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0033082962036133, + "mean_weights": [ + 0.08977866917848587, + 0.10048449039459229, + 0.08331224322319031, + 0.07561526447534561, + 0.06440272182226181, + 0.05276993662118912, + 0.16990166902542114, + 0.15739065408706665, + 0.2063443660736084 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1180944442749023, + "mean_weights": [ + 0.09887091815471649, + 0.09425252676010132, + 0.10716164112091064, + 0.11986196041107178, + 0.09374598413705826, + 0.07890886068344116, + 0.14312618970870972, + 0.12647047638893127, + 0.13760145008563995 + ], + "sources": 9 + }, + { + "entropy_mean": 2.041163682937622, + "mean_weights": [ + 0.0787363275885582, + 0.124203622341156, + 0.11193635314702988, + 0.10525508970022202, + 0.08614891022443771, + 0.07642446458339691, + 0.19110478460788727, + 0.13441255688667297, + 0.09177789092063904 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1350901126861572, + "mean_weights": [ + 0.10361722111701965, + 0.1331605315208435, + 0.11075668036937714, + 0.1406192183494568, + 0.11478176712989807, + 0.08772425353527069, + 0.11043667048215866, + 0.09601088613271713, + 0.10289278626441956 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0970618724823, + "mean_weights": [ + 0.09285374730825424, + 0.11225572228431702, + 0.11032570898532867, + 0.10395291447639465, + 0.08953994512557983, + 0.07913549989461899, + 0.16822844743728638, + 0.12680400907993317, + 0.11690403521060944 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05897562578320503, + 0.05551674962043762, + 0.06972260028123856, + 0.08269861340522766, + 0.15173132717609406, + 0.14499513804912567, + 0.16659259796142578, + 0.12118377536535263, + 0.15876378118991852, + 0.0937858521938324, + 0.10591574013233185, + 0.0980178564786911, + 0.1032194197177887, + 0.10331054031848907, + 0.09310676157474518, + 0.08528745174407959, + 0.10316919535398483, + 0.06011173129081726, + 0.08162962645292282, + 0.0737733468413353, + 0.0777570977807045, + 0.07729758322238922, + 0.07581526786088943, + 0.0697576031088829, + 0.08668693155050278, + 0.06468445807695389, + 0.06558235734701157, + 0.06788372248411179, + 0.0696391686797142, + 0.0635380893945694, + 0.06916190683841705, + 0.06121518835425377, + 0.07393106818199158, + 0.05319856479763985, + 0.058962129056453705, + 0.053640544414520264, + 0.06194892153143883, + 0.05605520308017731, + 0.06798876821994781, + 0.060800258070230484, + 0.06959682703018188, + 0.04032549634575844, + 0.053642891347408295, + 0.04607857018709183, + 0.05766084045171738, + 0.05640692636370659, + 0.05972195416688919, + 0.0657767653465271, + 0.05811592563986778, + 0.04918687790632248, + 0.05036557465791702, + 0.050394799560308456, + 0.05232074484229088, + 0.052619028836488724, + 0.055546555668115616, + 0.05557428300380707, + 0.06694450229406357, + 0.06750545650720596, + 0.05963582918047905, + 0.07127027958631516, + 0.06790094077587128, + 0.07101515680551529, + 0.06738729029893875, + 0.07538487017154694 + ], + "loss_nats": 1.350860595703125, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.8663020133972168, + "mean_weights": [ + 0.10718511044979095, + 0.08038097620010376, + 0.07475601136684418, + 0.06260354816913605, + 0.054793961346149445, + 0.05229249596595764, + 0.07308901846408844, + 0.13577046990394592, + 0.3591284155845642 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.07303544133901596, + 0.12014345079660416, + 0.2593322694301605, + 0.26731228828430176, + 0.30099040269851685, + 0.3029491901397705, + 0.30594387650489807, + 0.31159067153930664, + 0.06719767302274704, + 0.06965517997741699, + 0.07899806648492813, + 0.08030477166175842, + 0.0930912122130394, + 0.0959664061665535, + 0.11039920151233673, + 0.11720705777406693, + 0.03437936678528786, + 0.04052923247218132, + 0.06091400235891342, + 0.06900312006473541, + 0.07843481004238129, + 0.09020354598760605, + 0.10192256420850754, + 0.11894716322422028, + 0.035293661057949066, + 0.04706801846623421, + 0.06407224386930466, + 0.08083708584308624, + 0.09281916916370392, + 0.11319102346897125, + 0.12173866480588913, + 0.14463475346565247, + 0.035251349210739136, + 0.05436912551522255, + 0.07238258421421051, + 0.09273304045200348, + 0.09960390627384186, + 0.1164766401052475, + 0.13001231849193573, + 0.14852507412433624, + 0.042124342173337936, + 0.0572793073952198, + 0.07516071945428848, + 0.09725138545036316, + 0.11489575356245041, + 0.14356882870197296, + 0.16218890249729156, + 0.20402640104293823, + 0.08093558251857758, + 0.10730373114347458, + 0.12586736679077148, + 0.15423046052455902, + 0.1639469563961029, + 0.1994396299123764, + 0.20092174410820007, + 0.24681825935840607, + 0.018632294610142708, + 0.07070667296648026, + 0.07257922738790512, + 0.14180640876293182, + 0.1437942385673523, + 0.22800633311271667, + 0.23150524497032166, + 0.3292829096317291 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00011619397264439613, + 0.00012333222548477352, + 0.00011758109030779451, + 0.00011363717931089923, + 0.0001299128489335999, + 0.0001243110600626096, + 0.0001189774411614053, + 0.00010918003681581467, + 0.00010130138252861798, + 9.645402315072715e-05, + 9.342828707303852e-05, + 9.032322122948244e-05, + 0.00010702834697440267, + 9.990156104322523e-05, + 9.298718941863626e-05, + 8.801308285910636e-05, + 0.00010807957005454227, + 9.860304271569476e-05, + 9.78035168373026e-05, + 9.825862798606977e-05, + 0.00018890350474976003, + 0.00014797764015384018, + 0.00011057221854571253, + 8.234690903918818e-05, + 6.956196739338338e-05, + 6.582088826689869e-05, + 6.234115426195785e-05, + 6.020446744514629e-05, + 9.865802712738514e-05, + 6.298683729255572e-05, + 5.776801481260918e-05, + 4.9892361857928336e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.0001191407318401616, + "first_to_last_ratio": 1.807786227157747, + "imbalance_abs_log_ratio": 0.5921030177879053, + "last_quartile_mean": 6.590421480723307e-05, + "mean": 9.944817804807826e-05, + "normalized": [ + 1.168387143183479, + 1.2401657617613517, + 1.182335288746566, + 1.1426773374969352, + 1.3063371444652658, + 1.2500084214967857, + 1.1963762785466578, + 1.0978585928746885, + 1.0186348761426658, + 0.9698923101848725, + 0.9394670561774453, + 0.9082441026301723, + 1.0762223006504932, + 1.0045589874449765, + 0.935031603833723, + 0.8850145330621985, + 1.0867928621305778, + 0.9915017514752766, + 0.9834621282857434, + 0.988038493159388, + 1.899516999280113, + 1.4879874428901083, + 1.1118576600996788, + 0.8280383879871337, + 0.6994795556712323, + 0.6618611779401082, + 0.6268707530450582, + 0.6053853235605826, + 0.992054646588788, + 0.6333634112643542, + 0.5808856023956639, + 0.501692065527916 + ], + "population_cv": 0.28152213355296973 + }, + "activation_output_rms_by_block": [ + 0.10557802766561508, + 0.2583724558353424, + 0.2926461398601532, + 0.30053097009658813, + 0.05294578894972801, + 0.06118258833885193, + 0.07357313483953476, + 0.09621669352054596, + 0.03472397103905678, + 0.05730196461081505, + 0.0760834813117981, + 0.10359182953834534, + 0.03599332273006439, + 0.06543196737766266, + 0.09347184002399445, + 0.12118600308895111, + 0.04543427750468254, + 0.07841403037309647, + 0.09941539168357849, + 0.1277458220720291, + 0.05195997282862663, + 0.08666642010211945, + 0.12814365327358246, + 0.1823730319738388, + 0.09818319976329803, + 0.13811686635017395, + 0.17807537317276, + 0.21280638873577118, + 0.06062556430697441, + 0.11879219114780426, + 0.18227574229240417, + 0.26055294275283813 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.15513072488829494, + "first_to_last_ratio": 0.9932909558500862, + "imbalance_abs_log_ratio": 0.006731650956727549, + "last_quartile_mean": 0.15617853356525302, + "mean": 0.12120034522376955, + "normalized": [ + 0.8711033575910092, + 2.1317798671143633, + 2.4145652334557877, + 2.479621403237131, + 0.43684519917806636, + 0.5048053966009078, + 0.6070373372592156, + 0.793864847025833, + 0.28650059515050613, + 0.4727871402100356, + 0.6277497079016304, + 0.8547156309421893, + 0.2969737640896213, + 0.5398661798929454, + 0.7712176054566464, + 0.999881665891364, + 0.3748692086709673, + 0.6469786057813809, + 0.8202566708867869, + 1.0540054307285578, + 0.4287114259674267, + 0.7150674360053111, + 1.057287857035338, + 1.5047237005565246, + 0.81009009984274, + 1.1395748592561499, + 1.4692645705256313, + 1.7558232886455183, + 0.500209501837992, + 0.9801307985425358, + 1.5039209826991207, + 2.1497706320207666 + ], + "population_cv": 0.6083135214979585 + }, + "bits_per_byte": 1.8723727121862856, + "branch_output_rms_by_sublayer": [ + 0.06663628667593002, + 0.07813488692045212, + 0.18265415728092194, + 0.015101124532520771, + 0.05425833910703659, + 0.005600288510322571, + 0.027296584099531174, + 0.015474065206944942, + 0.050542060285806656, + 0.009944790974259377, + 0.018502239137887955, + 0.006146863102912903, + 0.020613152533769608, + 0.01066078245639801, + 0.03884607553482056, + 0.02266029268503189, + 0.028015729039907455, + 0.02038472890853882, + 0.030238129198551178, + 0.018571803346276283, + 0.027104107663035393, + 0.020757213234901428, + 0.02866903506219387, + 0.02825833484530449, + 0.02432532235980034, + 0.02241888828575611, + 0.029286012053489685, + 0.026882564648985863, + 0.029147984459996223, + 0.03393060341477394, + 0.02996431477367878, + 0.03828607499599457, + 0.02687123790383339, + 0.03350448235869408, + 0.03787726163864136, + 0.03721447288990021, + 0.027634277939796448, + 0.036467213183641434, + 0.038205601274967194, + 0.0434262715280056, + 0.036424826830625534, + 0.034937553107738495, + 0.0337483286857605, + 0.05015077814459801, + 0.04375720024108887, + 0.06956327706575394, + 0.04833395406603813, + 0.09568649530410767, + 0.07171578705310822, + 0.05999118089675903, + 0.040132876485586166, + 0.06315042078495026, + 0.04608408361673355, + 0.07199651747941971, + 0.02082001604139805, + 0.08451130986213684, + 0.014797233045101166, + 0.058196358382701874, + 0.012043262831866741, + 0.07848073542118073, + 0.014600119553506374, + 0.08618523925542831, + 0.022803297266364098, + 0.0989394560456276 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0004279988409014525, + 0.00030808063626583686, + 0.0001806343667006249, + 0.00020144247345266352, + 0.00017050965270525774, + 0.00011023688453293689, + 0.00015109864356334743, + 0.00025970775667622664, + 0.00021734514639344736, + 0.00017358496891950413, + 0.0001783908335717012, + 0.0002272173281351143, + 0.00020348488583523262, + 0.00025358104524733936, + 0.00029675220758047777, + 0.00030740988719747404, + 0.0003330143170025419, + 0.0004184534883930539, + 0.00037109203540948237, + 0.0004362348599212854, + 0.0008706016893848469, + 0.0008754435002526395, + 0.0008262916126492521, + 0.0007217509020821496, + 0.00043242198735303706, + 0.00037070641830158017, + 0.0003590069334527622, + 0.00034142393043198465, + 0.0004963438444126418, + 0.00039013983967290734, + 0.00034460516176607807, + 0.0003326965333098998 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00022621365684979332, + "first_to_last_ratio": 0.5899921469746839, + "imbalance_abs_log_ratio": 0.5276460523833537, + "last_quartile_mean": 0.0003834180810876114, + "mean": 0.0003621157066085869, + "normalized": [ + 1.1819394549601216, + 0.85077954544193, + 0.4988305213059253, + 0.5562931123294355, + 0.4708706349751426, + 0.30442447682086493, + 0.4172661964278475, + 0.7171955039137431, + 0.6002091111402055, + 0.47936326911976007, + 0.49263489629441826, + 0.6274716174648424, + 0.5619333326935213, + 0.7002762946193789, + 0.8194955429018136, + 0.8489272395183766, + 0.9196351081299523, + 1.155579503336382, + 1.0247885651936055, + 1.204683618964958, + 2.4042085816672008, + 2.41757947605104, + 2.2818441662968123, + 1.993149948787763, + 1.1941541873532833, + 1.0237236649397232, + 0.991414972896536, + 0.9428586614748304, + 1.3706774805798274, + 1.077389995940198, + 0.9516437853345144, + 0.9187575331260446 + ], + "population_cv": 0.55138786215429 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.25018399953842163, + "mean_weights": [ + 0.9283580780029297, + 0.0716419517993927 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5915409922599792, + "mean_weights": [ + 0.6339905261993408, + 0.3660094738006592 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6034965515136719, + "mean_weights": [ + 0.6926443576812744, + 0.3073556125164032 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6828886270523071, + "mean_weights": [ + 0.43748968839645386, + 0.5625102519989014 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6703654527664185, + "mean_weights": [ + 0.4876942038536072, + 0.5123057961463928 + ], + "sources": 2 + }, + { + "entropy_mean": 0.685522198677063, + "mean_weights": [ + 0.45093637704849243, + 0.5490636825561523 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6487185955047607, + "mean_weights": [ + 0.6082445383071899, + 0.39175549149513245 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6892790794372559, + "mean_weights": [ + 0.5081504583358765, + 0.4918496012687683 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0589048862457275, + "mean_weights": [ + 0.3457357883453369, + 0.2460753470659256, + 0.4081888794898987 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0905814170837402, + "mean_weights": [ + 0.3282802104949951, + 0.28823256492614746, + 0.3834872245788574 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0286166667938232, + "mean_weights": [ + 0.2793591618537903, + 0.23621734976768494, + 0.4844234883785248 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0793647766113281, + "mean_weights": [ + 0.34295910596847534, + 0.2595300078392029, + 0.3975108861923218 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0191105604171753, + "mean_weights": [ + 0.2575778365135193, + 0.24408063292503357, + 0.49834150075912476 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0723689794540405, + "mean_weights": [ + 0.37144023180007935, + 0.2328961193561554, + 0.39566367864608765 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0167454481124878, + "mean_weights": [ + 0.3652345538139343, + 0.17551980912685394, + 0.45924562215805054 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0840563774108887, + "mean_weights": [ + 0.3736764192581177, + 0.2635895609855652, + 0.3627340495586395 + ], + "sources": 3 + }, + { + "entropy_mean": 1.2764405012130737, + "mean_weights": [ + 0.23919621109962463, + 0.10453687608242035, + 0.263206422328949, + 0.39306050539016724 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3740202188491821, + "mean_weights": [ + 0.28299662470817566, + 0.21803641319274902, + 0.21747693419456482, + 0.2814900279045105 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3281174898147583, + "mean_weights": [ + 0.24697959423065186, + 0.15579551458358765, + 0.2719123363494873, + 0.3253125250339508 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3666620254516602, + "mean_weights": [ + 0.2708823084831238, + 0.20532462000846863, + 0.21383404731750488, + 0.3099590241909027 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3429303169250488, + "mean_weights": [ + 0.27752381563186646, + 0.19432586431503296, + 0.22970175743103027, + 0.2984485328197479 + ], + "sources": 4 + }, + { + "entropy_mean": 1.367378830909729, + "mean_weights": [ + 0.2777973413467407, + 0.19115662574768066, + 0.22952966392040253, + 0.3015163540840149 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3322054147720337, + "mean_weights": [ + 0.3412310481071472, + 0.16932155191898346, + 0.19953060150146484, + 0.2899167835712433 + ], + "sources": 4 + }, + { + "entropy_mean": 1.366931676864624, + "mean_weights": [ + 0.24345502257347107, + 0.22604259848594666, + 0.20761674642562866, + 0.3228856325149536 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5882664918899536, + "mean_weights": [ + 0.19564303755760193, + 0.15773959457874298, + 0.22847658395767212, + 0.20264220237731934, + 0.21549859642982483 + ], + "sources": 5 + }, + { + "entropy_mean": 1.584667682647705, + "mean_weights": [ + 0.19586648046970367, + 0.18009021878242493, + 0.14858797192573547, + 0.19375723600387573, + 0.2816981077194214 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5927224159240723, + "mean_weights": [ + 0.19985643029212952, + 0.17414313554763794, + 0.199208602309227, + 0.18960987031459808, + 0.23718197643756866 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5873792171478271, + "mean_weights": [ + 0.19913329184055328, + 0.18933847546577454, + 0.15693499147891998, + 0.17933908104896545, + 0.27525416016578674 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5420968532562256, + "mean_weights": [ + 0.1735360026359558, + 0.14197081327438354, + 0.19048985838890076, + 0.19546270370483398, + 0.2985406219959259 + ], + "sources": 5 + }, + { + "entropy_mean": 1.582078456878662, + "mean_weights": [ + 0.19881561398506165, + 0.18186192214488983, + 0.15247397124767303, + 0.18165400624275208, + 0.2851945161819458 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5412237644195557, + "mean_weights": [ + 0.22861704230308533, + 0.12470002472400665, + 0.16313378512859344, + 0.17281487584114075, + 0.31073427200317383 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5812281370162964, + "mean_weights": [ + 0.2022017240524292, + 0.1931133270263672, + 0.14693862199783325, + 0.1750599443912506, + 0.28268641233444214 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7531124353408813, + "mean_weights": [ + 0.15169863402843475, + 0.10659359395503998, + 0.1599212884902954, + 0.15024638175964355, + 0.22934028506278992, + 0.20219983160495758 + ], + "sources": 6 + }, + { + "entropy_mean": 1.762786626815796, + "mean_weights": [ + 0.16621586680412292, + 0.16845844686031342, + 0.11499074101448059, + 0.13360491394996643, + 0.1790938377380371, + 0.23763619363307953 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7369294166564941, + "mean_weights": [ + 0.14021016657352448, + 0.10459877550601959, + 0.1626015305519104, + 0.14244785904884338, + 0.178192600607872, + 0.27194905281066895 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7612131834030151, + "mean_weights": [ + 0.16026079654693604, + 0.1729058474302292, + 0.12150783836841583, + 0.13213862478733063, + 0.16643309593200684, + 0.24675379693508148 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6882884502410889, + "mean_weights": [ + 0.1315440833568573, + 0.09433871507644653, + 0.158755823969841, + 0.12785425782203674, + 0.16308915615081787, + 0.32441794872283936 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7684848308563232, + "mean_weights": [ + 0.17402957379817963, + 0.19325366616249084, + 0.12219862639904022, + 0.13086363673210144, + 0.15930381417274475, + 0.22035068273544312 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7019174098968506, + "mean_weights": [ + 0.14592596888542175, + 0.10422790795564651, + 0.15637663006782532, + 0.12806911766529083, + 0.15361696481704712, + 0.31178343296051025 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7769746780395508, + "mean_weights": [ + 0.1752081662416458, + 0.20116111636161804, + 0.13720004260540009, + 0.13742423057556152, + 0.15936335921287537, + 0.18964309990406036 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7729153633117676, + "mean_weights": [ + 0.09848621487617493, + 0.08230466395616531, + 0.11203177273273468, + 0.09956170618534088, + 0.1094764992594719, + 0.1334107518196106, + 0.3647283911705017 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8972569704055786, + "mean_weights": [ + 0.13790717720985413, + 0.1571238934993744, + 0.10740794241428375, + 0.10486835986375809, + 0.11720770597457886, + 0.13083387911319733, + 0.24465104937553406 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7305114269256592, + "mean_weights": [ + 0.09764835238456726, + 0.08818334341049194, + 0.10980058461427689, + 0.09766717255115509, + 0.09809298813343048, + 0.10970605164766312, + 0.3989015221595764 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9102001190185547, + "mean_weights": [ + 0.14414381980895996, + 0.17332738637924194, + 0.10535766184329987, + 0.10710084438323975, + 0.1217031478881836, + 0.13322804868221283, + 0.21513909101486206 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7222957611083984, + "mean_weights": [ + 0.10251537710428238, + 0.09414640069007874, + 0.11381721496582031, + 0.09696625173091888, + 0.0944550484418869, + 0.09727997332811356, + 0.4008197486400604 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9257829189300537, + "mean_weights": [ + 0.14694491028785706, + 0.17354407906532288, + 0.11239725351333618, + 0.11700987815856934, + 0.1277998387813568, + 0.13376536965370178, + 0.18853867053985596 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7845284938812256, + "mean_weights": [ + 0.10971026867628098, + 0.11362971365451813, + 0.12193788588047028, + 0.10297370702028275, + 0.10046073794364929, + 0.09683416038751602, + 0.35445353388786316 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0175135135650635, + "mean_weights": [ + 0.10853733122348785, + 0.12205591797828674, + 0.123784139752388, + 0.14330759644508362, + 0.11578401923179626, + 0.09900149703025818, + 0.156728595495224, + 0.13080091774463654 + ], + "sources": 8 + }, + { + "entropy_mean": 2.050873279571533, + "mean_weights": [ + 0.08464042842388153, + 0.0938267931342125, + 0.08138334006071091, + 0.07390965521335602, + 0.06850367039442062, + 0.07166063785552979, + 0.17788532376289368, + 0.1618492603302002, + 0.18634089827537537 + ], + "sources": 9 + }, + { + "entropy_mean": 2.145954132080078, + "mean_weights": [ + 0.08773529529571533, + 0.09906956553459167, + 0.12252385914325714, + 0.11031387746334076, + 0.09715825319290161, + 0.08549074828624725, + 0.12741495668888092, + 0.11914648860692978, + 0.15114694833755493 + ], + "sources": 9 + }, + { + "entropy_mean": 1.9562722444534302, + "mean_weights": [ + 0.08620478957891464, + 0.08938613533973694, + 0.08283404260873795, + 0.07130077481269836, + 0.061856623739004135, + 0.04858341068029404, + 0.14417162537574768, + 0.16443759202957153, + 0.25122499465942383 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1256468296051025, + "mean_weights": [ + 0.09738725423812866, + 0.10046981275081635, + 0.10905814915895462, + 0.12306356430053711, + 0.1002630665898323, + 0.08269863575696945, + 0.12526756525039673, + 0.11717700213193893, + 0.14461494982242584 + ], + "sources": 9 + }, + { + "entropy_mean": 2.028853416442871, + "mean_weights": [ + 0.0789184421300888, + 0.12036088109016418, + 0.10662862658500671, + 0.10067631304264069, + 0.08165188878774643, + 0.07559703290462494, + 0.1780017763376236, + 0.14643099904060364, + 0.1117340624332428 + ], + "sources": 9 + }, + { + "entropy_mean": 2.13504695892334, + "mean_weights": [ + 0.10072587430477142, + 0.13353405892848969, + 0.11221307516098022, + 0.15188650786876678, + 0.12113417685031891, + 0.08907364308834076, + 0.1011936143040657, + 0.08830933272838593, + 0.10192970931529999 + ], + "sources": 9 + }, + { + "entropy_mean": 2.078139066696167, + "mean_weights": [ + 0.09158287942409515, + 0.10898827761411667, + 0.10562451183795929, + 0.10203785449266434, + 0.08194266259670258, + 0.07498477399349213, + 0.1501031368970871, + 0.1326877474784851, + 0.15204817056655884 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06189367175102234, + 0.058095674961805344, + 0.06266746670007706, + 0.09409566223621368, + 0.1521945744752884, + 0.15536712110042572, + 0.16864776611328125, + 0.12795083224773407, + 0.15569773316383362, + 0.0865585058927536, + 0.09908005595207214, + 0.08596222847700119, + 0.09255138039588928, + 0.0898032858967781, + 0.08522933721542358, + 0.07401556521654129, + 0.09253095835447311, + 0.045857783406972885, + 0.07370869070291519, + 0.058967601507902145, + 0.06945785135030746, + 0.06690461188554764, + 0.06695349514484406, + 0.06054195389151573, + 0.07494638860225677, + 0.05348192900419235, + 0.056552816182374954, + 0.05649816617369652, + 0.059508051723241806, + 0.051248304545879364, + 0.05918953940272331, + 0.050500787794589996, + 0.064420685172081, + 0.043548379093408585, + 0.05189858376979828, + 0.0426921509206295, + 0.05426602065563202, + 0.04550497233867645, + 0.05912991985678673, + 0.05038738250732422, + 0.06151469424366951, + 0.032932329922914505, + 0.046855196356773376, + 0.03875989466905594, + 0.05171490088105202, + 0.04862629249691963, + 0.052897412329912186, + 0.05667373910546303, + 0.049510348588228226, + 0.04177618771791458, + 0.04272947832942009, + 0.04272143170237541, + 0.044300299137830734, + 0.04471025615930557, + 0.04706704989075661, + 0.047076087445020676, + 0.05500439926981926, + 0.05498846247792244, + 0.04848027601838112, + 0.06034955754876137, + 0.0545390322804451, + 0.061857983469963074, + 0.05428297072649002, + 0.06515993922948837 + ], + "loss_nats": 1.2978298664093018, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.7709457874298096, + "mean_weights": [ + 0.10222859680652618, + 0.07361805438995361, + 0.06582196056842804, + 0.05730090290307999, + 0.052203960716724396, + 0.048417992889881134, + 0.06012570112943649, + 0.12975022196769714, + 0.4105325937271118 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.06663628667593002, + 0.10557802766561508, + 0.2537385821342468, + 0.2583724558353424, + 0.291115939617157, + 0.2926461398601532, + 0.29593369364738464, + 0.30053097009658813, + 0.050542060285806656, + 0.05294578894972801, + 0.06017574667930603, + 0.06118258833885193, + 0.07116389274597168, + 0.07357313483953476, + 0.08982721716165543, + 0.09621669352054596, + 0.028015729039907455, + 0.03472397103905678, + 0.04916425794363022, + 0.05730196461081505, + 0.06544128060340881, + 0.0760834813117981, + 0.08768708258867264, + 0.10359182953834534, + 0.02432532235980034, + 0.03599332273006439, + 0.050393667072057724, + 0.06543196737766266, + 0.07533413171768188, + 0.09347184002399445, + 0.10066040605306625, + 0.12118600308895111, + 0.02687123790383339, + 0.04543427750468254, + 0.05995847284793854, + 0.07841403037309647, + 0.08243945240974426, + 0.09941539168357849, + 0.10896863043308258, + 0.1277458220720291, + 0.036424826830625534, + 0.05195997282862663, + 0.06485821306705475, + 0.08666642010211945, + 0.10029250383377075, + 0.12814365327358246, + 0.14264756441116333, + 0.1823730319738388, + 0.07171578705310822, + 0.09818319976329803, + 0.1120700091123581, + 0.13811686635017395, + 0.14860299229621887, + 0.17807537317276, + 0.17915822565555573, + 0.21280638873577118, + 0.014797233045101166, + 0.06062556430697441, + 0.06211893633008003, + 0.11879219114780426, + 0.12019115686416626, + 0.18227574229240417, + 0.18443551659584045, + 0.26055294275283813 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 7.9557801867581395, + "cross_entropy_nats": 5.514526605606079, + "step": 0 + }, + { + "bits_per_byte": 5.165502143339161, + "cross_entropy_nats": 3.580453246831894, + "step": 100 + }, + { + "bits_per_byte": 3.1006408158754044, + "cross_entropy_nats": 2.149200439453125, + "step": 500 + }, + { + "bits_per_byte": 1.9501257031782282, + "cross_entropy_nats": 1.3517241328954697, + "step": 2000 + }, + { + "bits_per_byte": 1.8007407342233697, + "cross_entropy_nats": 1.2481783628463745, + "step": 4000 + }, + { + "bits_per_byte": 1.709346788793424, + "cross_entropy_nats": 1.184828907251358, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.48.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.48.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "48": true, + "49": true, + "50": true, + "51": true, + "52": true, + "53": true, + "54": true, + "55": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_group_7_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "9bcea28df3f22bfe9fea1a0888ecbca65fbafedd5eefd0f4437614caf7792ec5", + "final_model_state": "d236caf6629d85f38213439dc396bd42cd31619a35ee846b17e4cb09b248a418", + "final_optimizer_state": "e519141adaf7b2618ffecdf90697f87a1e5f57985b27543fdddb591448445c7e", + "final_public_parameters": "ba39b8b6d689c3e96d11dd0cda3b2101c4ec0aa3e8a6ed3ab0554397311814b7", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "73bbe569e1a47981386ebaf59ca891174746bf81897fe5403676ad6d50423c58" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "65136111a29a042e61a7909132560d95cd4bcf0f9b52f64d0fb2e57773856434", + "1": "d995676b4e7dec8f661cd8c2345fe7fc7a513c17f528c02fc946a441a6995a94", + "7999": "2345e7ac3decca2bdaebf13094fdc92fcabef42e3e461a2fefd6e8e7e76baccc" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "54d7b082e9d5ad89e06fecdfc4c2fca45d09fa46389b463f82f04cccd075c592", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073001, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 212.6544312078803, + "measured_steps": 7980, + "median_ms": 213.40229449560866, + "p95_ms": 214.22775209503016, + "peak_allocated_bytes": 11621543936, + "peak_reserved_bytes": 12083789824, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 7.958777844362678, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.516604423522949, + "step": 1, + "unclipped_grad_norm": 21.322534561157227 + }, + { + "bits_per_byte": 7.247921265620827, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.023876190185547, + "step": 10, + "unclipped_grad_norm": 11.153740882873535 + }, + { + "bits_per_byte": 6.733664246455684, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.667420387268066, + "step": 20, + "unclipped_grad_norm": 3.4891021251678467 + }, + { + "bits_per_byte": 6.611912912395678, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.583028793334961, + "step": 30, + "unclipped_grad_norm": 3.0757997035980225 + }, + { + "bits_per_byte": 6.391848039935249, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.4304914474487305, + "step": 40, + "unclipped_grad_norm": 3.231386661529541 + }, + { + "bits_per_byte": 6.094190098850349, + "learning_rate": 3.75e-05, + "loss_nats": 4.224170684814453, + "step": 50, + "unclipped_grad_norm": 2.8728833198547363 + }, + { + "bits_per_byte": 5.886124799905919, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.07995080947876, + "step": 60, + "unclipped_grad_norm": 2.67061448097229 + }, + { + "bits_per_byte": 5.715795246283204, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.9618873596191406, + "step": 70, + "unclipped_grad_norm": 3.178194046020508 + }, + { + "bits_per_byte": 5.566151421940965, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.8581621646881104, + "step": 80, + "unclipped_grad_norm": 2.5578954219818115 + }, + { + "bits_per_byte": 5.355047877486394, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.711836338043213, + "step": 90, + "unclipped_grad_norm": 2.4267516136169434 + }, + { + "bits_per_byte": 5.181267319991465, + "learning_rate": 7.5e-05, + "loss_nats": 3.5913808345794678, + "step": 100, + "unclipped_grad_norm": 2.278679847717285 + }, + { + "bits_per_byte": 4.992526679628584, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.4605557918548584, + "step": 110, + "unclipped_grad_norm": 2.1029887199401855 + }, + { + "bits_per_byte": 4.775955806703084, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.3104403018951416, + "step": 120, + "unclipped_grad_norm": 1.8943580389022827 + }, + { + "bits_per_byte": 4.582269977197294, + "learning_rate": 9.75e-05, + "loss_nats": 3.176187515258789, + "step": 130, + "unclipped_grad_norm": 1.6944341659545898 + }, + { + "bits_per_byte": 4.428514047672926, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.0696120262145996, + "step": 140, + "unclipped_grad_norm": 1.4523073434829712 + }, + { + "bits_per_byte": 4.2718130872351825, + "learning_rate": 0.0001125, + "loss_nats": 2.9609951972961426, + "step": 150, + "unclipped_grad_norm": 1.2266520261764526 + }, + { + "bits_per_byte": 4.167506297417493, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.888695240020752, + "step": 160, + "unclipped_grad_norm": 1.061684489250183 + }, + { + "bits_per_byte": 4.027316703358485, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.7915232181549072, + "step": 170, + "unclipped_grad_norm": 0.9803622961044312 + }, + { + "bits_per_byte": 3.9484987742945683, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.7368907928466797, + "step": 180, + "unclipped_grad_norm": 0.8447042107582092 + }, + { + "bits_per_byte": 3.839664025244962, + "learning_rate": 0.0001425, + "loss_nats": 2.661452293395996, + "step": 190, + "unclipped_grad_norm": 0.8049373626708984 + }, + { + "bits_per_byte": 3.7800194100298348, + "learning_rate": 0.00015, + "loss_nats": 2.620109796524048, + "step": 200, + "unclipped_grad_norm": 1.06807541847229 + }, + { + "bits_per_byte": 3.7362481055587415, + "learning_rate": 0.0001575, + "loss_nats": 2.5897698402404785, + "step": 210, + "unclipped_grad_norm": 0.9001178741455078 + }, + { + "bits_per_byte": 3.634433688317355, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.519197463989258, + "step": 220, + "unclipped_grad_norm": 0.7548930644989014 + }, + { + "bits_per_byte": 3.6247170125089085, + "learning_rate": 0.0001725, + "loss_nats": 2.5124623775482178, + "step": 230, + "unclipped_grad_norm": 1.0285526514053345 + }, + { + "bits_per_byte": 3.4996233676120343, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.4257540702819824, + "step": 240, + "unclipped_grad_norm": 0.6599343419075012 + }, + { + "bits_per_byte": 3.5393427608288, + "learning_rate": 0.0001875, + "loss_nats": 2.4532854557037354, + "step": 250, + "unclipped_grad_norm": 1.2009886503219604 + }, + { + "bits_per_byte": 3.529286247302248, + "learning_rate": 0.000195, + "loss_nats": 2.446314811706543, + "step": 260, + "unclipped_grad_norm": 0.7951757311820984 + }, + { + "bits_per_byte": 3.5336329368201547, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.4493277072906494, + "step": 270, + "unclipped_grad_norm": 1.0654157400131226 + }, + { + "bits_per_byte": 3.4686124879043576, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.404258966445923, + "step": 280, + "unclipped_grad_norm": 0.9586172103881836 + }, + { + "bits_per_byte": 3.36614797624271, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.3332359790802, + "step": 290, + "unclipped_grad_norm": 0.7456421256065369 + }, + { + "bits_per_byte": 3.3991971943284125, + "learning_rate": 0.000225, + "loss_nats": 2.3561439514160156, + "step": 300, + "unclipped_grad_norm": 1.087968349456787 + }, + { + "bits_per_byte": 3.3698115506713897, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.335775375366211, + "step": 310, + "unclipped_grad_norm": 0.9577882885932922 + }, + { + "bits_per_byte": 3.3373914451225435, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.3133034706115723, + "step": 320, + "unclipped_grad_norm": 0.8726407885551453 + }, + { + "bits_per_byte": 3.3356557962100357, + "learning_rate": 0.0002475, + "loss_nats": 2.312100410461426, + "step": 330, + "unclipped_grad_norm": 0.9853833913803101 + }, + { + "bits_per_byte": 3.4121584948280375, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.3651280403137207, + "step": 340, + "unclipped_grad_norm": 1.0082039833068848 + }, + { + "bits_per_byte": 3.343134289800154, + "learning_rate": 0.0002625, + "loss_nats": 2.317284107208252, + "step": 350, + "unclipped_grad_norm": 0.8716307282447815 + }, + { + "bits_per_byte": 3.2895362406240616, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.28013277053833, + "step": 360, + "unclipped_grad_norm": 1.0366642475128174 + }, + { + "bits_per_byte": 3.2849718210701626, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.2769689559936523, + "step": 370, + "unclipped_grad_norm": 1.0000704526901245 + }, + { + "bits_per_byte": 3.3338926300733904, + "learning_rate": 0.000285, + "loss_nats": 2.310878276824951, + "step": 380, + "unclipped_grad_norm": 1.0161696672439575 + }, + { + "bits_per_byte": 3.2617479718282523, + "learning_rate": 0.0002925, + "loss_nats": 2.260871410369873, + "step": 390, + "unclipped_grad_norm": 1.4449015855789185 + }, + { + "bits_per_byte": 3.212265123522005, + "learning_rate": 0.0003, + "loss_nats": 2.2265725135803223, + "step": 400, + "unclipped_grad_norm": 1.0375726222991943 + }, + { + "bits_per_byte": 3.1998858123129, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.217991828918457, + "step": 410, + "unclipped_grad_norm": 0.9804793000221252 + }, + { + "bits_per_byte": 3.2142642498556344, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.2279582023620605, + "step": 420, + "unclipped_grad_norm": 0.7918646931648254 + }, + { + "bits_per_byte": 3.189477422282683, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.2107772827148438, + "step": 430, + "unclipped_grad_norm": 0.9536654353141785 + }, + { + "bits_per_byte": 3.24866559554242, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.251803398132324, + "step": 440, + "unclipped_grad_norm": 1.031852126121521 + }, + { + "bits_per_byte": 3.153853623913564, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.186084747314453, + "step": 450, + "unclipped_grad_norm": 1.0002880096435547 + }, + { + "bits_per_byte": 3.195667421852537, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.2150678634643555, + "step": 460, + "unclipped_grad_norm": 1.6038228273391724 + }, + { + "bits_per_byte": 3.14283985495231, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.178450584411621, + "step": 470, + "unclipped_grad_norm": 1.2647784948349 + }, + { + "bits_per_byte": 3.1429695298710607, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.1785404682159424, + "step": 480, + "unclipped_grad_norm": 1.0020755529403687 + }, + { + "bits_per_byte": 3.0734524225658046, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.130354881286621, + "step": 490, + "unclipped_grad_norm": 1.039467692375183 + }, + { + "bits_per_byte": 3.0349898785615443, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.1036946773529053, + "step": 500, + "unclipped_grad_norm": 1.1395293474197388 + }, + { + "bits_per_byte": 3.063324020291192, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.1233344078063965, + "step": 510, + "unclipped_grad_norm": 1.077130675315857 + }, + { + "bits_per_byte": 2.979445329742987, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.0651941299438477, + "step": 520, + "unclipped_grad_norm": 0.8358139991760254 + }, + { + "bits_per_byte": 3.055472668314003, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.117892265319824, + "step": 530, + "unclipped_grad_norm": 1.0353219509124756 + }, + { + "bits_per_byte": 3.012573315882784, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.0881567001342773, + "step": 540, + "unclipped_grad_norm": 1.1709613800048828 + }, + { + "bits_per_byte": 2.9864316089863356, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.0700366497039795, + "step": 550, + "unclipped_grad_norm": 1.1364045143127441 + }, + { + "bits_per_byte": 2.9710739022295884, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.059391498565674, + "step": 560, + "unclipped_grad_norm": 0.994520366191864 + }, + { + "bits_per_byte": 2.922462661997914, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.0256967544555664, + "step": 570, + "unclipped_grad_norm": 0.91049724817276 + }, + { + "bits_per_byte": 2.9852480243830977, + "learning_rate": 0.000299626473885547, + "loss_nats": 2.069216251373291, + "step": 580, + "unclipped_grad_norm": 0.8277824521064758 + }, + { + "bits_per_byte": 2.8849078424249077, + "learning_rate": 0.00029958384005397226, + "loss_nats": 1.9996657371520996, + "step": 590, + "unclipped_grad_norm": 0.9048410058021545 + }, + { + "bits_per_byte": 2.9369611434309504, + "learning_rate": 0.0002995389065559004, + "loss_nats": 2.0357463359832764, + "step": 600, + "unclipped_grad_norm": 0.9035901427268982 + }, + { + "bits_per_byte": 2.8609755966440935, + "learning_rate": 0.000299491674159121, + "loss_nats": 1.9830771684646606, + "step": 610, + "unclipped_grad_norm": 0.914832592010498 + }, + { + "bits_per_byte": 2.8899545013318257, + "learning_rate": 0.00029944214367070535, + "loss_nats": 2.0031638145446777, + "step": 620, + "unclipped_grad_norm": 0.8642157912254333 + }, + { + "bits_per_byte": 2.8343167379165, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.9645986557006836, + "step": 630, + "unclipped_grad_norm": 1.156167984008789 + }, + { + "bits_per_byte": 2.7918696998570507, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.9351766109466553, + "step": 640, + "unclipped_grad_norm": 1.289353370666504 + }, + { + "bits_per_byte": 2.7261497975001947, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.8896230459213257, + "step": 650, + "unclipped_grad_norm": 1.2669397592544556 + }, + { + "bits_per_byte": 2.670038565847044, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.8507297039031982, + "step": 660, + "unclipped_grad_norm": 1.1661701202392578 + }, + { + "bits_per_byte": 2.7134057110886096, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.8807895183563232, + "step": 670, + "unclipped_grad_norm": 1.3973356485366821 + }, + { + "bits_per_byte": 2.633774476068763, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.82559335231781, + "step": 680, + "unclipped_grad_norm": 0.7573888897895813 + }, + { + "bits_per_byte": 2.723386552248782, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.8877077102661133, + "step": 690, + "unclipped_grad_norm": 0.8143801689147949 + }, + { + "bits_per_byte": 2.6450522383990913, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.8334105014801025, + "step": 700, + "unclipped_grad_norm": 1.3509966135025024 + }, + { + "bits_per_byte": 2.647326021026149, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.8349865674972534, + "step": 710, + "unclipped_grad_norm": 1.151048183441162 + }, + { + "bits_per_byte": 2.6673853894921833, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.8488906621932983, + "step": 720, + "unclipped_grad_norm": 1.0485727787017822 + }, + { + "bits_per_byte": 2.62873693224234, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.8221015930175781, + "step": 730, + "unclipped_grad_norm": 1.4204100370407104 + }, + { + "bits_per_byte": 2.5651828074078, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.7780492305755615, + "step": 740, + "unclipped_grad_norm": 1.0180397033691406 + }, + { + "bits_per_byte": 2.5225455565364956, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.74849534034729, + "step": 750, + "unclipped_grad_norm": 1.0371862649917603 + }, + { + "bits_per_byte": 2.4588593490260924, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.7043514251708984, + "step": 760, + "unclipped_grad_norm": 1.0651884078979492 + }, + { + "bits_per_byte": 2.607158956969187, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.8071448802947998, + "step": 770, + "unclipped_grad_norm": 1.0117440223693848 + }, + { + "bits_per_byte": 2.4918002180417513, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.7271842956542969, + "step": 780, + "unclipped_grad_norm": 0.9473764300346375 + }, + { + "bits_per_byte": 2.4529723828870584, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.7002708911895752, + "step": 790, + "unclipped_grad_norm": 1.0626623630523682 + }, + { + "bits_per_byte": 2.4026655657005276, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.6654008626937866, + "step": 800, + "unclipped_grad_norm": 0.9912887215614319 + }, + { + "bits_per_byte": 2.45885487747717, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7043483257293701, + "step": 810, + "unclipped_grad_norm": 0.7789537906646729 + }, + { + "bits_per_byte": 2.403177558052146, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.6657557487487793, + "step": 820, + "unclipped_grad_norm": 1.076607584953308 + }, + { + "bits_per_byte": 2.47484496444114, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.715431809425354, + "step": 830, + "unclipped_grad_norm": 1.505598545074463 + }, + { + "bits_per_byte": 2.524337615758486, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.7497375011444092, + "step": 840, + "unclipped_grad_norm": 1.0162599086761475 + }, + { + "bits_per_byte": 2.4152144518033833, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.674099087715149, + "step": 850, + "unclipped_grad_norm": 1.0359419584274292 + }, + { + "bits_per_byte": 2.4180359991734384, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.676054835319519, + "step": 860, + "unclipped_grad_norm": 1.0067554712295532 + }, + { + "bits_per_byte": 2.32538430162207, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.6118335723876953, + "step": 870, + "unclipped_grad_norm": 0.9742794036865234 + }, + { + "bits_per_byte": 2.362478035708113, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.6375449895858765, + "step": 880, + "unclipped_grad_norm": 0.8886446356773376 + }, + { + "bits_per_byte": 2.289258657877734, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.5867931842803955, + "step": 890, + "unclipped_grad_norm": 0.8424803614616394 + }, + { + "bits_per_byte": 2.312433664046799, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.6028568744659424, + "step": 900, + "unclipped_grad_norm": 0.8854764103889465 + }, + { + "bits_per_byte": 2.342391150017941, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.6236218214035034, + "step": 910, + "unclipped_grad_norm": 0.898046612739563 + }, + { + "bits_per_byte": 2.3181276656515646, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.6068036556243896, + "step": 920, + "unclipped_grad_norm": 0.9413855075836182 + }, + { + "bits_per_byte": 2.3277418398000975, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.6134676933288574, + "step": 930, + "unclipped_grad_norm": 1.1345362663269043 + }, + { + "bits_per_byte": 2.3328292585952703, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.616994023323059, + "step": 940, + "unclipped_grad_norm": 1.0380187034606934 + }, + { + "bits_per_byte": 2.3110928873006724, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.6019275188446045, + "step": 950, + "unclipped_grad_norm": 0.8527032136917114 + }, + { + "bits_per_byte": 2.3466372296850926, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.6265649795532227, + "step": 960, + "unclipped_grad_norm": 0.860535740852356 + }, + { + "bits_per_byte": 2.258179329075014, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5652506351470947, + "step": 970, + "unclipped_grad_norm": 0.9813200831413269 + }, + { + "bits_per_byte": 2.331846377745589, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.6163127422332764, + "step": 980, + "unclipped_grad_norm": 1.030205249786377 + }, + { + "bits_per_byte": 2.2894072508880794, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.5868961811065674, + "step": 990, + "unclipped_grad_norm": 0.9522655010223389 + }, + { + "bits_per_byte": 2.260999328601211, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.5672053098678589, + "step": 1000, + "unclipped_grad_norm": 1.0130913257598877 + }, + { + "bits_per_byte": 2.2359509154162978, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.5498430728912354, + "step": 1010, + "unclipped_grad_norm": 0.9919615983963013 + }, + { + "bits_per_byte": 2.208706111797215, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.5309584140777588, + "step": 1020, + "unclipped_grad_norm": 0.9296489953994751 + }, + { + "bits_per_byte": 2.218266627358678, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.5375852584838867, + "step": 1030, + "unclipped_grad_norm": 1.2263902425765991 + }, + { + "bits_per_byte": 2.2587441201004475, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.5656421184539795, + "step": 1040, + "unclipped_grad_norm": 0.9231149554252625 + }, + { + "bits_per_byte": 2.2614872433817093, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5675435066223145, + "step": 1050, + "unclipped_grad_norm": 1.0185707807540894 + }, + { + "bits_per_byte": 2.163958977765126, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.4999420642852783, + "step": 1060, + "unclipped_grad_norm": 0.8106277585029602 + }, + { + "bits_per_byte": 2.229709665016486, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.5455169677734375, + "step": 1070, + "unclipped_grad_norm": 0.8736851811408997 + }, + { + "bits_per_byte": 2.2169017730414313, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.5366392135620117, + "step": 1080, + "unclipped_grad_norm": 0.8439515233039856 + }, + { + "bits_per_byte": 2.1854574970535805, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.5148437023162842, + "step": 1090, + "unclipped_grad_norm": 0.8693073987960815 + }, + { + "bits_per_byte": 2.2427502495181586, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5545560121536255, + "step": 1100, + "unclipped_grad_norm": 0.843380868434906 + }, + { + "bits_per_byte": 2.110973014843454, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.4632149934768677, + "step": 1110, + "unclipped_grad_norm": 1.0570415258407593 + }, + { + "bits_per_byte": 2.157980172890531, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.495797872543335, + "step": 1120, + "unclipped_grad_norm": 0.9471346139907837 + }, + { + "bits_per_byte": 2.0891057647515496, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.448057770729065, + "step": 1130, + "unclipped_grad_norm": 0.8324337005615234 + }, + { + "bits_per_byte": 2.2265334894203517, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.5433154106140137, + "step": 1140, + "unclipped_grad_norm": 0.9139943718910217 + }, + { + "bits_per_byte": 2.074005688005795, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.4375911951065063, + "step": 1150, + "unclipped_grad_norm": 0.8777573108673096 + }, + { + "bits_per_byte": 2.110620966357138, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.4629709720611572, + "step": 1160, + "unclipped_grad_norm": 0.92755526304245 + }, + { + "bits_per_byte": 2.1456767060304567, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.4872697591781616, + "step": 1170, + "unclipped_grad_norm": 0.8296754360198975 + }, + { + "bits_per_byte": 2.1448011423549143, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.4866628646850586, + "step": 1180, + "unclipped_grad_norm": 0.96583491563797 + }, + { + "bits_per_byte": 2.064303114774719, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.4308658838272095, + "step": 1190, + "unclipped_grad_norm": 0.8603277206420898 + }, + { + "bits_per_byte": 2.0379099692427087, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.4125715494155884, + "step": 1200, + "unclipped_grad_norm": 1.0588717460632324 + }, + { + "bits_per_byte": 2.0982475025581624, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.4543943405151367, + "step": 1210, + "unclipped_grad_norm": 0.9212584495544434 + }, + { + "bits_per_byte": 2.117589359404797, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4678010940551758, + "step": 1220, + "unclipped_grad_norm": 0.9724780321121216 + }, + { + "bits_per_byte": 2.086449836674275, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.4462168216705322, + "step": 1230, + "unclipped_grad_norm": 0.8407194018363953 + }, + { + "bits_per_byte": 2.0250644130671147, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.403667688369751, + "step": 1240, + "unclipped_grad_norm": 0.8642313480377197 + }, + { + "bits_per_byte": 2.097128239466347, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.4536185264587402, + "step": 1250, + "unclipped_grad_norm": 0.7980665564537048 + }, + { + "bits_per_byte": 2.0804638085283442, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.4420676231384277, + "step": 1260, + "unclipped_grad_norm": 0.8822349309921265 + }, + { + "bits_per_byte": 2.0868987113930273, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.4465279579162598, + "step": 1270, + "unclipped_grad_norm": 0.914838969707489 + }, + { + "bits_per_byte": 1.9713853600624658, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.3664602041244507, + "step": 1280, + "unclipped_grad_norm": 0.8883436322212219 + }, + { + "bits_per_byte": 2.0778157916530096, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4402321577072144, + "step": 1290, + "unclipped_grad_norm": 0.8041548728942871 + }, + { + "bits_per_byte": 2.0249441971941615, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.403584361076355, + "step": 1300, + "unclipped_grad_norm": 0.7877209186553955 + }, + { + "bits_per_byte": 2.0393131756910985, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.4135441780090332, + "step": 1310, + "unclipped_grad_norm": 0.8672157526016235 + }, + { + "bits_per_byte": 2.0233506059512667, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.4024797677993774, + "step": 1320, + "unclipped_grad_norm": 0.9325995445251465 + }, + { + "bits_per_byte": 2.091095604022032, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.4494370222091675, + "step": 1330, + "unclipped_grad_norm": 0.848271369934082 + }, + { + "bits_per_byte": 2.0409298126092086, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.4146647453308105, + "step": 1340, + "unclipped_grad_norm": 0.8920736908912659 + }, + { + "bits_per_byte": 2.0341445810847185, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.4099615812301636, + "step": 1350, + "unclipped_grad_norm": 0.7095686793327332 + }, + { + "bits_per_byte": 2.0853286817733006, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.4454396963119507, + "step": 1360, + "unclipped_grad_norm": 0.7967956066131592 + }, + { + "bits_per_byte": 2.079838479609807, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.441634178161621, + "step": 1370, + "unclipped_grad_norm": 0.8067360520362854 + }, + { + "bits_per_byte": 2.038052542860274, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.412670373916626, + "step": 1380, + "unclipped_grad_norm": 0.8834372758865356 + }, + { + "bits_per_byte": 2.0541291370976276, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.423813819885254, + "step": 1390, + "unclipped_grad_norm": 0.8489160537719727 + }, + { + "bits_per_byte": 1.9705531360149406, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.3658833503723145, + "step": 1400, + "unclipped_grad_norm": 0.7794339656829834 + }, + { + "bits_per_byte": 1.988124259488292, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.3780627250671387, + "step": 1410, + "unclipped_grad_norm": 0.7272437214851379 + }, + { + "bits_per_byte": 2.0400110812883, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.4140279293060303, + "step": 1420, + "unclipped_grad_norm": 0.7708798050880432 + }, + { + "bits_per_byte": 2.06701820488389, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.4327478408813477, + "step": 1430, + "unclipped_grad_norm": 0.7736554741859436 + }, + { + "bits_per_byte": 2.0182611233642835, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.3989520072937012, + "step": 1440, + "unclipped_grad_norm": 0.8357114195823669 + }, + { + "bits_per_byte": 1.956489770688587, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.356135368347168, + "step": 1450, + "unclipped_grad_norm": 0.7494769096374512 + }, + { + "bits_per_byte": 2.013430818632151, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.395603895187378, + "step": 1460, + "unclipped_grad_norm": 0.9647942185401917 + }, + { + "bits_per_byte": 1.93312902325659, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.3399429321289062, + "step": 1470, + "unclipped_grad_norm": 0.8799104690551758 + }, + { + "bits_per_byte": 2.0035156748446084, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.3887312412261963, + "step": 1480, + "unclipped_grad_norm": 0.8662192225456238 + }, + { + "bits_per_byte": 2.017567517333354, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.3984712362289429, + "step": 1490, + "unclipped_grad_norm": 0.8222736716270447 + }, + { + "bits_per_byte": 2.0092430410836415, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.3927011489868164, + "step": 1500, + "unclipped_grad_norm": 0.7836759686470032 + }, + { + "bits_per_byte": 2.014570375676768, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.3963937759399414, + "step": 1510, + "unclipped_grad_norm": 0.7882886528968811 + }, + { + "bits_per_byte": 1.9740357846949124, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.3682973384857178, + "step": 1520, + "unclipped_grad_norm": 0.7656131386756897 + }, + { + "bits_per_byte": 2.039669523743686, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.4137911796569824, + "step": 1530, + "unclipped_grad_norm": 0.7943911552429199 + }, + { + "bits_per_byte": 1.9998276788795062, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3861749172210693, + "step": 1540, + "unclipped_grad_norm": 0.9189959764480591 + }, + { + "bits_per_byte": 2.0584218240631635, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.4267892837524414, + "step": 1550, + "unclipped_grad_norm": 0.8234842419624329 + }, + { + "bits_per_byte": 1.9622986567041292, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.3601617813110352, + "step": 1560, + "unclipped_grad_norm": 0.7911669015884399 + }, + { + "bits_per_byte": 2.0459699361753945, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.4181582927703857, + "step": 1570, + "unclipped_grad_norm": 0.7436185479164124 + }, + { + "bits_per_byte": 1.9817189376395576, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.3736228942871094, + "step": 1580, + "unclipped_grad_norm": 0.79274582862854 + }, + { + "bits_per_byte": 1.9884410515311812, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3782823085784912, + "step": 1590, + "unclipped_grad_norm": 0.7606697082519531 + }, + { + "bits_per_byte": 2.0129520189321486, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.3952720165252686, + "step": 1600, + "unclipped_grad_norm": 0.7806884050369263 + }, + { + "bits_per_byte": 1.9564706806143413, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.3561221361160278, + "step": 1610, + "unclipped_grad_norm": 0.7601831555366516 + }, + { + "bits_per_byte": 1.9083005758821296, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.3227331638336182, + "step": 1620, + "unclipped_grad_norm": 0.7029422521591187 + }, + { + "bits_per_byte": 1.934965454002503, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.3412158489227295, + "step": 1630, + "unclipped_grad_norm": 0.774949848651886 + }, + { + "bits_per_byte": 1.9829843859846061, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.374500036239624, + "step": 1640, + "unclipped_grad_norm": 0.7523757219314575 + }, + { + "bits_per_byte": 2.036539611480634, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.4116216897964478, + "step": 1650, + "unclipped_grad_norm": 0.8146423697471619 + }, + { + "bits_per_byte": 1.9856305110507813, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.3763341903686523, + "step": 1660, + "unclipped_grad_norm": 0.8366428017616272 + }, + { + "bits_per_byte": 1.987395740979237, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.3775577545166016, + "step": 1670, + "unclipped_grad_norm": 0.7255176901817322 + }, + { + "bits_per_byte": 2.0158258490280665, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.397264003753662, + "step": 1680, + "unclipped_grad_norm": 0.870812177658081 + }, + { + "bits_per_byte": 2.0499276009370493, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.4209015369415283, + "step": 1690, + "unclipped_grad_norm": 0.7782557010650635 + }, + { + "bits_per_byte": 1.9465106493549234, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.3492183685302734, + "step": 1700, + "unclipped_grad_norm": 0.742743968963623 + }, + { + "bits_per_byte": 1.8809008158946197, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3037410974502563, + "step": 1710, + "unclipped_grad_norm": 0.7516124248504639 + }, + { + "bits_per_byte": 1.8772993272029015, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.3012447357177734, + "step": 1720, + "unclipped_grad_norm": 0.7469065189361572 + }, + { + "bits_per_byte": 1.8766640232906144, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.3008043766021729, + "step": 1730, + "unclipped_grad_norm": 0.8464695811271667 + }, + { + "bits_per_byte": 1.9707770574263639, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.3660385608673096, + "step": 1740, + "unclipped_grad_norm": 0.76495760679245 + }, + { + "bits_per_byte": 1.9893357052809697, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.3789024353027344, + "step": 1750, + "unclipped_grad_norm": 0.7948366403579712 + }, + { + "bits_per_byte": 1.994245637980452, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.3823057413101196, + "step": 1760, + "unclipped_grad_norm": 0.7990732192993164 + }, + { + "bits_per_byte": 1.9998593236872648, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.3861968517303467, + "step": 1770, + "unclipped_grad_norm": 0.7768941521644592 + }, + { + "bits_per_byte": 1.848577880543511, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.2813365459442139, + "step": 1780, + "unclipped_grad_norm": 0.726925790309906 + }, + { + "bits_per_byte": 1.9517884529446012, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3528766632080078, + "step": 1790, + "unclipped_grad_norm": 0.7306504249572754 + }, + { + "bits_per_byte": 1.8662595888613671, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.2935925722122192, + "step": 1800, + "unclipped_grad_norm": 0.7406154870986938 + }, + { + "bits_per_byte": 1.9350889375458225, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.341301441192627, + "step": 1810, + "unclipped_grad_norm": 0.7317749857902527 + }, + { + "bits_per_byte": 1.9065444610341695, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.3215159177780151, + "step": 1820, + "unclipped_grad_norm": 0.8795179724693298 + }, + { + "bits_per_byte": 1.8923866772324924, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.3117024898529053, + "step": 1830, + "unclipped_grad_norm": 0.7327521443367004 + }, + { + "bits_per_byte": 1.8896521530837738, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.3098070621490479, + "step": 1840, + "unclipped_grad_norm": 0.7564541101455688 + }, + { + "bits_per_byte": 1.9045121420489237, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.3201072216033936, + "step": 1850, + "unclipped_grad_norm": 0.7658549547195435 + }, + { + "bits_per_byte": 1.8985429682027775, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.315969705581665, + "step": 1860, + "unclipped_grad_norm": 0.7712892293930054 + }, + { + "bits_per_byte": 1.8800550052176763, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.3031548261642456, + "step": 1870, + "unclipped_grad_norm": 0.7315443754196167 + }, + { + "bits_per_byte": 1.8815657008228552, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.3042019605636597, + "step": 1880, + "unclipped_grad_norm": 0.7102474570274353 + }, + { + "bits_per_byte": 1.90223302795969, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3185274600982666, + "step": 1890, + "unclipped_grad_norm": 0.7768250703811646 + }, + { + "bits_per_byte": 1.9076367228497992, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.3222730159759521, + "step": 1900, + "unclipped_grad_norm": 0.7433425188064575 + }, + { + "bits_per_byte": 1.8546729456900888, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.2855613231658936, + "step": 1910, + "unclipped_grad_norm": 0.7386458516120911 + }, + { + "bits_per_byte": 1.8621256418825778, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.290727138519287, + "step": 1920, + "unclipped_grad_norm": 0.6984587907791138 + }, + { + "bits_per_byte": 1.9411895061372282, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.3455300331115723, + "step": 1930, + "unclipped_grad_norm": 0.7367396354675293 + }, + { + "bits_per_byte": 1.8175876387539378, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.2598557472229004, + "step": 1940, + "unclipped_grad_norm": 0.7296367287635803 + }, + { + "bits_per_byte": 1.9009060098256325, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.3176076412200928, + "step": 1950, + "unclipped_grad_norm": 0.7117495536804199 + }, + { + "bits_per_byte": 1.8133695922588766, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.256932020187378, + "step": 1960, + "unclipped_grad_norm": 0.7279300093650818 + }, + { + "bits_per_byte": 1.9069315939812619, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.321784257888794, + "step": 1970, + "unclipped_grad_norm": 0.6929229497909546 + }, + { + "bits_per_byte": 1.9313397157570134, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.33870267868042, + "step": 1980, + "unclipped_grad_norm": 0.7450703978538513 + }, + { + "bits_per_byte": 1.9358095448529378, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.3418009281158447, + "step": 1990, + "unclipped_grad_norm": 0.7788240313529968 + }, + { + "bits_per_byte": 1.8986578526135538, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.316049337387085, + "step": 2000, + "unclipped_grad_norm": 0.7929812073707581 + }, + { + "bits_per_byte": 1.8751547035466423, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.2997581958770752, + "step": 2010, + "unclipped_grad_norm": 0.6630290150642395 + }, + { + "bits_per_byte": 1.9025284941538723, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.3187322616577148, + "step": 2020, + "unclipped_grad_norm": 0.7224810123443604 + }, + { + "bits_per_byte": 1.9897981666291398, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.3792229890823364, + "step": 2030, + "unclipped_grad_norm": 0.8106074929237366 + }, + { + "bits_per_byte": 1.894711194741554, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.3133137226104736, + "step": 2040, + "unclipped_grad_norm": 0.7566019892692566 + }, + { + "bits_per_byte": 1.9600597865551972, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.358609914779663, + "step": 2050, + "unclipped_grad_norm": 0.7945913672447205 + }, + { + "bits_per_byte": 1.9036565283608813, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.3195141553878784, + "step": 2060, + "unclipped_grad_norm": 0.7025273442268372 + }, + { + "bits_per_byte": 1.8582849253235094, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.288064956665039, + "step": 2070, + "unclipped_grad_norm": 0.7128275632858276 + }, + { + "bits_per_byte": 1.8662040384651384, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.2935540676116943, + "step": 2080, + "unclipped_grad_norm": 0.8230834007263184 + }, + { + "bits_per_byte": 1.8580567043458145, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.2879067659378052, + "step": 2090, + "unclipped_grad_norm": 0.7080399394035339 + }, + { + "bits_per_byte": 1.848465919837799, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.2812589406967163, + "step": 2100, + "unclipped_grad_norm": 0.6781865954399109 + }, + { + "bits_per_byte": 1.9001021629154997, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.3170504570007324, + "step": 2110, + "unclipped_grad_norm": 0.7264413833618164 + }, + { + "bits_per_byte": 1.8535039796121742, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.284751057624817, + "step": 2120, + "unclipped_grad_norm": 0.6847045421600342 + }, + { + "bits_per_byte": 1.818818690568814, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.2607090473175049, + "step": 2130, + "unclipped_grad_norm": 0.7366615533828735 + }, + { + "bits_per_byte": 1.957108908231693, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.3565645217895508, + "step": 2140, + "unclipped_grad_norm": 0.7153334021568298 + }, + { + "bits_per_byte": 1.821195318821087, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2623564004898071, + "step": 2150, + "unclipped_grad_norm": 0.7194899320602417 + }, + { + "bits_per_byte": 1.8653518644301132, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.2929633855819702, + "step": 2160, + "unclipped_grad_norm": 0.682996392250061 + }, + { + "bits_per_byte": 1.9537624698112046, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.3542449474334717, + "step": 2170, + "unclipped_grad_norm": 0.7126298546791077 + }, + { + "bits_per_byte": 1.8131241730160954, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.256761908531189, + "step": 2180, + "unclipped_grad_norm": 0.7386030554771423 + }, + { + "bits_per_byte": 1.8409139896557623, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.276024341583252, + "step": 2190, + "unclipped_grad_norm": 0.7161043882369995 + }, + { + "bits_per_byte": 1.875607705849015, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.300072193145752, + "step": 2200, + "unclipped_grad_norm": 0.6700615882873535 + }, + { + "bits_per_byte": 1.8337089484805158, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.2710301876068115, + "step": 2210, + "unclipped_grad_norm": 0.7556111812591553 + }, + { + "bits_per_byte": 1.828240588113682, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2672398090362549, + "step": 2220, + "unclipped_grad_norm": 0.6959541440010071 + }, + { + "bits_per_byte": 1.902032324206133, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.3183883428573608, + "step": 2230, + "unclipped_grad_norm": 0.7245193123817444 + }, + { + "bits_per_byte": 1.8334989576638123, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.27088463306427, + "step": 2240, + "unclipped_grad_norm": 0.6954715847969055 + }, + { + "bits_per_byte": 1.8440444178667672, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2781941890716553, + "step": 2250, + "unclipped_grad_norm": 0.7106630206108093 + }, + { + "bits_per_byte": 1.8427714022850807, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.2773118019104004, + "step": 2260, + "unclipped_grad_norm": 0.7289770245552063 + }, + { + "bits_per_byte": 1.8113226547483074, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.2555131912231445, + "step": 2270, + "unclipped_grad_norm": 0.7055426836013794 + }, + { + "bits_per_byte": 1.8273519537566738, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.266623854637146, + "step": 2280, + "unclipped_grad_norm": 0.7293174862861633 + }, + { + "bits_per_byte": 1.8650995658812974, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.2927885055541992, + "step": 2290, + "unclipped_grad_norm": 0.7216933369636536 + }, + { + "bits_per_byte": 1.7969402616046033, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.2455440759658813, + "step": 2300, + "unclipped_grad_norm": 0.6945967078208923 + }, + { + "bits_per_byte": 1.8239633795867238, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.264275074005127, + "step": 2310, + "unclipped_grad_norm": 0.6992827653884888 + }, + { + "bits_per_byte": 1.797439527240058, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.2458901405334473, + "step": 2320, + "unclipped_grad_norm": 0.6910362243652344 + }, + { + "bits_per_byte": 1.8787790659309251, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.3022704124450684, + "step": 2330, + "unclipped_grad_norm": 0.7083248496055603 + }, + { + "bits_per_byte": 1.791792304916327, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.2419757843017578, + "step": 2340, + "unclipped_grad_norm": 0.7164211869239807 + }, + { + "bits_per_byte": 1.8459845541511506, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2795389890670776, + "step": 2350, + "unclipped_grad_norm": 0.6873194575309753 + }, + { + "bits_per_byte": 1.8478163413654902, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.280808687210083, + "step": 2360, + "unclipped_grad_norm": 0.694580078125 + }, + { + "bits_per_byte": 1.8740504029454523, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.2989927530288696, + "step": 2370, + "unclipped_grad_norm": 0.6925637125968933 + }, + { + "bits_per_byte": 1.8255325492931962, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.2653627395629883, + "step": 2380, + "unclipped_grad_norm": 0.7442867755889893 + }, + { + "bits_per_byte": 1.8786194660309246, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.3021597862243652, + "step": 2390, + "unclipped_grad_norm": 0.6913683414459229 + }, + { + "bits_per_byte": 1.8201859526431732, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.2616567611694336, + "step": 2400, + "unclipped_grad_norm": 0.7264137268066406 + }, + { + "bits_per_byte": 1.819480479809334, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2611677646636963, + "step": 2410, + "unclipped_grad_norm": 0.6944721937179565 + }, + { + "bits_per_byte": 1.8053809981262992, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.2513947486877441, + "step": 2420, + "unclipped_grad_norm": 0.7260400056838989 + }, + { + "bits_per_byte": 1.8193435816192471, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.2610728740692139, + "step": 2430, + "unclipped_grad_norm": 0.686664879322052 + }, + { + "bits_per_byte": 1.8780073797765038, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.301735520362854, + "step": 2440, + "unclipped_grad_norm": 0.7226059436798096 + }, + { + "bits_per_byte": 1.8239186640974996, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.2642440795898438, + "step": 2450, + "unclipped_grad_norm": 0.7185655236244202 + }, + { + "bits_per_byte": 1.8211920511507207, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.2623541355133057, + "step": 2460, + "unclipped_grad_norm": 0.6750132441520691 + }, + { + "bits_per_byte": 1.799440373400196, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.247277021408081, + "step": 2470, + "unclipped_grad_norm": 0.6895372271537781 + }, + { + "bits_per_byte": 1.8350810260690993, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.2719812393188477, + "step": 2480, + "unclipped_grad_norm": 0.7329404354095459 + }, + { + "bits_per_byte": 1.8136609308694382, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.257133960723877, + "step": 2490, + "unclipped_grad_norm": 0.6904310584068298 + }, + { + "bits_per_byte": 1.830889464902271, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.269075870513916, + "step": 2500, + "unclipped_grad_norm": 0.7081047892570496 + }, + { + "bits_per_byte": 1.7780357566391125, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.23244047164917, + "step": 2510, + "unclipped_grad_norm": 0.68377685546875 + }, + { + "bits_per_byte": 1.8371145489329013, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.273390769958496, + "step": 2520, + "unclipped_grad_norm": 0.7171579003334045 + }, + { + "bits_per_byte": 1.8554432559833032, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.2860952615737915, + "step": 2530, + "unclipped_grad_norm": 0.7040185332298279 + }, + { + "bits_per_byte": 1.8074733390566962, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.252845048904419, + "step": 2540, + "unclipped_grad_norm": 0.7001106142997742 + }, + { + "bits_per_byte": 1.8015585117282225, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.2487452030181885, + "step": 2550, + "unclipped_grad_norm": 0.7488880157470703 + }, + { + "bits_per_byte": 1.8297978910172448, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.2683192491531372, + "step": 2560, + "unclipped_grad_norm": 0.7303985953330994 + }, + { + "bits_per_byte": 1.8215824517681791, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.262624740600586, + "step": 2570, + "unclipped_grad_norm": 0.7168057560920715 + }, + { + "bits_per_byte": 1.8344753031727605, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.2715613842010498, + "step": 2580, + "unclipped_grad_norm": 0.6675634980201721 + }, + { + "bits_per_byte": 1.7919849254852933, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.2421092987060547, + "step": 2590, + "unclipped_grad_norm": 0.6636251211166382 + }, + { + "bits_per_byte": 1.7988516767862925, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.2468689680099487, + "step": 2600, + "unclipped_grad_norm": 0.7330243587493896 + }, + { + "bits_per_byte": 1.7487594939480384, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.2121477127075195, + "step": 2610, + "unclipped_grad_norm": 0.6684263944625854 + }, + { + "bits_per_byte": 1.7990964080984704, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.2470386028289795, + "step": 2620, + "unclipped_grad_norm": 0.6630443334579468 + }, + { + "bits_per_byte": 1.8017267107607662, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.2488617897033691, + "step": 2630, + "unclipped_grad_norm": 0.7096472382545471 + }, + { + "bits_per_byte": 1.6781729985197176, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.1632208824157715, + "step": 2640, + "unclipped_grad_norm": 0.6569882035255432 + }, + { + "bits_per_byte": 1.7748444465697024, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2302284240722656, + "step": 2650, + "unclipped_grad_norm": 0.7178364992141724 + }, + { + "bits_per_byte": 1.8235702272468515, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.2640025615692139, + "step": 2660, + "unclipped_grad_norm": 0.7833260297775269 + }, + { + "bits_per_byte": 1.8214443496995363, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.2625290155410767, + "step": 2670, + "unclipped_grad_norm": 0.7380775213241577 + }, + { + "bits_per_byte": 1.7325511610001239, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.2009129524230957, + "step": 2680, + "unclipped_grad_norm": 0.623815655708313 + }, + { + "bits_per_byte": 1.903327181584479, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.3192858695983887, + "step": 2690, + "unclipped_grad_norm": 0.7134268879890442 + }, + { + "bits_per_byte": 1.7973354777362862, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.245818018913269, + "step": 2700, + "unclipped_grad_norm": 0.6990713477134705 + }, + { + "bits_per_byte": 1.723019882489307, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.1943063735961914, + "step": 2710, + "unclipped_grad_norm": 0.6659460663795471 + }, + { + "bits_per_byte": 1.7371288232181392, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.2040859460830688, + "step": 2720, + "unclipped_grad_norm": 0.6658623218536377 + }, + { + "bits_per_byte": 1.7739613156575218, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.2296162843704224, + "step": 2730, + "unclipped_grad_norm": 0.6642745733261108 + }, + { + "bits_per_byte": 1.774000011753966, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.2296431064605713, + "step": 2740, + "unclipped_grad_norm": 0.7008056044578552 + }, + { + "bits_per_byte": 1.7833926722481874, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.2361536026000977, + "step": 2750, + "unclipped_grad_norm": 0.6654229760169983 + }, + { + "bits_per_byte": 1.8127580219524084, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.2565081119537354, + "step": 2760, + "unclipped_grad_norm": 0.7117574214935303 + }, + { + "bits_per_byte": 1.7051441777412775, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.1819158792495728, + "step": 2770, + "unclipped_grad_norm": 0.6891454458236694 + }, + { + "bits_per_byte": 1.71650242795221, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.189788818359375, + "step": 2780, + "unclipped_grad_norm": 0.6296228170394897 + }, + { + "bits_per_byte": 1.7972534420118245, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2457611560821533, + "step": 2790, + "unclipped_grad_norm": 0.7141041159629822 + }, + { + "bits_per_byte": 1.8492599637368328, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.2818093299865723, + "step": 2800, + "unclipped_grad_norm": 0.7304943799972534 + }, + { + "bits_per_byte": 1.7877589677882924, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.239180088043213, + "step": 2810, + "unclipped_grad_norm": 0.7354668378829956 + }, + { + "bits_per_byte": 1.7754795784993387, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2306686639785767, + "step": 2820, + "unclipped_grad_norm": 0.668363630771637 + }, + { + "bits_per_byte": 1.849954085715715, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.2822904586791992, + "step": 2830, + "unclipped_grad_norm": 0.6875283122062683 + }, + { + "bits_per_byte": 1.748989606734893, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.2123072147369385, + "step": 2840, + "unclipped_grad_norm": 0.7010104656219482 + }, + { + "bits_per_byte": 1.7694087629065325, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.2264606952667236, + "step": 2850, + "unclipped_grad_norm": 0.7100708484649658 + }, + { + "bits_per_byte": 1.7470867906857468, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.2109882831573486, + "step": 2860, + "unclipped_grad_norm": 0.7309325933456421 + }, + { + "bits_per_byte": 1.8302120252405223, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.2686063051223755, + "step": 2870, + "unclipped_grad_norm": 0.6812261343002319 + }, + { + "bits_per_byte": 1.8293796292103464, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2680293321609497, + "step": 2880, + "unclipped_grad_norm": 0.7015396952629089 + }, + { + "bits_per_byte": 1.7246339396676544, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.1954251527786255, + "step": 2890, + "unclipped_grad_norm": 0.6804452538490295 + }, + { + "bits_per_byte": 1.696008631310096, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.1755836009979248, + "step": 2900, + "unclipped_grad_norm": 0.6795438528060913 + }, + { + "bits_per_byte": 1.7582123483700618, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.2186999320983887, + "step": 2910, + "unclipped_grad_norm": 0.7162076234817505 + }, + { + "bits_per_byte": 1.7311384955059368, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.1999337673187256, + "step": 2920, + "unclipped_grad_norm": 0.642425537109375 + }, + { + "bits_per_byte": 1.7472603211804674, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.2111085653305054, + "step": 2930, + "unclipped_grad_norm": 0.6681431531906128 + }, + { + "bits_per_byte": 1.784986435473733, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.2372583150863647, + "step": 2940, + "unclipped_grad_norm": 0.7457805871963501 + }, + { + "bits_per_byte": 1.7723696162237865, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2285130023956299, + "step": 2950, + "unclipped_grad_norm": 0.7907178997993469 + }, + { + "bits_per_byte": 1.7976032547236795, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.2460036277770996, + "step": 2960, + "unclipped_grad_norm": 0.7076123356819153 + }, + { + "bits_per_byte": 1.7638955150678233, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.2226392030715942, + "step": 2970, + "unclipped_grad_norm": 0.700858473777771 + }, + { + "bits_per_byte": 1.7518768514775778, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.214308500289917, + "step": 2980, + "unclipped_grad_norm": 0.7114378213882446 + }, + { + "bits_per_byte": 1.7624200759060713, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.221616506576538, + "step": 2990, + "unclipped_grad_norm": 0.708852231502533 + }, + { + "bits_per_byte": 1.7811185456558285, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.2345772981643677, + "step": 3000, + "unclipped_grad_norm": 0.6479299068450928 + }, + { + "bits_per_byte": 1.7787508605014002, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.232936143875122, + "step": 3010, + "unclipped_grad_norm": 0.675828218460083 + }, + { + "bits_per_byte": 1.7149724702901346, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.1887283325195312, + "step": 3020, + "unclipped_grad_norm": 0.6895548701286316 + }, + { + "bits_per_byte": 1.793058269209328, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.2428532838821411, + "step": 3030, + "unclipped_grad_norm": 0.6785224676132202 + }, + { + "bits_per_byte": 1.7164472215212832, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.1897505521774292, + "step": 3040, + "unclipped_grad_norm": 0.6890314817428589 + }, + { + "bits_per_byte": 1.7444410095848732, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.2091543674468994, + "step": 3050, + "unclipped_grad_norm": 0.6946629881858826 + }, + { + "bits_per_byte": 1.7550958507537768, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.216539740562439, + "step": 3060, + "unclipped_grad_norm": 0.6756144762039185 + }, + { + "bits_per_byte": 1.68935582642677, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.170972228050232, + "step": 3070, + "unclipped_grad_norm": 0.6355137825012207 + }, + { + "bits_per_byte": 1.7227424744734654, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.194114089012146, + "step": 3080, + "unclipped_grad_norm": 0.7417197227478027 + }, + { + "bits_per_byte": 1.7729151451923233, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.228891134262085, + "step": 3090, + "unclipped_grad_norm": 0.7240374684333801 + }, + { + "bits_per_byte": 1.6888324832201944, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.170609474182129, + "step": 3100, + "unclipped_grad_norm": 0.6905462741851807 + }, + { + "bits_per_byte": 1.7862178312539108, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.2381118535995483, + "step": 3110, + "unclipped_grad_norm": 0.6700783967971802 + }, + { + "bits_per_byte": 1.738933265190992, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.2053366899490356, + "step": 3120, + "unclipped_grad_norm": 0.683162271976471 + }, + { + "bits_per_byte": 1.8009926888068837, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.2483530044555664, + "step": 3130, + "unclipped_grad_norm": 0.6745309829711914 + }, + { + "bits_per_byte": 1.7664030221174032, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.2243772745132446, + "step": 3140, + "unclipped_grad_norm": 0.7042497396469116 + }, + { + "bits_per_byte": 1.7378956938583365, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.2046175003051758, + "step": 3150, + "unclipped_grad_norm": 0.7415223717689514 + }, + { + "bits_per_byte": 1.7528420181142197, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.214977502822876, + "step": 3160, + "unclipped_grad_norm": 0.7002313137054443 + }, + { + "bits_per_byte": 1.6640232979026308, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.1534130573272705, + "step": 3170, + "unclipped_grad_norm": 0.6698608994483948 + }, + { + "bits_per_byte": 1.758118101877389, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.2186346054077148, + "step": 3180, + "unclipped_grad_norm": 0.7398433089256287 + }, + { + "bits_per_byte": 1.733427240623619, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.2015202045440674, + "step": 3190, + "unclipped_grad_norm": 0.7051085829734802 + }, + { + "bits_per_byte": 1.6827728464996943, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.1664092540740967, + "step": 3200, + "unclipped_grad_norm": 0.6569372415542603 + }, + { + "bits_per_byte": 1.696978957426264, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.1762561798095703, + "step": 3210, + "unclipped_grad_norm": 0.6764466166496277 + }, + { + "bits_per_byte": 1.706478935094624, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.1828410625457764, + "step": 3220, + "unclipped_grad_norm": 0.6592810153961182 + }, + { + "bits_per_byte": 1.784065124413061, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.2366197109222412, + "step": 3230, + "unclipped_grad_norm": 0.7269020676612854 + }, + { + "bits_per_byte": 1.676728172269819, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.1622194051742554, + "step": 3240, + "unclipped_grad_norm": 0.669075071811676 + }, + { + "bits_per_byte": 1.8351107790676986, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.27200186252594, + "step": 3250, + "unclipped_grad_norm": 0.7432712912559509 + }, + { + "bits_per_byte": 1.6801260335029156, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.1645746231079102, + "step": 3260, + "unclipped_grad_norm": 0.6634126901626587 + }, + { + "bits_per_byte": 1.7169232694988714, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.1900805234909058, + "step": 3270, + "unclipped_grad_norm": 0.6755850911140442 + }, + { + "bits_per_byte": 1.770880934397918, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.2274811267852783, + "step": 3280, + "unclipped_grad_norm": 0.6847168803215027 + }, + { + "bits_per_byte": 1.760565243016516, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.220330834388733, + "step": 3290, + "unclipped_grad_norm": 0.6773887276649475 + }, + { + "bits_per_byte": 1.6893532466870071, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.1709704399108887, + "step": 3300, + "unclipped_grad_norm": 0.6897063851356506 + }, + { + "bits_per_byte": 1.725841773824664, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.1962623596191406, + "step": 3310, + "unclipped_grad_norm": 0.6907094120979309 + }, + { + "bits_per_byte": 1.7237597518533188, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.1948192119598389, + "step": 3320, + "unclipped_grad_norm": 0.7034400105476379 + }, + { + "bits_per_byte": 1.7310239550604622, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.1998543739318848, + "step": 3330, + "unclipped_grad_norm": 0.7049024105072021 + }, + { + "bits_per_byte": 1.7151881365343165, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.188877820968628, + "step": 3340, + "unclipped_grad_norm": 0.7043690085411072 + }, + { + "bits_per_byte": 1.7013854969066708, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.1793105602264404, + "step": 3350, + "unclipped_grad_norm": 0.6487130522727966 + }, + { + "bits_per_byte": 1.733242875221894, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.201392412185669, + "step": 3360, + "unclipped_grad_norm": 0.7069518566131592 + }, + { + "bits_per_byte": 1.6775552368378184, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.162792682647705, + "step": 3370, + "unclipped_grad_norm": 0.6950260996818542 + }, + { + "bits_per_byte": 1.7055392218903094, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.182189702987671, + "step": 3380, + "unclipped_grad_norm": 0.6694713830947876 + }, + { + "bits_per_byte": 1.7381113601025184, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.2047669887542725, + "step": 3390, + "unclipped_grad_norm": 0.6793012022972107 + }, + { + "bits_per_byte": 1.7880630331150178, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.2393908500671387, + "step": 3400, + "unclipped_grad_norm": 0.7019150853157043 + }, + { + "bits_per_byte": 1.822066754913009, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.262960433959961, + "step": 3410, + "unclipped_grad_norm": 0.705566942691803 + }, + { + "bits_per_byte": 1.684419064433753, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.1675503253936768, + "step": 3420, + "unclipped_grad_norm": 0.6660227179527283 + }, + { + "bits_per_byte": 1.6392639875538175, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1362512111663818, + "step": 3430, + "unclipped_grad_norm": 0.7216922640800476 + }, + { + "bits_per_byte": 1.7065609708190854, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.182897925376892, + "step": 3440, + "unclipped_grad_norm": 0.7238938212394714 + }, + { + "bits_per_byte": 1.7774283139162652, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.2320194244384766, + "step": 3450, + "unclipped_grad_norm": 0.7299274206161499 + }, + { + "bits_per_byte": 1.704590909553452, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.181532382965088, + "step": 3460, + "unclipped_grad_norm": 0.6929846405982971 + }, + { + "bits_per_byte": 1.6909616284378761, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.1720852851867676, + "step": 3470, + "unclipped_grad_norm": 0.7074986696243286 + }, + { + "bits_per_byte": 1.6381101559491789, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.1354514360427856, + "step": 3480, + "unclipped_grad_norm": 0.6581223607063293 + }, + { + "bits_per_byte": 1.8411740274238668, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.2762045860290527, + "step": 3490, + "unclipped_grad_norm": 0.7560490965843201 + }, + { + "bits_per_byte": 1.6957697474080475, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1754180192947388, + "step": 3500, + "unclipped_grad_norm": 0.674267053604126 + }, + { + "bits_per_byte": 1.7297523153399825, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.1989729404449463, + "step": 3510, + "unclipped_grad_norm": 0.6964191794395447 + }, + { + "bits_per_byte": 1.7397433034765557, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.2058981657028198, + "step": 3520, + "unclipped_grad_norm": 0.6925695538520813 + }, + { + "bits_per_byte": 1.6948379454056728, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.1747721433639526, + "step": 3530, + "unclipped_grad_norm": 0.6672089099884033 + }, + { + "bits_per_byte": 1.706901324485143, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.183133840560913, + "step": 3540, + "unclipped_grad_norm": 0.6990481019020081 + }, + { + "bits_per_byte": 1.7118731709389359, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.1865800619125366, + "step": 3550, + "unclipped_grad_norm": 0.7408817410469055 + }, + { + "bits_per_byte": 1.6410732450408942, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.137505292892456, + "step": 3560, + "unclipped_grad_norm": 0.6699543595314026 + }, + { + "bits_per_byte": 1.68360919813084, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.1669889688491821, + "step": 3570, + "unclipped_grad_norm": 0.7012400031089783 + }, + { + "bits_per_byte": 1.686249303804235, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.1688189506530762, + "step": 3580, + "unclipped_grad_norm": 0.6917738318443298 + }, + { + "bits_per_byte": 1.7161242381029627, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.1895266771316528, + "step": 3590, + "unclipped_grad_norm": 0.6996616125106812 + }, + { + "bits_per_byte": 1.7404492922583474, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.2063875198364258, + "step": 3600, + "unclipped_grad_norm": 0.7486506104469299 + }, + { + "bits_per_byte": 1.6715428953463054, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.1586252450942993, + "step": 3610, + "unclipped_grad_norm": 0.7099218964576721 + }, + { + "bits_per_byte": 1.641107297605765, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.137528896331787, + "step": 3620, + "unclipped_grad_norm": 0.679558277130127 + }, + { + "bits_per_byte": 1.6857791032367762, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.1684930324554443, + "step": 3630, + "unclipped_grad_norm": 0.6757200360298157 + }, + { + "bits_per_byte": 1.6828182499195221, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.166440725326538, + "step": 3640, + "unclipped_grad_norm": 0.7022830247879028 + }, + { + "bits_per_byte": 1.7172737701413296, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.1903234720230103, + "step": 3650, + "unclipped_grad_norm": 0.7042719125747681 + }, + { + "bits_per_byte": 1.6349776639463638, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.1332801580429077, + "step": 3660, + "unclipped_grad_norm": 0.6939542293548584 + }, + { + "bits_per_byte": 1.7189297910864878, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.1914713382720947, + "step": 3670, + "unclipped_grad_norm": 0.6930156350135803 + }, + { + "bits_per_byte": 1.6495354793939478, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.1433708667755127, + "step": 3680, + "unclipped_grad_norm": 0.7055285573005676 + }, + { + "bits_per_byte": 1.630906834600441, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.1304584741592407, + "step": 3690, + "unclipped_grad_norm": 0.691986620426178 + }, + { + "bits_per_byte": 1.6499286317338202, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.1436433792114258, + "step": 3700, + "unclipped_grad_norm": 0.7110812067985535 + }, + { + "bits_per_byte": 1.6873843892999296, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.1696057319641113, + "step": 3710, + "unclipped_grad_norm": 0.6750229001045227 + }, + { + "bits_per_byte": 1.676168024775959, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.1618311405181885, + "step": 3720, + "unclipped_grad_norm": 0.6922392249107361 + }, + { + "bits_per_byte": 1.6081509501515285, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.114685297012329, + "step": 3730, + "unclipped_grad_norm": 0.6754937171936035 + }, + { + "bits_per_byte": 1.6607097081684572, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.1511162519454956, + "step": 3740, + "unclipped_grad_norm": 0.6955247521400452 + }, + { + "bits_per_byte": 1.6855065107351586, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1683040857315063, + "step": 3750, + "unclipped_grad_norm": 0.686244785785675 + }, + { + "bits_per_byte": 1.7684872798632094, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.2258219718933105, + "step": 3760, + "unclipped_grad_norm": 0.7056378722190857 + }, + { + "bits_per_byte": 1.7298699514731728, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.199054479598999, + "step": 3770, + "unclipped_grad_norm": 0.7548426985740662 + }, + { + "bits_per_byte": 1.5554475547252242, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.0781540870666504, + "step": 3780, + "unclipped_grad_norm": 0.6753653883934021 + }, + { + "bits_per_byte": 1.6604109743039084, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.150909185409546, + "step": 3790, + "unclipped_grad_norm": 0.6861468553543091 + }, + { + "bits_per_byte": 1.6653038807209553, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.1543006896972656, + "step": 3800, + "unclipped_grad_norm": 0.6723336577415466 + }, + { + "bits_per_byte": 1.688550947620732, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.1704143285751343, + "step": 3810, + "unclipped_grad_norm": 0.6901013851165771 + }, + { + "bits_per_byte": 1.660442963076969, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1509313583374023, + "step": 3820, + "unclipped_grad_norm": 0.6948636174201965 + }, + { + "bits_per_byte": 1.7101088009237342, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.1853570938110352, + "step": 3830, + "unclipped_grad_norm": 0.7168598771095276 + }, + { + "bits_per_byte": 1.648008445436937, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.1423124074935913, + "step": 3840, + "unclipped_grad_norm": 0.6718893647193909 + }, + { + "bits_per_byte": 1.5877923318729927, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1005737781524658, + "step": 3850, + "unclipped_grad_norm": 0.6956433653831482 + }, + { + "bits_per_byte": 1.6519181270390013, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.1450223922729492, + "step": 3860, + "unclipped_grad_norm": 0.6815741062164307 + }, + { + "bits_per_byte": 1.7277514691798446, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.1975860595703125, + "step": 3870, + "unclipped_grad_norm": 0.7105231881141663 + }, + { + "bits_per_byte": 1.6282699965974126, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1286307573318481, + "step": 3880, + "unclipped_grad_norm": 0.6807541847229004 + }, + { + "bits_per_byte": 1.6016768352424489, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.1101977825164795, + "step": 3890, + "unclipped_grad_norm": 0.6707821488380432 + }, + { + "bits_per_byte": 1.6689946284084711, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.1568589210510254, + "step": 3900, + "unclipped_grad_norm": 0.7559595704078674 + }, + { + "bits_per_byte": 1.6810941238446224, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1652456521987915, + "step": 3910, + "unclipped_grad_norm": 0.7163210511207581 + }, + { + "bits_per_byte": 1.6455324112124652, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.1405961513519287, + "step": 3920, + "unclipped_grad_norm": 0.7001592516899109 + }, + { + "bits_per_byte": 1.6730757767134457, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.1596877574920654, + "step": 3930, + "unclipped_grad_norm": 0.7008352279663086 + }, + { + "bits_per_byte": 1.6977472039276682, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.1767886877059937, + "step": 3940, + "unclipped_grad_norm": 0.694952130317688 + }, + { + "bits_per_byte": 1.7255263576429816, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1960437297821045, + "step": 3950, + "unclipped_grad_norm": 0.6989170908927917 + }, + { + "bits_per_byte": 1.7056455071685426, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.1822633743286133, + "step": 3960, + "unclipped_grad_norm": 0.7249977588653564 + }, + { + "bits_per_byte": 1.6340527412500234, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1326390504837036, + "step": 3970, + "unclipped_grad_norm": 0.6942048668861389 + }, + { + "bits_per_byte": 1.6691655791634288, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.1569774150848389, + "step": 3980, + "unclipped_grad_norm": 0.7154849171638489 + }, + { + "bits_per_byte": 1.694122153612782, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1742759943008423, + "step": 3990, + "unclipped_grad_norm": 0.7409498691558838 + }, + { + "bits_per_byte": 1.5744717596157156, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.0913406610488892, + "step": 4000, + "unclipped_grad_norm": 0.6975210905075073 + }, + { + "bits_per_byte": 1.6650266446877644, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.1541085243225098, + "step": 4010, + "unclipped_grad_norm": 0.7477675676345825 + }, + { + "bits_per_byte": 1.6456847878411296, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.1407017707824707, + "step": 4020, + "unclipped_grad_norm": 0.6922195553779602 + }, + { + "bits_per_byte": 1.6912369726619074, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.1722761392593384, + "step": 4030, + "unclipped_grad_norm": 0.7471164464950562 + }, + { + "bits_per_byte": 1.6154770671129823, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1197633743286133, + "step": 4040, + "unclipped_grad_norm": 0.7349501252174377 + }, + { + "bits_per_byte": 1.6476533012629053, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.142066240310669, + "step": 4050, + "unclipped_grad_norm": 0.6809535622596741 + }, + { + "bits_per_byte": 1.6566491977815863, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1483017206192017, + "step": 4060, + "unclipped_grad_norm": 0.6904850602149963 + }, + { + "bits_per_byte": 1.6789126959010785, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.1637336015701294, + "step": 4070, + "unclipped_grad_norm": 0.7032252550125122 + }, + { + "bits_per_byte": 1.7607214032634992, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.220439076423645, + "step": 4080, + "unclipped_grad_norm": 0.7632616758346558 + }, + { + "bits_per_byte": 1.6501013023152866, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.1437630653381348, + "step": 4090, + "unclipped_grad_norm": 0.747443437576294 + }, + { + "bits_per_byte": 1.6403294200759126, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.136989712715149, + "step": 4100, + "unclipped_grad_norm": 0.7087969779968262 + }, + { + "bits_per_byte": 1.6314411846966719, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.130828857421875, + "step": 4110, + "unclipped_grad_norm": 0.7110900282859802 + }, + { + "bits_per_byte": 1.6753813761309124, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.1612858772277832, + "step": 4120, + "unclipped_grad_norm": 0.7461647391319275 + }, + { + "bits_per_byte": 1.6246910376329573, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.1261500120162964, + "step": 4130, + "unclipped_grad_norm": 0.7174811363220215 + }, + { + "bits_per_byte": 1.6784760319505379, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.16343092918396, + "step": 4140, + "unclipped_grad_norm": 0.7715107798576355 + }, + { + "bits_per_byte": 1.6147891365095313, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1192865371704102, + "step": 4150, + "unclipped_grad_norm": 0.7233452200889587 + }, + { + "bits_per_byte": 1.6515136238441719, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.1447420120239258, + "step": 4160, + "unclipped_grad_norm": 0.7199777960777283 + }, + { + "bits_per_byte": 1.6926123179208572, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.173229455947876, + "step": 4170, + "unclipped_grad_norm": 0.7814860939979553 + }, + { + "bits_per_byte": 1.727389961647731, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.1973354816436768, + "step": 4180, + "unclipped_grad_norm": 0.7282893061637878 + }, + { + "bits_per_byte": 1.6396929122850694, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.1365485191345215, + "step": 4190, + "unclipped_grad_norm": 0.7101180553436279 + }, + { + "bits_per_byte": 1.6774444800106627, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1627159118652344, + "step": 4200, + "unclipped_grad_norm": 0.7076637148857117 + }, + { + "bits_per_byte": 1.6691000537734502, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.15693199634552, + "step": 4210, + "unclipped_grad_norm": 0.7152689695358276 + }, + { + "bits_per_byte": 1.6877589675135087, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.169865369796753, + "step": 4220, + "unclipped_grad_norm": 0.7122268676757812 + }, + { + "bits_per_byte": 1.6533994136108825, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.14604914188385, + "step": 4230, + "unclipped_grad_norm": 0.7337707877159119 + }, + { + "bits_per_byte": 1.6184998341845471, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.1218585968017578, + "step": 4240, + "unclipped_grad_norm": 0.6965084075927734 + }, + { + "bits_per_byte": 1.646351392595874, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1411638259887695, + "step": 4250, + "unclipped_grad_norm": 0.7121448516845703 + }, + { + "bits_per_byte": 1.629267323989766, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1293220520019531, + "step": 4260, + "unclipped_grad_norm": 0.7370728254318237 + }, + { + "bits_per_byte": 1.6177155932966127, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.1213150024414062, + "step": 4270, + "unclipped_grad_norm": 0.7052253484725952 + }, + { + "bits_per_byte": 1.619418909470758, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1224956512451172, + "step": 4280, + "unclipped_grad_norm": 0.6932440996170044 + }, + { + "bits_per_byte": 1.6646357281223534, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.1538375616073608, + "step": 4290, + "unclipped_grad_norm": 0.7263500094413757 + }, + { + "bits_per_byte": 1.587350680425577, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.1002676486968994, + "step": 4300, + "unclipped_grad_norm": 0.7119481563568115 + }, + { + "bits_per_byte": 1.6717487585793882, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.1587679386138916, + "step": 4310, + "unclipped_grad_norm": 0.7742461562156677 + }, + { + "bits_per_byte": 1.6490056008466396, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.1430035829544067, + "step": 4320, + "unclipped_grad_norm": 0.7111946940422058 + }, + { + "bits_per_byte": 1.6070678034163945, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.1139345169067383, + "step": 4330, + "unclipped_grad_norm": 0.7143876552581787 + }, + { + "bits_per_byte": 1.6437745765379963, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.1393777132034302, + "step": 4340, + "unclipped_grad_norm": 0.7429751753807068 + }, + { + "bits_per_byte": 1.6382068101989637, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.1355184316635132, + "step": 4350, + "unclipped_grad_norm": 0.7563796043395996 + }, + { + "bits_per_byte": 1.6243823287746586, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.1259360313415527, + "step": 4360, + "unclipped_grad_norm": 0.7154512405395508 + }, + { + "bits_per_byte": 1.631903646044842, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.131149411201477, + "step": 4370, + "unclipped_grad_norm": 0.768316924571991 + }, + { + "bits_per_byte": 1.6774572067268265, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.1627247333526611, + "step": 4380, + "unclipped_grad_norm": 0.7411073446273804 + }, + { + "bits_per_byte": 1.6543224444980633, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1466889381408691, + "step": 4390, + "unclipped_grad_norm": 0.7413867115974426 + }, + { + "bits_per_byte": 1.593424419723448, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.1044776439666748, + "step": 4400, + "unclipped_grad_norm": 0.7185443043708801 + }, + { + "bits_per_byte": 1.6845700652012106, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.1676549911499023, + "step": 4410, + "unclipped_grad_norm": 0.7652710676193237 + }, + { + "bits_per_byte": 1.6098398197830013, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.1158559322357178, + "step": 4420, + "unclipped_grad_norm": 0.6882814764976501 + }, + { + "bits_per_byte": 1.65897612304776, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.1499146223068237, + "step": 4430, + "unclipped_grad_norm": 0.7100256681442261 + }, + { + "bits_per_byte": 1.605377213958413, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.112762689590454, + "step": 4440, + "unclipped_grad_norm": 0.7294040322303772 + }, + { + "bits_per_byte": 1.5867810738859194, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.0998728275299072, + "step": 4450, + "unclipped_grad_norm": 0.7101866602897644 + }, + { + "bits_per_byte": 1.5848906405876353, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.098562479019165, + "step": 4460, + "unclipped_grad_norm": 0.7119124531745911 + }, + { + "bits_per_byte": 1.6342331510507786, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1327641010284424, + "step": 4470, + "unclipped_grad_norm": 0.7329701781272888 + }, + { + "bits_per_byte": 1.64634864087346, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.1411619186401367, + "step": 4480, + "unclipped_grad_norm": 0.7847994565963745 + }, + { + "bits_per_byte": 1.6229555607031008, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.1249470710754395, + "step": 4490, + "unclipped_grad_norm": 0.7213096022605896 + }, + { + "bits_per_byte": 1.6440263591388595, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.1395522356033325, + "step": 4500, + "unclipped_grad_norm": 0.7609583139419556 + }, + { + "bits_per_byte": 1.5904800767406766, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.1024367809295654, + "step": 4510, + "unclipped_grad_norm": 0.7066100239753723 + }, + { + "bits_per_byte": 1.6412930388686968, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.137657642364502, + "step": 4520, + "unclipped_grad_norm": 0.7309631109237671 + }, + { + "bits_per_byte": 1.6487988777003024, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.1428602933883667, + "step": 4530, + "unclipped_grad_norm": 0.7163345813751221 + }, + { + "bits_per_byte": 1.7034279628683175, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.1807262897491455, + "step": 4540, + "unclipped_grad_norm": 0.7811716198921204 + }, + { + "bits_per_byte": 1.5496087437284318, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.0741069316864014, + "step": 4550, + "unclipped_grad_norm": 0.6963047981262207 + }, + { + "bits_per_byte": 1.6360193628626398, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.1340022087097168, + "step": 4560, + "unclipped_grad_norm": 0.8012465238571167 + }, + { + "bits_per_byte": 1.6074582040338532, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.1142051219940186, + "step": 4570, + "unclipped_grad_norm": 0.7191913723945618 + }, + { + "bits_per_byte": 1.5703962867382195, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.0885157585144043, + "step": 4580, + "unclipped_grad_norm": 0.7079671621322632 + }, + { + "bits_per_byte": 1.6046404402821168, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.1122519969940186, + "step": 4590, + "unclipped_grad_norm": 0.7629184722900391 + }, + { + "bits_per_byte": 1.6134060520312925, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.1183278560638428, + "step": 4600, + "unclipped_grad_norm": 0.7373279929161072 + }, + { + "bits_per_byte": 1.6568449140382682, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.1484373807907104, + "step": 4610, + "unclipped_grad_norm": 0.8001417517662048 + }, + { + "bits_per_byte": 1.669816017548992, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.15742826461792, + "step": 4620, + "unclipped_grad_norm": 0.7757049798965454 + }, + { + "bits_per_byte": 1.6522538371734854, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.1452550888061523, + "step": 4630, + "unclipped_grad_norm": 0.7616952061653137 + }, + { + "bits_per_byte": 1.6954637902721625, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.175205945968628, + "step": 4640, + "unclipped_grad_norm": 0.733183741569519 + }, + { + "bits_per_byte": 1.597219216914736, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.1071079969406128, + "step": 4650, + "unclipped_grad_norm": 0.7152910828590393 + }, + { + "bits_per_byte": 1.5723536212876892, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.0898724794387817, + "step": 4660, + "unclipped_grad_norm": 0.7606828808784485 + }, + { + "bits_per_byte": 1.7091893816722217, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.1847198009490967, + "step": 4670, + "unclipped_grad_norm": 0.7436702251434326 + }, + { + "bits_per_byte": 1.701842798775315, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.179627537727356, + "step": 4680, + "unclipped_grad_norm": 0.7743264436721802 + }, + { + "bits_per_byte": 1.6736467591143103, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.160083532333374, + "step": 4690, + "unclipped_grad_norm": 0.796971321105957 + }, + { + "bits_per_byte": 1.5964458109338058, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.106571912765503, + "step": 4700, + "unclipped_grad_norm": 0.718313992023468 + }, + { + "bits_per_byte": 1.6637857898617894, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.1532484292984009, + "step": 4710, + "unclipped_grad_norm": 0.7606609463691711 + }, + { + "bits_per_byte": 1.6089547970616613, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.1152424812316895, + "step": 4720, + "unclipped_grad_norm": 0.7416554093360901 + }, + { + "bits_per_byte": 1.521603088861932, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.0546948909759521, + "step": 4730, + "unclipped_grad_norm": 0.7076453566551208 + }, + { + "bits_per_byte": 1.5766635065183112, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.0928598642349243, + "step": 4740, + "unclipped_grad_norm": 0.7157532572746277 + }, + { + "bits_per_byte": 1.6281109126453643, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.1285204887390137, + "step": 4750, + "unclipped_grad_norm": 0.7242756485939026 + }, + { + "bits_per_byte": 1.6012322600899687, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.1098896265029907, + "step": 4760, + "unclipped_grad_norm": 0.7614837884902954 + }, + { + "bits_per_byte": 1.5809303960862175, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.0958174467086792, + "step": 4770, + "unclipped_grad_norm": 0.7335332036018372 + }, + { + "bits_per_byte": 1.6132691538412056, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.1182329654693604, + "step": 4780, + "unclipped_grad_norm": 0.7687906622886658 + }, + { + "bits_per_byte": 1.5788953253785578, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.0944068431854248, + "step": 4790, + "unclipped_grad_norm": 0.751784086227417 + }, + { + "bits_per_byte": 1.5841191264158647, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.0980277061462402, + "step": 4800, + "unclipped_grad_norm": 0.7794368863105774 + }, + { + "bits_per_byte": 1.631063510795377, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.1305670738220215, + "step": 4810, + "unclipped_grad_norm": 0.7645516991615295 + }, + { + "bits_per_byte": 1.5681269756600849, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.0869427919387817, + "step": 4820, + "unclipped_grad_norm": 0.7387099266052246 + }, + { + "bits_per_byte": 1.6129957014263339, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.1180434226989746, + "step": 4830, + "unclipped_grad_norm": 0.7900767922401428 + }, + { + "bits_per_byte": 1.6036066525677806, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.1115354299545288, + "step": 4840, + "unclipped_grad_norm": 0.7677624225616455 + }, + { + "bits_per_byte": 1.6188881710101954, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.1221277713775635, + "step": 4850, + "unclipped_grad_norm": 0.7432094812393188 + }, + { + "bits_per_byte": 1.5547854215194024, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.0776951313018799, + "step": 4860, + "unclipped_grad_norm": 0.7294195294380188 + }, + { + "bits_per_byte": 1.6124811293349524, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.1176867485046387, + "step": 4870, + "unclipped_grad_norm": 0.7619287371635437 + }, + { + "bits_per_byte": 1.4852198151066032, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.0294759273529053, + "step": 4880, + "unclipped_grad_norm": 0.7347192764282227 + }, + { + "bits_per_byte": 1.5421178673874514, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.0689146518707275, + "step": 4890, + "unclipped_grad_norm": 0.7358648180961609 + }, + { + "bits_per_byte": 1.5853420950461503, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.0988754034042358, + "step": 4900, + "unclipped_grad_norm": 0.7333055138587952 + }, + { + "bits_per_byte": 1.7021448003102302, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.1798368692398071, + "step": 4910, + "unclipped_grad_norm": 0.7979458570480347 + }, + { + "bits_per_byte": 1.6229650197488983, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.1249536275863647, + "step": 4920, + "unclipped_grad_norm": 0.7752370238304138 + }, + { + "bits_per_byte": 1.5867110769470183, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.09982430934906, + "step": 4930, + "unclipped_grad_norm": 0.7365944385528564 + }, + { + "bits_per_byte": 1.5563109076325556, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.0787525177001953, + "step": 4940, + "unclipped_grad_norm": 0.7066284418106079 + }, + { + "bits_per_byte": 1.5303298325320127, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.060743808746338, + "step": 4950, + "unclipped_grad_norm": 0.7254693508148193 + }, + { + "bits_per_byte": 1.512894403370192, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.0486584901809692, + "step": 4960, + "unclipped_grad_norm": 0.7525309324264526 + }, + { + "bits_per_byte": 1.600197440479727, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.1091723442077637, + "step": 4970, + "unclipped_grad_norm": 0.7567957043647766 + }, + { + "bits_per_byte": 1.6496871680920089, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.1434760093688965, + "step": 4980, + "unclipped_grad_norm": 0.7351704239845276 + }, + { + "bits_per_byte": 1.5847750682462556, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.098482370376587, + "step": 4990, + "unclipped_grad_norm": 0.7969898581504822 + }, + { + "bits_per_byte": 1.545808959040269, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.0714731216430664, + "step": 5000, + "unclipped_grad_norm": 0.7163731455802917 + }, + { + "bits_per_byte": 1.5480677791767012, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.0730388164520264, + "step": 5010, + "unclipped_grad_norm": 0.7628421187400818 + }, + { + "bits_per_byte": 1.5831831968298693, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.0973789691925049, + "step": 5020, + "unclipped_grad_norm": 0.7119072079658508 + }, + { + "bits_per_byte": 1.57715159328146, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.0931981801986694, + "step": 5030, + "unclipped_grad_norm": 0.7844903469085693 + }, + { + "bits_per_byte": 1.548983070844593, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.0736732482910156, + "step": 5040, + "unclipped_grad_norm": 0.76976078748703 + }, + { + "bits_per_byte": 1.511416556451328, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.0476341247558594, + "step": 5050, + "unclipped_grad_norm": 0.70943284034729 + }, + { + "bits_per_byte": 1.5266934313621696, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.0582232475280762, + "step": 5060, + "unclipped_grad_norm": 0.7284770607948303 + }, + { + "bits_per_byte": 1.6001582284353304, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.109145164489746, + "step": 5070, + "unclipped_grad_norm": 0.7526230812072754 + }, + { + "bits_per_byte": 1.566026895510399, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.0854871273040771, + "step": 5080, + "unclipped_grad_norm": 0.7718839645385742 + }, + { + "bits_per_byte": 1.553912093618321, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.077089786529541, + "step": 5090, + "unclipped_grad_norm": 0.7798281908035278 + }, + { + "bits_per_byte": 1.5540562150797441, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.0771896839141846, + "step": 5100, + "unclipped_grad_norm": 0.7245976328849792 + }, + { + "bits_per_byte": 1.4841705489536892, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.028748631477356, + "step": 5110, + "unclipped_grad_norm": 0.748368501663208 + }, + { + "bits_per_byte": 1.5713501025199048, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.089176893234253, + "step": 5120, + "unclipped_grad_norm": 0.7235202193260193 + }, + { + "bits_per_byte": 1.5831219710061621, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.0973365306854248, + "step": 5130, + "unclipped_grad_norm": 0.7777110934257507 + }, + { + "bits_per_byte": 1.5408393483609373, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.068028450012207, + "step": 5140, + "unclipped_grad_norm": 0.7170120477676392 + }, + { + "bits_per_byte": 1.6338809305818116, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.1325199604034424, + "step": 5150, + "unclipped_grad_norm": 0.8047548532485962 + }, + { + "bits_per_byte": 1.5007888885412608, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.0402675867080688, + "step": 5160, + "unclipped_grad_norm": 0.7608790397644043 + }, + { + "bits_per_byte": 1.6724268861717404, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.1592379808425903, + "step": 5170, + "unclipped_grad_norm": 0.7964147329330444 + }, + { + "bits_per_byte": 1.5753916948151807, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.0919783115386963, + "step": 5180, + "unclipped_grad_norm": 0.7615920305252075 + }, + { + "bits_per_byte": 1.523376058009677, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.0559238195419312, + "step": 5190, + "unclipped_grad_norm": 0.7538801431655884 + }, + { + "bits_per_byte": 1.5502652015067753, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.0745619535446167, + "step": 5200, + "unclipped_grad_norm": 0.7872256636619568 + }, + { + "bits_per_byte": 1.5485015194221772, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.0733394622802734, + "step": 5210, + "unclipped_grad_norm": 0.7413632869720459 + }, + { + "bits_per_byte": 1.6345117629451764, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.1329572200775146, + "step": 5220, + "unclipped_grad_norm": 0.7880448698997498 + }, + { + "bits_per_byte": 1.540837284569127, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.0680270195007324, + "step": 5230, + "unclipped_grad_norm": 0.8009716868400574 + }, + { + "bits_per_byte": 1.5492568672247666, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0738630294799805, + "step": 5240, + "unclipped_grad_norm": 0.7578597664833069 + }, + { + "bits_per_byte": 1.5140736884071584, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.049475908279419, + "step": 5250, + "unclipped_grad_norm": 0.7606198787689209 + }, + { + "bits_per_byte": 1.54030173059434, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.0676558017730713, + "step": 5260, + "unclipped_grad_norm": 0.784372091293335 + }, + { + "bits_per_byte": 1.5392065450736458, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.066896677017212, + "step": 5270, + "unclipped_grad_norm": 0.71877521276474 + }, + { + "bits_per_byte": 1.5451067538767962, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.0709863901138306, + "step": 5280, + "unclipped_grad_norm": 0.7596135139465332 + }, + { + "bits_per_byte": 1.5595992159170524, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.0810317993164062, + "step": 5290, + "unclipped_grad_norm": 0.7750282287597656 + }, + { + "bits_per_byte": 1.5524867014079702, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.0761017799377441, + "step": 5300, + "unclipped_grad_norm": 0.7604010701179504 + }, + { + "bits_per_byte": 1.5429103634426273, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.0694639682769775, + "step": 5310, + "unclipped_grad_norm": 0.773445188999176 + }, + { + "bits_per_byte": 1.5479797240594593, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0729777812957764, + "step": 5320, + "unclipped_grad_norm": 0.7661488652229309 + }, + { + "bits_per_byte": 1.5433208860302368, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.0697485208511353, + "step": 5330, + "unclipped_grad_norm": 0.748544454574585 + }, + { + "bits_per_byte": 1.494452875700824, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.0358757972717285, + "step": 5340, + "unclipped_grad_norm": 0.7756491899490356 + }, + { + "bits_per_byte": 1.5812664501860034, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.0960503816604614, + "step": 5350, + "unclipped_grad_norm": 0.8031984567642212 + }, + { + "bits_per_byte": 1.4948907435299208, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.0361793041229248, + "step": 5360, + "unclipped_grad_norm": 0.7937144041061401 + }, + { + "bits_per_byte": 1.4756104564722947, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.022815227508545, + "step": 5370, + "unclipped_grad_norm": 0.7795528769493103 + }, + { + "bits_per_byte": 1.5882824824279518, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.1009135246276855, + "step": 5380, + "unclipped_grad_norm": 0.789566695690155 + }, + { + "bits_per_byte": 1.5348968318256746, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.0639094114303589, + "step": 5390, + "unclipped_grad_norm": 0.809786319732666 + }, + { + "bits_per_byte": 1.5434392100940304, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.0698305368423462, + "step": 5400, + "unclipped_grad_norm": 0.8030325174331665 + }, + { + "bits_per_byte": 1.5666205796211774, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.0858986377716064, + "step": 5410, + "unclipped_grad_norm": 0.7818418741226196 + }, + { + "bits_per_byte": 1.537330902283336, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.065596580505371, + "step": 5420, + "unclipped_grad_norm": 0.8286970853805542 + }, + { + "bits_per_byte": 1.5844857934275043, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.0982818603515625, + "step": 5430, + "unclipped_grad_norm": 0.8261771202087402 + }, + { + "bits_per_byte": 1.5170660145495203, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.051550030708313, + "step": 5440, + "unclipped_grad_norm": 0.821189284324646 + }, + { + "bits_per_byte": 1.515756710628502, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.050642490386963, + "step": 5450, + "unclipped_grad_norm": 0.7645202279090881 + }, + { + "bits_per_byte": 1.4938175717885367, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.035435438156128, + "step": 5460, + "unclipped_grad_norm": 0.7786561846733093 + }, + { + "bits_per_byte": 1.5422939776219349, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0690367221832275, + "step": 5470, + "unclipped_grad_norm": 0.8064988851547241 + }, + { + "bits_per_byte": 1.5553732582200515, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.0781025886535645, + "step": 5480, + "unclipped_grad_norm": 0.7898780703544617 + }, + { + "bits_per_byte": 1.5540899236793133, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.0772130489349365, + "step": 5490, + "unclipped_grad_norm": 0.8092604279518127 + }, + { + "bits_per_byte": 1.5405913493783931, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.0678565502166748, + "step": 5500, + "unclipped_grad_norm": 0.817954957485199 + }, + { + "bits_per_byte": 1.6356771173874227, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.1337649822235107, + "step": 5510, + "unclipped_grad_norm": 0.8459510207176208 + }, + { + "bits_per_byte": 1.5494489718457805, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.0739961862564087, + "step": 5520, + "unclipped_grad_norm": 0.8158296346664429 + }, + { + "bits_per_byte": 1.5129227805075844, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.0486781597137451, + "step": 5530, + "unclipped_grad_norm": 0.779143750667572 + }, + { + "bits_per_byte": 1.5902692260107187, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.1022906303405762, + "step": 5540, + "unclipped_grad_norm": 0.8701900839805603 + }, + { + "bits_per_byte": 1.5128725615735326, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.0486433506011963, + "step": 5550, + "unclipped_grad_norm": 0.7457809448242188 + }, + { + "bits_per_byte": 1.528391416074138, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.059400200843811, + "step": 5560, + "unclipped_grad_norm": 0.822853684425354 + }, + { + "bits_per_byte": 1.4870197855305334, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.0307235717773438, + "step": 5570, + "unclipped_grad_norm": 0.8034393191337585 + }, + { + "bits_per_byte": 1.5543282916334091, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.077378273010254, + "step": 5580, + "unclipped_grad_norm": 0.7782469391822815 + }, + { + "bits_per_byte": 1.497381224297065, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.0379055738449097, + "step": 5590, + "unclipped_grad_norm": 0.7850054502487183 + }, + { + "bits_per_byte": 1.5507458930159368, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.0748951435089111, + "step": 5600, + "unclipped_grad_norm": 0.7828869223594666 + }, + { + "bits_per_byte": 1.5482038174535335, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.073133111000061, + "step": 5610, + "unclipped_grad_norm": 0.7924382090568542 + }, + { + "bits_per_byte": 1.5682750527224778, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.087045431137085, + "step": 5620, + "unclipped_grad_norm": 0.8280176520347595 + }, + { + "bits_per_byte": 1.5440270467946795, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0702379941940308, + "step": 5630, + "unclipped_grad_norm": 0.7848796248435974 + }, + { + "bits_per_byte": 1.5130732653270895, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.048782467842102, + "step": 5640, + "unclipped_grad_norm": 0.7761145830154419 + }, + { + "bits_per_byte": 1.5331831966924776, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.062721610069275, + "step": 5650, + "unclipped_grad_norm": 0.8054314851760864 + }, + { + "bits_per_byte": 1.6054367199556117, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.1128039360046387, + "step": 5660, + "unclipped_grad_norm": 0.8357570767402649 + }, + { + "bits_per_byte": 1.504610858991385, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.0429167747497559, + "step": 5670, + "unclipped_grad_norm": 0.80352383852005 + }, + { + "bits_per_byte": 1.5237114241788594, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0561562776565552, + "step": 5680, + "unclipped_grad_norm": 0.7443665266036987 + }, + { + "bits_per_byte": 1.5913329387063053, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.1030279397964478, + "step": 5690, + "unclipped_grad_norm": 0.8210363388061523 + }, + { + "bits_per_byte": 1.5164768019876644, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.051141619682312, + "step": 5700, + "unclipped_grad_norm": 0.7999663352966309 + }, + { + "bits_per_byte": 1.516076254393805, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.0508639812469482, + "step": 5710, + "unclipped_grad_norm": 0.8051102161407471 + }, + { + "bits_per_byte": 1.5394727742171814, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0670812129974365, + "step": 5720, + "unclipped_grad_norm": 0.7637665271759033 + }, + { + "bits_per_byte": 1.5478730948159245, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.0729038715362549, + "step": 5730, + "unclipped_grad_norm": 0.7915765643119812 + }, + { + "bits_per_byte": 1.5318329609005537, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.0617856979370117, + "step": 5740, + "unclipped_grad_norm": 0.7885494828224182 + }, + { + "bits_per_byte": 1.5354263664076813, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.0642764568328857, + "step": 5750, + "unclipped_grad_norm": 0.8333257436752319 + }, + { + "bits_per_byte": 1.555185969113262, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.0779727697372437, + "step": 5760, + "unclipped_grad_norm": 0.8248261213302612 + }, + { + "bits_per_byte": 1.4991041464934087, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.0390998125076294, + "step": 5770, + "unclipped_grad_norm": 0.7788416743278503 + }, + { + "bits_per_byte": 1.4981652932023486, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.0384490489959717, + "step": 5780, + "unclipped_grad_norm": 0.8010360598564148 + }, + { + "bits_per_byte": 1.5075724002392425, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 1.0449695587158203, + "step": 5790, + "unclipped_grad_norm": 0.7946521043777466 + }, + { + "bits_per_byte": 1.489468474513518, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.0324208736419678, + "step": 5800, + "unclipped_grad_norm": 0.784231424331665 + }, + { + "bits_per_byte": 1.5676999427379925, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.0866467952728271, + "step": 5810, + "unclipped_grad_norm": 0.8325434327125549 + }, + { + "bits_per_byte": 1.4842359023610172, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.0287939310073853, + "step": 5820, + "unclipped_grad_norm": 0.7431433796882629 + }, + { + "bits_per_byte": 1.5035395790591606, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.042174220085144, + "step": 5830, + "unclipped_grad_norm": 0.8024688363075256 + }, + { + "bits_per_byte": 1.5003278030542975, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.0399479866027832, + "step": 5840, + "unclipped_grad_norm": 0.8202845454216003 + }, + { + "bits_per_byte": 1.4448463718686064, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.0014911890029907, + "step": 5850, + "unclipped_grad_norm": 0.8498019576072693 + }, + { + "bits_per_byte": 1.5306056927039968, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.0609350204467773, + "step": 5860, + "unclipped_grad_norm": 0.8159018158912659 + }, + { + "bits_per_byte": 1.5427610825016782, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0693604946136475, + "step": 5870, + "unclipped_grad_norm": 0.8125935792922974 + }, + { + "bits_per_byte": 1.5428677117452132, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.069434404373169, + "step": 5880, + "unclipped_grad_norm": 0.851513683795929 + }, + { + "bits_per_byte": 1.5508807274142131, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.074988603591919, + "step": 5890, + "unclipped_grad_norm": 0.817274808883667 + }, + { + "bits_per_byte": 1.5499320711120539, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.0743310451507568, + "step": 5900, + "unclipped_grad_norm": 0.8410898447036743 + }, + { + "bits_per_byte": 1.5619450592748212, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.082657814025879, + "step": 5910, + "unclipped_grad_norm": 0.8427087068557739 + }, + { + "bits_per_byte": 1.4941575814892925, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.0356711149215698, + "step": 5920, + "unclipped_grad_norm": 0.8013628721237183 + }, + { + "bits_per_byte": 1.5822992060044345, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.0967662334442139, + "step": 5930, + "unclipped_grad_norm": 0.8513590693473816 + }, + { + "bits_per_byte": 1.458890819103365, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.0112260580062866, + "step": 5940, + "unclipped_grad_norm": 0.7659777998924255 + }, + { + "bits_per_byte": 1.5070609238355765, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.0446150302886963, + "step": 5950, + "unclipped_grad_norm": 0.8170812726020813 + }, + { + "bits_per_byte": 1.5335545072356904, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.062978982925415, + "step": 5960, + "unclipped_grad_norm": 0.8636343479156494 + }, + { + "bits_per_byte": 1.523859845206554, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.0562591552734375, + "step": 5970, + "unclipped_grad_norm": 0.8548757433891296 + }, + { + "bits_per_byte": 1.5530504605374984, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.0764925479888916, + "step": 5980, + "unclipped_grad_norm": 0.8406524062156677 + }, + { + "bits_per_byte": 1.5280471068071106, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0591615438461304, + "step": 5990, + "unclipped_grad_norm": 0.836118757724762 + }, + { + "bits_per_byte": 1.4834490817333197, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.0282485485076904, + "step": 6000, + "unclipped_grad_norm": 0.8186500668525696 + }, + { + "bits_per_byte": 1.518377726227651, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0524592399597168, + "step": 6010, + "unclipped_grad_norm": 0.8388415575027466 + }, + { + "bits_per_byte": 1.4820114787547576, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0272520780563354, + "step": 6020, + "unclipped_grad_norm": 0.8129734396934509 + }, + { + "bits_per_byte": 1.5147464845373337, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.0499422550201416, + "step": 6030, + "unclipped_grad_norm": 0.8447511196136475 + }, + { + "bits_per_byte": 1.5666377778862637, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.0859105587005615, + "step": 6040, + "unclipped_grad_norm": 0.8490147590637207 + }, + { + "bits_per_byte": 1.4835182187589666, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.0282964706420898, + "step": 6050, + "unclipped_grad_norm": 0.8586865663528442 + }, + { + "bits_per_byte": 1.4810951551909606, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.0266169309616089, + "step": 6060, + "unclipped_grad_norm": 0.8068554401397705 + }, + { + "bits_per_byte": 1.5341194702437748, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 1.0633705854415894, + "step": 6070, + "unclipped_grad_norm": 0.875556230545044 + }, + { + "bits_per_byte": 1.4802790975126165, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.0260512828826904, + "step": 6080, + "unclipped_grad_norm": 0.8416600823402405 + }, + { + "bits_per_byte": 1.5028163920122823, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.041672945022583, + "step": 6090, + "unclipped_grad_norm": 0.8416868448257446 + }, + { + "bits_per_byte": 1.4890754941562965, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.0321484804153442, + "step": 6100, + "unclipped_grad_norm": 0.8068794012069702 + }, + { + "bits_per_byte": 1.558204436618555, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.0800650119781494, + "step": 6110, + "unclipped_grad_norm": 0.8628730177879333 + }, + { + "bits_per_byte": 1.5371320903389385, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0654587745666504, + "step": 6120, + "unclipped_grad_norm": 0.8250192403793335 + }, + { + "bits_per_byte": 1.471730183903528, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.0201256275177002, + "step": 6130, + "unclipped_grad_norm": 0.8464975953102112 + }, + { + "bits_per_byte": 1.481260086553138, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.026731252670288, + "step": 6140, + "unclipped_grad_norm": 0.8263784050941467 + }, + { + "bits_per_byte": 1.536146285784193, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.0647754669189453, + "step": 6150, + "unclipped_grad_norm": 0.845112144947052 + }, + { + "bits_per_byte": 1.4128025643598479, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 0.9792801141738892, + "step": 6160, + "unclipped_grad_norm": 0.8350163102149963 + }, + { + "bits_per_byte": 1.5097835811813856, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 1.0465022325515747, + "step": 6170, + "unclipped_grad_norm": 0.8165205121040344 + }, + { + "bits_per_byte": 1.4830332276835334, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0279603004455566, + "step": 6180, + "unclipped_grad_norm": 0.8218697905540466 + }, + { + "bits_per_byte": 1.513046264050904, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.0487637519836426, + "step": 6190, + "unclipped_grad_norm": 0.8420271873474121 + }, + { + "bits_per_byte": 1.5079499021578864, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.0452312231063843, + "step": 6200, + "unclipped_grad_norm": 0.829068660736084 + }, + { + "bits_per_byte": 1.504116924818107, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.042574405670166, + "step": 6210, + "unclipped_grad_norm": 0.8516910076141357 + }, + { + "bits_per_byte": 1.5126421048213763, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0484836101531982, + "step": 6220, + "unclipped_grad_norm": 0.8263625502586365 + }, + { + "bits_per_byte": 1.4536916115851315, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.007622241973877, + "step": 6230, + "unclipped_grad_norm": 0.8093429803848267 + }, + { + "bits_per_byte": 1.491850434227968, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.034071922302246, + "step": 6240, + "unclipped_grad_norm": 0.8618929982185364 + }, + { + "bits_per_byte": 1.5362708012234176, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.06486177444458, + "step": 6250, + "unclipped_grad_norm": 0.8253007531166077 + }, + { + "bits_per_byte": 1.4076165135144052, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 0.9756854176521301, + "step": 6260, + "unclipped_grad_norm": 0.8210739493370056 + }, + { + "bits_per_byte": 1.5053622511930045, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.0434376001358032, + "step": 6270, + "unclipped_grad_norm": 0.8734357357025146 + }, + { + "bits_per_byte": 1.502430806909048, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.0414056777954102, + "step": 6280, + "unclipped_grad_norm": 0.8565708994865417 + }, + { + "bits_per_byte": 1.4622835208569356, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 1.0135776996612549, + "step": 6290, + "unclipped_grad_norm": 0.831123948097229 + }, + { + "bits_per_byte": 1.5126936996166351, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.0485193729400635, + "step": 6300, + "unclipped_grad_norm": 0.8611893653869629 + }, + { + "bits_per_byte": 1.477229673130168, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.0239375829696655, + "step": 6310, + "unclipped_grad_norm": 0.8457248210906982 + }, + { + "bits_per_byte": 1.4308884319072317, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 0.9918162822723389, + "step": 6320, + "unclipped_grad_norm": 0.7951399087905884 + }, + { + "bits_per_byte": 1.5110622721905507, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 1.0473885536193848, + "step": 6330, + "unclipped_grad_norm": 0.8688725829124451 + }, + { + "bits_per_byte": 1.4543420499706945, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 1.008073091506958, + "step": 6340, + "unclipped_grad_norm": 0.8249567151069641 + }, + { + "bits_per_byte": 1.5884348590566162, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.1010191440582275, + "step": 6350, + "unclipped_grad_norm": 0.9589917659759521 + }, + { + "bits_per_byte": 1.4579653804590722, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.0105845928192139, + "step": 6360, + "unclipped_grad_norm": 0.8681607246398926 + }, + { + "bits_per_byte": 1.4348239109069252, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 0.9945441484451294, + "step": 6370, + "unclipped_grad_norm": 0.802064061164856 + }, + { + "bits_per_byte": 1.4802718742412802, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.0260462760925293, + "step": 6380, + "unclipped_grad_norm": 0.8152594566345215 + }, + { + "bits_per_byte": 1.5283737018610992, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0593879222869873, + "step": 6390, + "unclipped_grad_norm": 0.854841947555542 + }, + { + "bits_per_byte": 1.5295027679640136, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.0601705312728882, + "step": 6400, + "unclipped_grad_norm": 0.8946288824081421 + }, + { + "bits_per_byte": 1.4872130940301032, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.0308575630187988, + "step": 6410, + "unclipped_grad_norm": 0.8961010575294495 + }, + { + "bits_per_byte": 1.4762688060597975, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0232715606689453, + "step": 6420, + "unclipped_grad_norm": 0.8554261922836304 + }, + { + "bits_per_byte": 1.5208351863258296, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.054162621498108, + "step": 6430, + "unclipped_grad_norm": 0.8489636778831482 + }, + { + "bits_per_byte": 1.4515358090565662, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.006127953529358, + "step": 6440, + "unclipped_grad_norm": 0.8574435114860535 + }, + { + "bits_per_byte": 1.5236080626056907, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0560846328735352, + "step": 6450, + "unclipped_grad_norm": 0.9158070683479309 + }, + { + "bits_per_byte": 1.5725536371106426, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.0900111198425293, + "step": 6460, + "unclipped_grad_norm": 0.8968605995178223 + }, + { + "bits_per_byte": 1.5675038825160088, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.0865108966827393, + "step": 6470, + "unclipped_grad_norm": 0.8953620791435242 + }, + { + "bits_per_byte": 1.4569379561028177, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 1.0098724365234375, + "step": 6480, + "unclipped_grad_norm": 0.8446434736251831 + }, + { + "bits_per_byte": 1.55000103615505, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0743788480758667, + "step": 6490, + "unclipped_grad_norm": 0.9021739959716797 + }, + { + "bits_per_byte": 1.5171040227153612, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.0515763759613037, + "step": 6500, + "unclipped_grad_norm": 0.8947733044624329 + }, + { + "bits_per_byte": 1.4604054703095137, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 1.0122759342193604, + "step": 6510, + "unclipped_grad_norm": 0.8653041124343872 + }, + { + "bits_per_byte": 1.4216102258671595, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 0.9853851199150085, + "step": 6520, + "unclipped_grad_norm": 0.834362268447876 + }, + { + "bits_per_byte": 1.4736784033725019, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.0214760303497314, + "step": 6530, + "unclipped_grad_norm": 0.8773725032806396 + }, + { + "bits_per_byte": 1.4974202643588108, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.0379326343536377, + "step": 6540, + "unclipped_grad_norm": 0.8439525365829468 + }, + { + "bits_per_byte": 1.435593877234838, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 0.9950778484344482, + "step": 6550, + "unclipped_grad_norm": 0.8415363430976868 + }, + { + "bits_per_byte": 1.5073125344537888, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.044789433479309, + "step": 6560, + "unclipped_grad_norm": 0.8618022799491882 + }, + { + "bits_per_byte": 1.4712658307461983, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 1.019803762435913, + "step": 6570, + "unclipped_grad_norm": 0.8514965176582336 + }, + { + "bits_per_byte": 1.501176021488353, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.0405359268188477, + "step": 6580, + "unclipped_grad_norm": 0.904940664768219 + }, + { + "bits_per_byte": 1.4867824494723427, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.0305590629577637, + "step": 6590, + "unclipped_grad_norm": 0.8391159176826477 + }, + { + "bits_per_byte": 1.4269765145307063, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9891047477722168, + "step": 6600, + "unclipped_grad_norm": 0.8761541843414307 + }, + { + "bits_per_byte": 1.4371100762848446, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 0.9961287975311279, + "step": 6610, + "unclipped_grad_norm": 0.8312658667564392 + }, + { + "bits_per_byte": 1.4346115123331098, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 0.9943969249725342, + "step": 6620, + "unclipped_grad_norm": 0.8476570844650269 + }, + { + "bits_per_byte": 1.4666164517627733, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.0165810585021973, + "step": 6630, + "unclipped_grad_norm": 0.862541913986206 + }, + { + "bits_per_byte": 1.5125382273002552, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.0484116077423096, + "step": 6640, + "unclipped_grad_norm": 0.8855039477348328 + }, + { + "bits_per_byte": 1.5033240847976295, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.042024850845337, + "step": 6650, + "unclipped_grad_norm": 0.8511106967926025 + }, + { + "bits_per_byte": 1.5117594898571485, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.0478718280792236, + "step": 6660, + "unclipped_grad_norm": 0.8526108264923096 + }, + { + "bits_per_byte": 1.4847781636591875, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.0291697978973389, + "step": 6670, + "unclipped_grad_norm": 0.9127287268638611 + }, + { + "bits_per_byte": 1.4579413028879515, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.0105679035186768, + "step": 6680, + "unclipped_grad_norm": 0.8571484088897705 + }, + { + "bits_per_byte": 1.500717171775851, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.0402178764343262, + "step": 6690, + "unclipped_grad_norm": 0.8760013580322266 + }, + { + "bits_per_byte": 1.5456352565628975, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.0713527202606201, + "step": 6700, + "unclipped_grad_norm": 0.9106501340866089 + }, + { + "bits_per_byte": 1.5164198757302287, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.0511021614074707, + "step": 6710, + "unclipped_grad_norm": 0.8854292631149292 + }, + { + "bits_per_byte": 1.5229777461902785, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.0556477308273315, + "step": 6720, + "unclipped_grad_norm": 0.8762082457542419 + }, + { + "bits_per_byte": 1.474139660842116, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 1.0217957496643066, + "step": 6730, + "unclipped_grad_norm": 0.8872997760772705 + }, + { + "bits_per_byte": 1.4335132311246999, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 0.9936356544494629, + "step": 6740, + "unclipped_grad_norm": 0.8565142154693604 + }, + { + "bits_per_byte": 1.4980283950122617, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 1.0383541584014893, + "step": 6750, + "unclipped_grad_norm": 0.8804352879524231 + }, + { + "bits_per_byte": 1.479008661670693, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 1.0251706838607788, + "step": 6760, + "unclipped_grad_norm": 0.8506757020950317 + }, + { + "bits_per_byte": 1.3778143278769925, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 0.9550281167030334, + "step": 6770, + "unclipped_grad_norm": 0.831382155418396 + }, + { + "bits_per_byte": 1.4751172102296202, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 1.0224733352661133, + "step": 6780, + "unclipped_grad_norm": 0.8634604811668396 + }, + { + "bits_per_byte": 1.4470271118815468, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.0030027627944946, + "step": 6790, + "unclipped_grad_norm": 0.8902454972267151 + }, + { + "bits_per_byte": 1.5055180674746862, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0435456037521362, + "step": 6800, + "unclipped_grad_norm": 0.8958489298820496 + }, + { + "bits_per_byte": 1.4961685746258313, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.037065029144287, + "step": 6810, + "unclipped_grad_norm": 0.9283017516136169 + }, + { + "bits_per_byte": 1.4853408908928107, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 1.029559850692749, + "step": 6820, + "unclipped_grad_norm": 0.9333142638206482 + }, + { + "bits_per_byte": 1.4657145747416487, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.015955924987793, + "step": 6830, + "unclipped_grad_norm": 0.8490593433380127 + }, + { + "bits_per_byte": 1.4370137660003615, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 0.9960620403289795, + "step": 6840, + "unclipped_grad_norm": 0.8986743688583374 + }, + { + "bits_per_byte": 1.4920808909801242, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.0342316627502441, + "step": 6850, + "unclipped_grad_norm": 0.8717131018638611 + }, + { + "bits_per_byte": 1.4594891467457167, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.0116407871246338, + "step": 6860, + "unclipped_grad_norm": 0.8995506167411804 + }, + { + "bits_per_byte": 1.4048853430273784, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 0.973792314529419, + "step": 6870, + "unclipped_grad_norm": 0.8454365730285645 + }, + { + "bits_per_byte": 1.4161775519003796, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 0.9816194772720337, + "step": 6880, + "unclipped_grad_norm": 0.7990046739578247 + }, + { + "bits_per_byte": 1.4263389748439577, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 0.988662838935852, + "step": 6890, + "unclipped_grad_norm": 0.8933730125427246 + }, + { + "bits_per_byte": 1.5292186526247882, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 1.0599735975265503, + "step": 6900, + "unclipped_grad_norm": 0.886705756187439 + }, + { + "bits_per_byte": 1.4980559122364, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.0383732318878174, + "step": 6910, + "unclipped_grad_norm": 0.9239560961723328 + }, + { + "bits_per_byte": 1.439153488151071, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 0.9975451827049255, + "step": 6920, + "unclipped_grad_norm": 0.8407799005508423 + }, + { + "bits_per_byte": 1.433509877463008, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.9936333298683167, + "step": 6930, + "unclipped_grad_norm": 0.919253945350647 + }, + { + "bits_per_byte": 1.4113962622237426, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 0.9783053398132324, + "step": 6940, + "unclipped_grad_norm": 0.8700326085090637 + }, + { + "bits_per_byte": 1.4249663813074216, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 0.9877114295959473, + "step": 6950, + "unclipped_grad_norm": 0.9012777209281921 + }, + { + "bits_per_byte": 1.4921331737059864, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 1.0342679023742676, + "step": 6960, + "unclipped_grad_norm": 0.8893595337867737 + }, + { + "bits_per_byte": 1.4768642099970846, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 1.0236842632293701, + "step": 6970, + "unclipped_grad_norm": 0.8872846364974976 + }, + { + "bits_per_byte": 1.4490473920812321, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.0044031143188477, + "step": 6980, + "unclipped_grad_norm": 0.8574569225311279 + }, + { + "bits_per_byte": 1.4852600590469052, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.0295038223266602, + "step": 6990, + "unclipped_grad_norm": 0.9787434339523315 + }, + { + "bits_per_byte": 1.5264186030860907, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 1.058032751083374, + "step": 7000, + "unclipped_grad_norm": 0.871791660785675 + }, + { + "bits_per_byte": 1.5228485872194806, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 1.055558204650879, + "step": 7010, + "unclipped_grad_norm": 0.9414756894111633 + }, + { + "bits_per_byte": 1.460263412639901, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 1.0121774673461914, + "step": 7020, + "unclipped_grad_norm": 0.8892385959625244 + }, + { + "bits_per_byte": 1.5088211662671573, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 1.0458351373672485, + "step": 7030, + "unclipped_grad_norm": 0.9397335648536682 + }, + { + "bits_per_byte": 1.4974728910499748, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.0379691123962402, + "step": 7040, + "unclipped_grad_norm": 0.8716332912445068 + }, + { + "bits_per_byte": 1.4854363412640395, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.0296260118484497, + "step": 7050, + "unclipped_grad_norm": 0.8889445066452026 + }, + { + "bits_per_byte": 1.48319798706306, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 1.0280745029449463, + "step": 7060, + "unclipped_grad_norm": 0.9145016074180603 + }, + { + "bits_per_byte": 1.4622532519103837, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 1.013556718826294, + "step": 7070, + "unclipped_grad_norm": 0.9412884712219238 + }, + { + "bits_per_byte": 1.4229886668138247, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 0.9863405823707581, + "step": 7080, + "unclipped_grad_norm": 0.8688827157020569 + }, + { + "bits_per_byte": 1.4612093172196465, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 1.0128331184387207, + "step": 7090, + "unclipped_grad_norm": 0.9287014603614807 + }, + { + "bits_per_byte": 1.460298669083328, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 1.0122019052505493, + "step": 7100, + "unclipped_grad_norm": 0.8761314749717712 + }, + { + "bits_per_byte": 1.4674756770864839, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 1.017176628112793, + "step": 7110, + "unclipped_grad_norm": 0.8686223030090332 + }, + { + "bits_per_byte": 1.5133561767877588, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.048978567123413, + "step": 7120, + "unclipped_grad_norm": 0.911088764667511 + }, + { + "bits_per_byte": 1.4736061706591395, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 1.0214259624481201, + "step": 7130, + "unclipped_grad_norm": 0.977756679058075 + }, + { + "bits_per_byte": 1.4635850855586654, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.0144798755645752, + "step": 7140, + "unclipped_grad_norm": 0.9076306819915771 + }, + { + "bits_per_byte": 1.436832668269003, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 0.9959365129470825, + "step": 7150, + "unclipped_grad_norm": 0.8820585012435913 + }, + { + "bits_per_byte": 1.5138293010602824, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0493065118789673, + "step": 7160, + "unclipped_grad_norm": 0.9440810084342957 + }, + { + "bits_per_byte": 1.44811008663403, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.003753423690796, + "step": 7170, + "unclipped_grad_norm": 0.887324333190918 + }, + { + "bits_per_byte": 1.4742652081772458, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 1.0218827724456787, + "step": 7180, + "unclipped_grad_norm": 0.9199241399765015 + }, + { + "bits_per_byte": 1.4308750172604643, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 0.9918069839477539, + "step": 7190, + "unclipped_grad_norm": 0.900752067565918 + }, + { + "bits_per_byte": 1.4865029776646908, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 1.0303653478622437, + "step": 7200, + "unclipped_grad_norm": 0.892007052898407 + }, + { + "bits_per_byte": 1.5336568369129537, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 1.0630499124526978, + "step": 7210, + "unclipped_grad_norm": 0.9430897831916809 + }, + { + "bits_per_byte": 1.5072394418271722, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 1.04473876953125, + "step": 7220, + "unclipped_grad_norm": 0.9257102608680725 + }, + { + "bits_per_byte": 1.441053294503827, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 0.9988620281219482, + "step": 7230, + "unclipped_grad_norm": 0.9163823127746582 + }, + { + "bits_per_byte": 1.4539550890062534, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 1.0078048706054688, + "step": 7240, + "unclipped_grad_norm": 0.8671759366989136 + }, + { + "bits_per_byte": 1.5020022261430979, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 1.0411086082458496, + "step": 7250, + "unclipped_grad_norm": 0.945029616355896 + }, + { + "bits_per_byte": 1.3868825431003613, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 0.9613137245178223, + "step": 7260, + "unclipped_grad_norm": 0.8957721590995789 + }, + { + "bits_per_byte": 1.4217346553150587, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 0.9854713678359985, + "step": 7270, + "unclipped_grad_norm": 0.9168151021003723 + }, + { + "bits_per_byte": 1.4888973201300026, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.0320249795913696, + "step": 7280, + "unclipped_grad_norm": 0.9551942348480225 + }, + { + "bits_per_byte": 1.4189836208318571, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 0.9835644960403442, + "step": 7290, + "unclipped_grad_norm": 0.9004480242729187 + }, + { + "bits_per_byte": 1.4463531118728155, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 1.0025355815887451, + "step": 7300, + "unclipped_grad_norm": 0.9373093843460083 + }, + { + "bits_per_byte": 1.451019689121327, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 1.005770206451416, + "step": 7310, + "unclipped_grad_norm": 0.9348265528678894 + }, + { + "bits_per_byte": 1.4062352348540006, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 0.974727988243103, + "step": 7320, + "unclipped_grad_norm": 0.8961029648780823 + }, + { + "bits_per_byte": 1.426095447410336, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 0.9884940385818481, + "step": 7330, + "unclipped_grad_norm": 0.9225013852119446 + }, + { + "bits_per_byte": 1.3983553337567685, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.9692660570144653, + "step": 7340, + "unclipped_grad_norm": 0.9267120957374573 + }, + { + "bits_per_byte": 1.4697059481028727, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 1.0187225341796875, + "step": 7350, + "unclipped_grad_norm": 0.9416163563728333 + }, + { + "bits_per_byte": 1.4281009371020472, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 0.9898841381072998, + "step": 7360, + "unclipped_grad_norm": 0.9168581962585449 + }, + { + "bits_per_byte": 1.458665177865433, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 1.011069655418396, + "step": 7370, + "unclipped_grad_norm": 0.9165750741958618 + }, + { + "bits_per_byte": 1.4006749497602806, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 0.9708738923072815, + "step": 7380, + "unclipped_grad_norm": 0.9057982563972473 + }, + { + "bits_per_byte": 1.4775941043673464, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 1.0241901874542236, + "step": 7390, + "unclipped_grad_norm": 0.9100049138069153 + }, + { + "bits_per_byte": 1.405927557891607, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.9745147228240967, + "step": 7400, + "unclipped_grad_norm": 0.9113430976867676 + }, + { + "bits_per_byte": 1.41443949523076, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 0.9804147481918335, + "step": 7410, + "unclipped_grad_norm": 0.9049294590950012 + }, + { + "bits_per_byte": 1.4680574943943527, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0175799131393433, + "step": 7420, + "unclipped_grad_norm": 0.9212203025817871 + }, + { + "bits_per_byte": 1.4932624117915516, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 1.035050630569458, + "step": 7430, + "unclipped_grad_norm": 0.9374892115592957 + }, + { + "bits_per_byte": 1.4998624180010627, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 1.0396254062652588, + "step": 7440, + "unclipped_grad_norm": 0.9596559405326843 + }, + { + "bits_per_byte": 1.457143303387948, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 1.0100147724151611, + "step": 7450, + "unclipped_grad_norm": 0.9177109599113464 + }, + { + "bits_per_byte": 1.4507226750832867, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 1.0055643320083618, + "step": 7460, + "unclipped_grad_norm": 0.9369127750396729 + }, + { + "bits_per_byte": 1.4228539184068738, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 0.986247181892395, + "step": 7470, + "unclipped_grad_norm": 0.9056556820869446 + }, + { + "bits_per_byte": 1.461617432050144, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 1.0131160020828247, + "step": 7480, + "unclipped_grad_norm": 0.9412074089050293 + }, + { + "bits_per_byte": 1.5217437706703378, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 1.0547924041748047, + "step": 7490, + "unclipped_grad_norm": 0.9305914044380188 + }, + { + "bits_per_byte": 1.4756734021225104, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.0228588581085205, + "step": 7500, + "unclipped_grad_norm": 0.9498947858810425 + }, + { + "bits_per_byte": 1.4402614863792547, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 0.9983131885528564, + "step": 7510, + "unclipped_grad_norm": 0.8852996230125427 + }, + { + "bits_per_byte": 1.4488121198148518, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 1.0042400360107422, + "step": 7520, + "unclipped_grad_norm": 0.9398816823959351 + }, + { + "bits_per_byte": 1.4941945577592282, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.0356967449188232, + "step": 7530, + "unclipped_grad_norm": 0.9878621697425842 + }, + { + "bits_per_byte": 1.4486946556643125, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.004158616065979, + "step": 7540, + "unclipped_grad_norm": 0.9093635678291321 + }, + { + "bits_per_byte": 1.4801702324946204, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 1.0259758234024048, + "step": 7550, + "unclipped_grad_norm": 0.9258415699005127 + }, + { + "bits_per_byte": 1.4457804096454423, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 1.002138614654541, + "step": 7560, + "unclipped_grad_norm": 0.9157785177230835 + }, + { + "bits_per_byte": 1.3973001342023996, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 0.9685346484184265, + "step": 7570, + "unclipped_grad_norm": 0.9060177803039551 + }, + { + "bits_per_byte": 1.5787606629629323, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 1.0943135023117065, + "step": 7580, + "unclipped_grad_norm": 1.018172025680542 + }, + { + "bits_per_byte": 1.4404157548170786, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 0.9984201192855835, + "step": 7590, + "unclipped_grad_norm": 0.9298149943351746 + }, + { + "bits_per_byte": 1.417510761409868, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 0.9825435876846313, + "step": 7600, + "unclipped_grad_norm": 0.9303708672523499 + }, + { + "bits_per_byte": 1.442996698458577, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 1.000209093093872, + "step": 7610, + "unclipped_grad_norm": 0.9450833201408386 + }, + { + "bits_per_byte": 1.4012220265726751, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 0.9712530970573425, + "step": 7620, + "unclipped_grad_norm": 0.8888522982597351 + }, + { + "bits_per_byte": 1.4096005053747587, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 0.9770606160163879, + "step": 7630, + "unclipped_grad_norm": 0.9539974927902222 + }, + { + "bits_per_byte": 1.4168209389972573, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.9820654392242432, + "step": 7640, + "unclipped_grad_norm": 0.9127889275550842 + }, + { + "bits_per_byte": 1.5657469077547943, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 1.0852930545806885, + "step": 7650, + "unclipped_grad_norm": 0.9893083572387695 + }, + { + "bits_per_byte": 1.470483997615376, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 1.0192618370056152, + "step": 7660, + "unclipped_grad_norm": 0.9705356359481812 + }, + { + "bits_per_byte": 1.482172798481267, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 1.027363896369934, + "step": 7670, + "unclipped_grad_norm": 0.9572810530662537 + }, + { + "bits_per_byte": 1.4313922550829343, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 0.9921655058860779, + "step": 7680, + "unclipped_grad_norm": 0.9474135637283325 + }, + { + "bits_per_byte": 1.441330960493645, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 0.999054491519928, + "step": 7690, + "unclipped_grad_norm": 0.9126761555671692 + }, + { + "bits_per_byte": 1.4000399038219697, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 0.9704337120056152, + "step": 7700, + "unclipped_grad_norm": 0.9203900098800659 + }, + { + "bits_per_byte": 1.479325625696233, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 1.025390386581421, + "step": 7710, + "unclipped_grad_norm": 0.9269430041313171 + }, + { + "bits_per_byte": 1.487660764870299, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 1.0311678647994995, + "step": 7720, + "unclipped_grad_norm": 0.9559805393218994 + }, + { + "bits_per_byte": 1.4747367846059116, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 1.022209644317627, + "step": 7730, + "unclipped_grad_norm": 0.9193494915962219 + }, + { + "bits_per_byte": 1.4197359589380565, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 0.9840859770774841, + "step": 7740, + "unclipped_grad_norm": 0.9774965643882751 + }, + { + "bits_per_byte": 1.4403299354742982, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9983606338500977, + "step": 7750, + "unclipped_grad_norm": 0.9309295415878296 + }, + { + "bits_per_byte": 1.4635539566988591, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 1.0144582986831665, + "step": 7760, + "unclipped_grad_norm": 0.9333544969558716 + }, + { + "bits_per_byte": 1.3843541401587018, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 0.9595611691474915, + "step": 7770, + "unclipped_grad_norm": 0.9147452116012573 + }, + { + "bits_per_byte": 1.475734111998265, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 1.022900938987732, + "step": 7780, + "unclipped_grad_norm": 0.9597358703613281 + }, + { + "bits_per_byte": 1.4130438560190084, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 0.9794473648071289, + "step": 7790, + "unclipped_grad_norm": 0.9232692718505859 + }, + { + "bits_per_byte": 1.4432274991760348, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 1.0003690719604492, + "step": 7800, + "unclipped_grad_norm": 0.9291107058525085 + }, + { + "bits_per_byte": 1.4130027521654522, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9794188737869263, + "step": 7810, + "unclipped_grad_norm": 0.9548395872116089 + }, + { + "bits_per_byte": 1.3971521431313323, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 0.9684320688247681, + "step": 7820, + "unclipped_grad_norm": 0.9297690987586975 + }, + { + "bits_per_byte": 1.4245129490484219, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 0.9873971343040466, + "step": 7830, + "unclipped_grad_norm": 0.9460906386375427 + }, + { + "bits_per_byte": 1.436919175542387, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.9959964752197266, + "step": 7840, + "unclipped_grad_norm": 0.9562280774116516 + }, + { + "bits_per_byte": 1.373255153793945, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.951867938041687, + "step": 7850, + "unclipped_grad_norm": 0.9045599699020386 + }, + { + "bits_per_byte": 1.456991786672538, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 1.009909749031067, + "step": 7860, + "unclipped_grad_norm": 0.9479863047599792 + }, + { + "bits_per_byte": 1.467501818449415, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0171947479248047, + "step": 7870, + "unclipped_grad_norm": 0.976621150970459 + }, + { + "bits_per_byte": 1.3715153772978168, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9506620168685913, + "step": 7880, + "unclipped_grad_norm": 0.9740973114967346 + }, + { + "bits_per_byte": 1.4661631054950988, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 1.0162668228149414, + "step": 7890, + "unclipped_grad_norm": 0.9810726046562195 + }, + { + "bits_per_byte": 1.441722822963636, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 0.9993261098861694, + "step": 7900, + "unclipped_grad_norm": 0.9438725709915161 + }, + { + "bits_per_byte": 1.4487646526032136, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.0042071342468262, + "step": 7910, + "unclipped_grad_norm": 0.937912106513977 + }, + { + "bits_per_byte": 1.4543117810241428, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 1.008052110671997, + "step": 7920, + "unclipped_grad_norm": 0.9619103670120239 + }, + { + "bits_per_byte": 1.417985347534924, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.9828725457191467, + "step": 7930, + "unclipped_grad_norm": 0.9347084760665894 + }, + { + "bits_per_byte": 1.4201617019902675, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 0.9843810796737671, + "step": 7940, + "unclipped_grad_norm": 0.9456982016563416 + }, + { + "bits_per_byte": 1.4770365366132492, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 1.0238037109375, + "step": 7950, + "unclipped_grad_norm": 0.9922394156455994 + }, + { + "bits_per_byte": 1.4095551879462562, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 0.9770292043685913, + "step": 7960, + "unclipped_grad_norm": 0.9342193603515625 + }, + { + "bits_per_byte": 1.4320214536111158, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 0.9926016330718994, + "step": 7970, + "unclipped_grad_norm": 0.9633327722549438 + }, + { + "bits_per_byte": 1.4838786943951752, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 1.0285463333129883, + "step": 7980, + "unclipped_grad_norm": 1.0099246501922607 + }, + { + "bits_per_byte": 1.5144307243903496, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 1.0497233867645264, + "step": 7990, + "unclipped_grad_norm": 1.0371782779693604 + }, + { + "bits_per_byte": 1.4192880301238844, + "learning_rate": 3e-05, + "loss_nats": 0.9837754964828491, + "step": 8000, + "unclipped_grad_norm": 0.9597862958908081 + } + ], + "variant": "uniform_group_7_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073002.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073002.json new file mode 100644 index 0000000..27763ed --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073002.json @@ -0,0 +1,14045 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "86ff6909fb3db6f59740a3fab91e20198e19ffc9cda7f2e0d63cc464912ff3cd", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.00040311424527317286, + 0.00035970969474874437, + 0.00032544726855121553, + 0.00029308689408935606, + 0.00026978139067068696, + 0.00025443214690312743, + 0.00023969457834027708, + 0.0002233884297311306, + 0.0002095247182296589, + 0.0001984660339076072, + 0.0001901928335428238, + 0.00018121300672646612, + 0.0001734284160193056, + 0.0001677417167229578, + 0.00016109299031086266, + 0.00015624657680746168, + 0.0001518603676231578, + 0.00014820833166595548, + 0.00014440594532061368, + 0.0001414585713064298, + 0.00013923499500378966, + 0.00013687640603166074, + 0.00013471563579514623, + 0.00013265138841234148, + 0.00013098257477395236, + 0.0001293915556743741, + 0.00012844328011851758, + 0.0001269879430765286, + 0.0001256384130101651, + 0.00012456141121219844, + 0.0001231025526067242, + 0.00012163266364950687 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00029608183103846386, + "first_to_last_ratio": 2.343484699021421, + "imbalance_abs_log_ratio": 0.8516390092453431, + "last_quartile_mean": 0.0001263425492652459, + "mean": 0.0001858347804954974, + "normalized": [ + 2.16920774571012, + 1.9356424765570717, + 1.751272112160408, + 1.5771369240348272, + 1.451727119925344, + 1.369130935687747, + 1.289826251583108, + 1.2020808437231323, + 1.127478492836466, + 1.067970341065492, + 1.0234512239081746, + 0.9751296621832141, + 0.9332398141881068, + 0.9026389800429313, + 0.8668613586828855, + 0.8407822065969367, + 0.8171794710239251, + 0.7975274126338607, + 0.777066300159633, + 0.7612061150730458, + 0.7492407752334779, + 0.7365489154759011, + 0.7249215428669998, + 0.7138135717041164, + 0.7048334785593375, + 0.6962720074755282, + 0.6911692191098191, + 0.683337870004401, + 0.6760758813563923, + 0.6702804011180051, + 0.662430102042749, + 0.6545204472768429 + ], + "population_cv": 0.39438035124793563 + }, + "activation_output_rms_by_block": [ + 0.0036361459642648697, + 0.005152894649654627, + 0.006386642809957266, + 0.007284422870725393, + 0.0037505170330405235, + 0.0054016136564314365, + 0.006585344672203064, + 0.007922882214188576, + 0.0044892593286931515, + 0.006135277915745974, + 0.007316684816032648, + 0.0082859443500638, + 0.0044554853811860085, + 0.006464940495789051, + 0.007893112488090992, + 0.009196000173687935, + 0.004961193539202213, + 0.006907128728926182, + 0.008217990398406982, + 0.009676996618509293, + 0.005381465889513493, + 0.007502692751586437, + 0.0091153085231781, + 0.010543142445385456, + 0.006136673036962748, + 0.008469858206808567, + 0.010564134456217289, + 0.012301022186875343, + 0.005773921962827444, + 0.008156762458384037, + 0.010457510128617287, + 0.012218860909342766 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.005765057983808219, + "first_to_last_ratio": 0.622587017371887, + "imbalance_abs_log_ratio": 0.4738718734697682, + "last_quartile_mean": 0.009259842918254435, + "mean": 0.007398182220640592, + "normalized": [ + 0.49149180917985336, + 0.6965082091758006, + 0.8632718983507628, + 0.9846233376628902, + 0.5069511565390686, + 0.730127144119427, + 0.8901300989627224, + 1.0709228264321597, + 0.6068057253534959, + 0.8292953231982888, + 0.9889841312125875, + 1.119997332175246, + 0.6022405569783624, + 0.8738552664669654, + 1.0668989020126547, + 1.2430080659586231, + 0.6705962885532487, + 0.9336251153230055, + 1.1108121094231989, + 1.3080235563150786, + 0.7274038039370601, + 1.01412651484229, + 1.232101109614035, + 1.4250990487866828, + 0.8294838994154143, + 1.1448566626512724, + 1.427936504016329, + 1.6627087330393202, + 0.7804514393709395, + 1.1025360304896303, + 1.4135242707919937, + 1.6516031296515918 + ], + "population_cv": 0.30590027610070303 + }, + "bits_per_byte": 8.038491115106986, + "branch_output_rms_by_sublayer": [ + 0.0023513585329055786, + 0.0027412420604377985, + 0.0023208872880786657, + 0.002683233004063368, + 0.002778682392090559, + 0.002711885841563344, + 0.002497476991266012, + 0.0027249811682850122, + 0.002628475893288851, + 0.0027012599166482687, + 0.002920303260907531, + 0.002700963756069541, + 0.002932395786046982, + 0.0026683746837079525, + 0.0032228652853518724, + 0.002738022478297353, + 0.003561305580660701, + 0.0027370047755539417, + 0.003354643704369664, + 0.002654933836311102, + 0.0034162327647209167, + 0.002663377905264497, + 0.0036349669098854065, + 0.002684312639757991, + 0.0035210708156228065, + 0.0027356327045708895, + 0.0038089773152023554, + 0.002652956871315837, + 0.003915763925760984, + 0.002679797587916255, + 0.0037244565319269896, + 0.002698024269193411, + 0.004162859171628952, + 0.0026484597474336624, + 0.004167158622294664, + 0.0026680538430809975, + 0.004440344404429197, + 0.002654345240443945, + 0.0041700867004692554, + 0.002591830212622881, + 0.004737259820103645, + 0.002620960585772991, + 0.004552462603896856, + 0.002651234855875373, + 0.004706811159849167, + 0.0026270023081451654, + 0.005186605267226696, + 0.002615687670186162, + 0.005580419674515724, + 0.00258000404573977, + 0.004905648995190859, + 0.002626793459057808, + 0.005227713380008936, + 0.002573782578110695, + 0.005349088925868273, + 0.002592637902125716, + 0.005235649179667234, + 0.002581534907221794, + 0.004925078712403774, + 0.002551236655563116, + 0.005810525268316269, + 0.002563327783718705, + 0.005564783234149218, + 0.002589041367173195 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.010012756197717748, + 0.008904953530422711, + 0.008980942528705073, + 0.007787912846174446, + 0.007907551845183929, + 0.008289745083748344, + 0.007663833460373458, + 0.007350080301903749, + 0.00709564946584838, + 0.006310668475383808, + 0.006285078227083486, + 0.006362421438950662, + 0.005604026016423943, + 0.005959901601525332, + 0.006167718108848569, + 0.005523196426059091, + 0.0054641843609823, + 0.00557989785603791, + 0.005260407000687673, + 0.004763357766419076, + 0.004970214176820314, + 0.004845025235106639, + 0.004921304265254701, + 0.004820106373346748, + 0.0054804096177630225, + 0.0042302744374162005, + 0.004588429583035324, + 0.004469681656036737, + 0.004349111730454971, + 0.003928502929307311, + 0.004468266803351117, + 0.003904642007649288 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.008362221974278682, + "first_to_last_ratio": 1.8887369414995314, + "imbalance_abs_log_ratio": 0.6359083207487641, + "last_quartile_mean": 0.004427414845626747, + "mean": 0.00600782035481319, + "normalized": [ + 1.6666204390908572, + 1.4822269982304765, + 1.4948753455169401, + 1.2962958920592769, + 1.3162097696294732, + 1.3798257261648947, + 1.2756429133626719, + 1.2234187888150154, + 1.1810688480662828, + 1.0504089840715678, + 1.0461494944748422, + 1.0590232502297414, + 0.9327885465041001, + 0.9920239370590588, + 1.0266149359654666, + 0.9193344840336578, + 0.9095119424808776, + 0.9287724210274617, + 0.875593258455752, + 0.7928595538984271, + 0.8272907449435313, + 0.8064530809788657, + 0.8191497039874, + 0.8023053434820334, + 0.9122126318860999, + 0.7041279844573081, + 0.7637428072161453, + 0.7439772483303089, + 0.7239084182952745, + 0.6538982022256998, + 0.7437417465006834, + 0.6499265585598059 + ], + "population_cv": 0.2648157248292805 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.027978340163826942, + 0.014031016267836094, + 0.0140878576785326, + 0.014151837676763535, + 0.014187770895659924, + 0.014265215955674648, + 0.014337530359625816, + 0.014401593245565891, + 0.014477464370429516, + 0.009722241200506687, + 0.009757626801729202, + 0.009786579757928848, + 0.009825476445257664, + 0.009839320555329323, + 0.00987118761986494, + 0.00996283907443285, + 0.009999874979257584, + 0.007562503684312105, + 0.007595349568873644, + 0.007635751739144325, + 0.007658981718122959, + 0.007660485804080963, + 0.007689184043556452, + 0.007726665120571852, + 0.007755559403449297, + 0.006238248664885759, + 0.00625597033649683, + 0.006290202960371971, + 0.006315375678241253, + 0.006327399518340826, + 0.00635211868211627, + 0.006382675375789404, + 0.006408242508769035, + 0.005390914157032967, + 0.005408597644418478, + 0.0054517327807843685, + 0.005475745070725679, + 0.005509276874363422, + 0.005517482291907072, + 0.0055727302096784115, + 0.005585856735706329, + 0.0048178439028561115, + 0.004830927588045597, + 0.004884926602244377, + 0.004899685736745596, + 0.004938251338899136, + 0.004948681220412254, + 0.004975184798240662, + 0.004989070352166891, + 0.004412420094013214, + 0.004419817589223385, + 0.004455919377505779, + 0.004463871940970421, + 0.004530393984168768, + 0.004542885348200798, + 0.0045819394290447235, + 0.004592607729136944, + 0.004127359017729759, + 0.004137600306421518, + 0.004185030702501535, + 0.004193917382508516, + 0.004221425391733646, + 0.004230021964758635, + 0.004281661007553339 + ], + "loss_nats": 5.571857452392578, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0023513585329055786, + 0.0036361459642648697, + 0.004408807028084993, + 0.005152894649654627, + 0.005783648230135441, + 0.006386642809957266, + 0.006751363631337881, + 0.007284422870725393, + 0.002628475893288851, + 0.0037505170330405235, + 0.004723479505628347, + 0.0054016136564314365, + 0.006024772301316261, + 0.006585344672203064, + 0.007445912808179855, + 0.007922882214188576, + 0.003561305580660701, + 0.0044892593286931515, + 0.005543973762542009, + 0.006135277915745974, + 0.006855612155050039, + 0.007316684816032648, + 0.007876631803810596, + 0.0082859443500638, + 0.0035210708156228065, + 0.0044554853811860085, + 0.0058924350887537, + 0.006464940495789051, + 0.007390674669295549, + 0.007893112488090992, + 0.008790239691734314, + 0.009196000173687935, + 0.004162859171628952, + 0.004961193539202213, + 0.006355749908834696, + 0.006907128728926182, + 0.007784162648022175, + 0.008217990398406982, + 0.00937180407345295, + 0.009676996618509293, + 0.004737259820103645, + 0.005381465889513493, + 0.006999304983764887, + 0.007502692751586437, + 0.008728929795324802, + 0.0091153085231781, + 0.010217384435236454, + 0.010543142445385456, + 0.005580419674515724, + 0.006136673036962748, + 0.008008211851119995, + 0.008469858206808567, + 0.010209310799837112, + 0.010564134456217289, + 0.012040933594107628, + 0.012301022186875343, + 0.005235649179667234, + 0.005773921962827444, + 0.007745372597128153, + 0.008156762458384037, + 0.010180563665926456, + 0.010457510128617287, + 0.011906592175364494, + 0.012218860909342766 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.926399510120973e-05, + 6.577204476343468e-05, + 6.218211638042703e-05, + 5.8770092437043786e-05, + 5.367739868233912e-05, + 5.0109330913983285e-05, + 4.671733040595427e-05, + 4.3298852688167244e-05, + 3.890254447469488e-05, + 3.6236666346667334e-05, + 3.3639971661614254e-05, + 3.115744766546413e-05, + 2.8600141376955435e-05, + 2.6927489670924842e-05, + 2.5347570044687018e-05, + 2.3944694476085715e-05, + 2.2843018086859956e-05, + 2.1806372387800366e-05, + 2.097480683005415e-05, + 2.0194695025566034e-05, + 1.9649985915748402e-05, + 1.9083467122982256e-05, + 1.8597838788991794e-05, + 1.8172058844356798e-05, + 1.7718870367389172e-05, + 1.734611032588873e-05, + 1.7022190149873495e-05, + 1.675012390478514e-05, + 1.6364345356123522e-05, + 1.6174701158888638e-05, + 1.6004853023332544e-05, + 1.5865722161834128e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.622389517156989e-05, + "first_to_last_ratio": 3.3756215405382535, + "imbalance_abs_log_ratio": 1.216579467528575, + "last_quartile_mean": 1.665586455601442e-05, + "mean": 3.0909901454378996e-05, + "normalized": [ + 2.2408351965611693, + 2.127863295213346, + 2.011721599054717, + 1.901335483834674, + 1.7365761829284208, + 1.6211417234034666, + 1.5114034082219903, + 1.400808499893587, + 1.258578728635305, + 1.1723319920689588, + 1.0883234846692948, + 1.0080086379909847, + 0.9252744276511973, + 0.8711606444513602, + 0.820046938101643, + 0.7746609775326048, + 0.739019440763173, + 0.7054817829162332, + 0.6785788968305706, + 0.6533406473447381, + 0.6357181676800394, + 0.6173900991288572, + 0.6016790062058591, + 0.5879041339286563, + 0.5732425382701745, + 0.5611829708189288, + 0.5507034752277402, + 0.541901562821455, + 0.5294208194185359, + 0.5232854327523964, + 0.5177904901105772, + 0.513289315569346 + ], + "population_cv": 0.5273304217602091 + }, + "activation_output_rms_by_block": [ + 0.005867320112884045, + 0.009899391792714596, + 0.014524620026350021, + 0.01919795759022236, + 0.006809636950492859, + 0.012263516895473003, + 0.017802543938159943, + 0.02365284040570259, + 0.008289473131299019, + 0.01478586532175541, + 0.02155701257288456, + 0.028366420418024063, + 0.009277361445128918, + 0.01674789935350418, + 0.025121228769421577, + 0.03308790177106857, + 0.010398848913609982, + 0.019822876900434494, + 0.02951648458838463, + 0.04015320539474487, + 0.013488546945154667, + 0.024839792400598526, + 0.0367128849029541, + 0.05055355280637741, + 0.015256829559803009, + 0.029597369953989983, + 0.04400104284286499, + 0.060339901596307755, + 0.0194606501609087, + 0.037548135966062546, + 0.05599934607744217, + 0.0798909142613411 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013752228463999927, + "first_to_last_ratio": 0.32160098240001844, + "imbalance_abs_log_ratio": 1.1344436869555659, + "last_quartile_mean": 0.04276177380234003, + "mean": 0.02608848043018952, + "normalized": [ + 0.2249007997451012, + 0.37945451898605204, + 0.556744577945681, + 0.7358787201728519, + 0.26102083518106195, + 0.4700740209185083, + 0.6823909880760585, + 0.9066392528685414, + 0.31774457517680726, + 0.5667583959641147, + 0.826303878854471, + 1.0873159321766597, + 0.3556114151590516, + 0.6419653071906617, + 0.9629241855095308, + 1.268295478520065, + 0.39859925691863834, + 0.7598325610983273, + 1.1313991501869243, + 1.5391162970258585, + 0.5170307631082167, + 0.9521364215546255, + 1.4072450482961074, + 1.937772993013306, + 0.5848109705212208, + 1.1344995747525404, + 1.6866081165826392, + 2.312894450014903, + 0.7459480138363631, + 1.4392611354478102, + 2.1465162076914175, + 3.062306157505883 + ], + "population_cv": 0.654673769961395 + }, + "bits_per_byte": 5.11867904972417, + "branch_output_rms_by_sublayer": [ + 0.003081807866692543, + 0.004967149347066879, + 0.002654623705893755, + 0.004918999969959259, + 0.0032724784687161446, + 0.005560901015996933, + 0.0027460670098662376, + 0.005834068637341261, + 0.00310681015253067, + 0.0061525688506662846, + 0.0030510802753269672, + 0.0061499434523284435, + 0.0035158160608261824, + 0.006374158896505833, + 0.003831089474260807, + 0.0064691416919231415, + 0.004720454569905996, + 0.006541545502841473, + 0.004049105104058981, + 0.007037522736936808, + 0.004866583272814751, + 0.006699826568365097, + 0.005043732933700085, + 0.0067327627912163734, + 0.005189232993870974, + 0.006989984307438135, + 0.005120361223816872, + 0.00668152142316103, + 0.006588839925825596, + 0.006939259823411703, + 0.005846239626407623, + 0.007169775664806366, + 0.00616056052967906, + 0.007252488750964403, + 0.007113628555089235, + 0.007557970471680164, + 0.0075523825362324715, + 0.007292622234672308, + 0.008080934174358845, + 0.007722324691712856, + 0.008936326019465923, + 0.007893937639892101, + 0.008840988390147686, + 0.007910637184977531, + 0.009113340638577938, + 0.008089525625109673, + 0.011403516866266727, + 0.00818744208663702, + 0.010449781082570553, + 0.0077924239449203014, + 0.011243577115237713, + 0.008265088312327862, + 0.011091372929513454, + 0.008234383538365364, + 0.012929096817970276, + 0.008376178331673145, + 0.013472795486450195, + 0.009116982109844685, + 0.013874825090169907, + 0.009178484790027142, + 0.015348496846854687, + 0.008168955333530903, + 0.020384609699249268, + 0.008922582492232323 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00029030723080141445, + 0.0002270586409557858, + 0.0002319042157513621, + 0.00020120955231584675, + 0.00020774704069727038, + 0.0001908707895647932, + 0.00021057005546742315, + 0.00020421352922141305, + 0.00020930190742115979, + 0.00018010399049308616, + 0.00019000275907744827, + 0.00018968784095296372, + 0.00018597699234360767, + 0.00017180065794073537, + 0.00020683654354420236, + 0.0001791648291160963, + 0.00017662239563642637, + 0.0001836325612556074, + 0.00017410405821042212, + 0.0001724716108434136, + 0.00017784778360235653, + 0.00016744282622143235, + 0.00016227303348456386, + 0.00016787071404028009, + 0.00016274994272864193, + 0.00015524812472521982, + 0.0001455257764068362, + 0.00015853283543903026, + 0.00015105313865365248, + 0.00014735127243844647, + 0.000147458843578028, + 0.00017255729977418862 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00022048513184691362, + "first_to_last_ratio": 1.4219374663181144, + "imbalance_abs_log_ratio": 0.3520203545477774, + "last_quartile_mean": 0.0001550596542180055, + "mean": 0.00018435933727197358, + "normalized": [ + 1.5746814622854859, + 1.2316091189936351, + 1.2578924354086205, + 1.0913987612084715, + 1.1268593377008858, + 1.0353193518113688, + 1.1421719050594252, + 1.1076929015000192, + 1.1352932296148908, + 0.9769181922551079, + 1.0306109898689282, + 1.0289028142530656, + 1.008774467816879, + 0.9318793421745101, + 1.1219206285118588, + 0.9718240028807754, + 0.9580333616401897, + 0.9960578290900775, + 0.9443734219633123, + 0.9355187179316946, + 0.9646800983016609, + 0.9082416366815999, + 0.8801997006811366, + 0.9105625813388075, + 0.8827865468432842, + 0.8420952636436969, + 0.7893594030019283, + 0.8599121573384529, + 0.8193408638197338, + 0.7992612395924842, + 0.7998447258491247, + 0.9359835109388891 + ], + "population_cv": 0.15907041222818682 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6923078298568726, + "mean_weights": [ + 0.48022064566612244, + 0.5197793245315552 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927924156188965, + "mean_weights": [ + 0.5129886865615845, + 0.4870113134384155 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928653120994568, + "mean_weights": [ + 0.5081620812416077, + 0.49183791875839233 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6926438808441162, + "mean_weights": [ + 0.5150425434112549, + 0.48495739698410034 + ], + "sources": 2 + }, + { + "entropy_mean": 0.692533016204834, + "mean_weights": [ + 0.5161668062210083, + 0.4838332235813141 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928011178970337, + "mean_weights": [ + 0.5107715129852295, + 0.4892284572124481 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6924368143081665, + "mean_weights": [ + 0.5170955657958984, + 0.48290443420410156 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929628849029541, + "mean_weights": [ + 0.5073082447052002, + 0.4926917254924774 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0978710651397705, + "mean_weights": [ + 0.34931546449661255, + 0.32299599051475525, + 0.3276885449886322 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0983192920684814, + "mean_weights": [ + 0.34330591559410095, + 0.3268829584121704, + 0.329811155796051 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0981428623199463, + "mean_weights": [ + 0.3443334102630615, + 0.3302147090435028, + 0.32545188069343567 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098449468612671, + "mean_weights": [ + 0.34008580446243286, + 0.32977747917175293, + 0.3301366865634918 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982602834701538, + "mean_weights": [ + 0.343481183052063, + 0.33094125986099243, + 0.3255775570869446 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098495602607727, + "mean_weights": [ + 0.33823129534721375, + 0.3338133692741394, + 0.32795533537864685 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0980889797210693, + "mean_weights": [ + 0.344626784324646, + 0.3318534791469574, + 0.3235197067260742 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985524654388428, + "mean_weights": [ + 0.33589303493499756, + 0.3345220983028412, + 0.32958489656448364 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3860301971435547, + "mean_weights": [ + 0.25750914216041565, + 0.2500179409980774, + 0.24574515223503113, + 0.24672774970531464 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862011432647705, + "mean_weights": [ + 0.25315457582473755, + 0.2523956894874573, + 0.246758371591568, + 0.24769136309623718 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3858857154846191, + "mean_weights": [ + 0.2581929862499237, + 0.2524409890174866, + 0.244889035820961, + 0.24447700381278992 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861610889434814, + "mean_weights": [ + 0.2537955641746521, + 0.25338372588157654, + 0.247650608420372, + 0.24517011642456055 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3855738639831543, + "mean_weights": [ + 0.2592603862285614, + 0.2544667720794678, + 0.24628230929374695, + 0.23999053239822388 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861018419265747, + "mean_weights": [ + 0.2514016926288605, + 0.2558872699737549, + 0.24896535277366638, + 0.24374568462371826 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3856632709503174, + "mean_weights": [ + 0.25744929909706116, + 0.2521619200706482, + 0.2460113912820816, + 0.24437737464904785 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862519264221191, + "mean_weights": [ + 0.2500326335430145, + 0.25259697437286377, + 0.25027787685394287, + 0.24709251523017883 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6089379787445068, + "mean_weights": [ + 0.20497840642929077, + 0.20672225952148438, + 0.20000134408473969, + 0.19576416909694672, + 0.19253379106521606 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093038320541382, + "mean_weights": [ + 0.20156770944595337, + 0.20420178771018982, + 0.20098158717155457, + 0.19671964645385742, + 0.19652926921844482 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085598468780518, + "mean_weights": [ + 0.20554512739181519, + 0.20929095149040222, + 0.20159050822257996, + 0.19413088262081146, + 0.18944251537322998 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093418598175049, + "mean_weights": [ + 0.20106612145900726, + 0.20403799414634705, + 0.20018501579761505, + 0.19849810004234314, + 0.1962127536535263 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6080281734466553, + "mean_weights": [ + 0.20544177293777466, + 0.2129155397415161, + 0.2034081518650055, + 0.1930021345615387, + 0.18523238599300385 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609381914138794, + "mean_weights": [ + 0.2011038064956665, + 0.20251014828681946, + 0.19947662949562073, + 0.19839151203632355, + 0.19851791858673096 + ], + "sources": 5 + }, + { + "entropy_mean": 1.607670783996582, + "mean_weights": [ + 0.20587989687919617, + 0.21559011936187744, + 0.20290350914001465, + 0.19257184863090515, + 0.18305464088916779 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093558073043823, + "mean_weights": [ + 0.20061834156513214, + 0.2033236026763916, + 0.20117148756980896, + 0.19842398166656494, + 0.19646257162094116 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7901573181152344, + "mean_weights": [ + 0.17128190398216248, + 0.1812058985233307, + 0.1714954972267151, + 0.16200610995292664, + 0.1543060839176178, + 0.1597045212984085 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791698694229126, + "mean_weights": [ + 0.16690760850906372, + 0.16927064955234528, + 0.16836006939411163, + 0.16567635536193848, + 0.1651538908481598, + 0.16463139653205872 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7896721363067627, + "mean_weights": [ + 0.17207267880439758, + 0.18267722427845, + 0.17347368597984314, + 0.16240294277668, + 0.15663090348243713, + 0.15274259448051453 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917170524597168, + "mean_weights": [ + 0.16645413637161255, + 0.16856080293655396, + 0.1666550636291504, + 0.16677668690681458, + 0.16459819674491882, + 0.1669551134109497 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7894011735916138, + "mean_weights": [ + 0.17308178544044495, + 0.18356293439865112, + 0.1726442277431488, + 0.163834810256958, + 0.1555432677268982, + 0.15133295953273773 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791741132736206, + "mean_weights": [ + 0.1661975383758545, + 0.1674206405878067, + 0.16676664352416992, + 0.1668645292520523, + 0.1653316617012024, + 0.16741898655891418 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7883213758468628, + "mean_weights": [ + 0.17293024063110352, + 0.18678990006446838, + 0.17593230307102203, + 0.16286009550094604, + 0.15400847792625427, + 0.14747895300388336 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917324304580688, + "mean_weights": [ + 0.16663050651550293, + 0.16805237531661987, + 0.16745169460773468, + 0.16644707322120667, + 0.1653808057308197, + 0.16603754460811615 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9417405128479004, + "mean_weights": [ + 0.15035754442214966, + 0.16412562131881714, + 0.1530984342098236, + 0.14178363978862762, + 0.13223418593406677, + 0.12711435556411743, + 0.13128620386123657 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458882808685303, + "mean_weights": [ + 0.14314928650856018, + 0.1433088183403015, + 0.14312919974327087, + 0.14236611127853394, + 0.14220046997070312, + 0.14389349520206451, + 0.14195260405540466 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9410626888275146, + "mean_weights": [ + 0.15053829550743103, + 0.16551050543785095, + 0.154804065823555, + 0.14239895343780518, + 0.13261198997497559, + 0.12792108952999115, + 0.12621507048606873 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459002017974854, + "mean_weights": [ + 0.14251340925693512, + 0.14261993765830994, + 0.143624410033226, + 0.1420992612838745, + 0.14353276789188385, + 0.14268305897712708, + 0.1429271399974823 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9416556358337402, + "mean_weights": [ + 0.15045669674873352, + 0.16387400031089783, + 0.15373434126377106, + 0.14195328950881958, + 0.1341332495212555, + 0.12903481721878052, + 0.1268136352300644 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458903074264526, + "mean_weights": [ + 0.14262178540229797, + 0.1436314433813095, + 0.14290504157543182, + 0.14244627952575684, + 0.14210966229438782, + 0.14198371767997742, + 0.14430207014083862 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9401353597640991, + "mean_weights": [ + 0.1507047414779663, + 0.16667938232421875, + 0.15622127056121826, + 0.1427082121372223, + 0.13368895649909973, + 0.12734976410865784, + 0.12264766544103622 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079289436340332, + "mean_weights": [ + 0.1242607980966568, + 0.12249665707349777, + 0.12262935191392899, + 0.1231529712677002, + 0.12555232644081116, + 0.12621420621871948, + 0.12732592225074768, + 0.12836779654026031 + ], + "sources": 8 + }, + { + "entropy_mean": 2.190579414367676, + "mean_weights": [ + 0.1189664974808693, + 0.13226917386054993, + 0.12553352117538452, + 0.11686883866786957, + 0.10872945189476013, + 0.10385753959417343, + 0.09846515208482742, + 0.09485052525997162, + 0.10045930743217468 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197192668914795, + "mean_weights": [ + 0.11042745411396027, + 0.1100408136844635, + 0.11079590022563934, + 0.1104060709476471, + 0.1112930029630661, + 0.11132574081420898, + 0.11097690463066101, + 0.11236201226711273, + 0.11237211525440216 + ], + "sources": 9 + }, + { + "entropy_mean": 2.190338373184204, + "mean_weights": [ + 0.11927518248558044, + 0.13219580054283142, + 0.12625843286514282, + 0.11612206697463989, + 0.11070695519447327, + 0.10494448244571686, + 0.10013142228126526, + 0.09518042206764221, + 0.09518522769212723 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197110414505005, + "mean_weights": [ + 0.11036710441112518, + 0.10918071866035461, + 0.10962090641260147, + 0.10945728421211243, + 0.11106371879577637, + 0.1117025762796402, + 0.11177483201026917, + 0.1131560355424881, + 0.11367681622505188 + ], + "sources": 9 + }, + { + "entropy_mean": 2.189465045928955, + "mean_weights": [ + 0.11986248195171356, + 0.13407397270202637, + 0.12663333117961884, + 0.11625789105892181, + 0.10988449305295944, + 0.10405769944190979, + 0.10018329322338104, + 0.09442566335201263, + 0.09462115913629532 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1969709396362305, + "mean_weights": [ + 0.10976533591747284, + 0.10752047598361969, + 0.10839568078517914, + 0.11001709848642349, + 0.11048836261034012, + 0.11146146059036255, + 0.11390301585197449, + 0.11433165520429611, + 0.11411691457033157 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1906051635742188, + "mean_weights": [ + 0.11944625526666641, + 0.13184162974357605, + 0.12487303465604782, + 0.11625587195158005, + 0.1106325164437294, + 0.1050422340631485, + 0.10128623247146606, + 0.09609198570251465, + 0.09453022480010986 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028569549322128296, + 0.013707922771573067, + 0.015033164992928505, + 0.014950621873140335, + 0.015675773844122887, + 0.01578313298523426, + 0.016540130600333214, + 0.01675545424222946, + 0.017751751467585564, + 0.012115918099880219, + 0.012944274581968784, + 0.013099806383252144, + 0.01404075138270855, + 0.014190138317644596, + 0.015272458083927631, + 0.015522830188274384, + 0.016727697104215622, + 0.01280202716588974, + 0.013760232366621494, + 0.01393538061529398, + 0.015030239708721638, + 0.015212027356028557, + 0.016274183988571167, + 0.01645084097981453, + 0.017627451568841934, + 0.014370152726769447, + 0.015271984972059727, + 0.015505253337323666, + 0.016413388773798943, + 0.01673721894621849, + 0.017764968797564507, + 0.017884712666273117, + 0.019052358344197273, + 0.016093486919999123, + 0.017061937600374222, + 0.017343224957585335, + 0.01837259531021118, + 0.018591875210404396, + 0.019714361056685448, + 0.019787346944212914, + 0.021149588748812675, + 0.018277853727340698, + 0.019465044140815735, + 0.019502343609929085, + 0.020828787237405777, + 0.02076481096446514, + 0.02221948467195034, + 0.022036166861653328, + 0.023862091824412346, + 0.02151009812951088, + 0.022239817306399345, + 0.02299020253121853, + 0.02375924214720726, + 0.02449645847082138, + 0.025248456746339798, + 0.026197096332907677, + 0.027227070182561874, + 0.023571351543068886, + 0.025639232248067856, + 0.02510184608399868, + 0.027453068643808365, + 0.02654949203133583, + 0.02936660125851631, + 0.028482547029852867 + ], + "loss_nats": 3.5479979515075684, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1936135292053223, + "mean_weights": [ + 0.10072851181030273, + 0.09496782720088959, + 0.10225237905979156, + 0.11062133312225342, + 0.11559101939201355, + 0.11889128386974335, + 0.11974428594112396, + 0.1193113625049591, + 0.11789201200008392 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.003081807866692543, + 0.005867320112884045, + 0.006682500708848238, + 0.009899391792714596, + 0.01063432078808546, + 0.014524620026350021, + 0.014863750897347927, + 0.01919795759022236, + 0.00310681015253067, + 0.006809636950492859, + 0.007718713488429785, + 0.012263516895473003, + 0.012948796153068542, + 0.017802543938159943, + 0.01853540539741516, + 0.02365284040570259, + 0.004720454569905996, + 0.008289473131299019, + 0.009587250649929047, + 0.01478586532175541, + 0.01634249836206436, + 0.02155701257288456, + 0.023112906143069267, + 0.028366420418024063, + 0.005189232993870974, + 0.009277361445128918, + 0.011948572471737862, + 0.01674789935350418, + 0.019886920228600502, + 0.025121228769421577, + 0.027501115575432777, + 0.03308790177106857, + 0.00616056052967906, + 0.010398848913609982, + 0.014274678193032742, + 0.019822876900434494, + 0.024045025929808617, + 0.02951648458838463, + 0.034214604645967484, + 0.04015320539474487, + 0.008936326019465923, + 0.013488546945154667, + 0.01910671778023243, + 0.024839792400598526, + 0.030611615628004074, + 0.0367128849029541, + 0.04405519738793373, + 0.05055355280637741, + 0.010449781082570553, + 0.015256829559803009, + 0.02353988029062748, + 0.029597369953989983, + 0.037702690809965134, + 0.04400104284286499, + 0.05370606482028961, + 0.060339901596307755, + 0.013472795486450195, + 0.0194606501609087, + 0.030457759276032448, + 0.037548135966062546, + 0.04991046339273453, + 0.05599934607744217, + 0.07273774594068527, + 0.0798909142613411 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00014674535486847162, + 0.00015406608872581273, + 0.00015301661915145814, + 0.00014467330765910447, + 0.00013958175259176642, + 0.0001245148596353829, + 0.00011333916336297989, + 0.00010458521137479693, + 9.559052705299109e-05, + 8.698612509761006e-05, + 8.05820818641223e-05, + 7.492824079236016e-05, + 7.638182432856411e-05, + 6.906630733283237e-05, + 6.281030073296279e-05, + 5.820171281811781e-05, + 5.882260302314535e-05, + 5.38319036422763e-05, + 4.864278162131086e-05, + 4.433374124346301e-05, + 4.5786706323269755e-05, + 4.13609086535871e-05, + 3.7367106415331364e-05, + 3.502814070088789e-05, + 2.9256816560518928e-05, + 2.7571915779844858e-05, + 2.628043876029551e-05, + 2.5460676624788903e-05, + 2.5792392989387736e-05, + 2.5549903512001038e-05, + 2.543673872423824e-05, + 2.5420929887332022e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00013506529467122164, + "first_to_last_ratio": 5.126551771425573, + "imbalance_abs_log_ratio": 1.6344332638628496, + "last_quartile_mean": 2.6346226604800904e-05, + "mean": 7.065666193284414e-05, + "normalized": [ + 2.0768792475357274, + 2.180489206697103, + 2.165636119307381, + 2.0475536729517407, + 1.9754931633259487, + 1.7622522240539533, + 1.6040831856832334, + 1.4801889661048562, + 1.3528876745386786, + 1.2311100286663175, + 1.140473943429592, + 1.0604554297169593, + 1.0810279206391247, + 0.977491795444238, + 0.8889508648549098, + 0.8237257637989723, + 0.832513190037954, + 0.761880085609493, + 0.6884387160483685, + 0.6274530954434298, + 0.6480168333849097, + 0.5853787530027772, + 0.5288546811176426, + 0.4957514230460038, + 0.41407017767590215, + 0.3902238660248438, + 0.37194566006067253, + 0.3603436099059999, + 0.365038374072953, + 0.36160643332238124, + 0.36000481806534634, + 0.35978107643258644 + ], + "population_cv": 0.6028793211295248 + }, + "activation_output_rms_by_block": [ + 0.021257206797599792, + 0.039118532091379166, + 0.049036867916584015, + 0.058654993772506714, + 0.019036665558815002, + 0.029960447922348976, + 0.0357137955725193, + 0.04143182933330536, + 0.026369430124759674, + 0.03188152238726616, + 0.03730445355176926, + 0.04281774163246155, + 0.027975991368293762, + 0.03802811726927757, + 0.04204143211245537, + 0.05121698975563049, + 0.02476053684949875, + 0.04124032333493233, + 0.04760165512561798, + 0.057892896234989166, + 0.025168301537632942, + 0.03825869783759117, + 0.05622135475277901, + 0.07485418766736984, + 0.02300684154033661, + 0.042030565440654755, + 0.06508335471153259, + 0.08959902822971344, + 0.029342075809836388, + 0.05616040900349617, + 0.09125300496816635, + 0.12482858449220657 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.03677629237063229, + "first_to_last_ratio": 0.5643739844876218, + "imbalance_abs_log_ratio": 0.5720381541699868, + "last_quartile_mean": 0.06516298302449286, + "mean": 0.046223369834478945, + "normalized": [ + 0.4598800752459119, + 0.8462933843088147, + 1.060867437665837, + 1.2689467250558346, + 0.4118407123276237, + 0.6481666747715324, + 0.7726350480375332, + 0.8963394378572659, + 0.5704783147396186, + 0.68972735007055, + 0.8070474672303773, + 0.9263223729855137, + 0.6052347863964238, + 0.8227032647219855, + 0.9095276320831074, + 1.108032364127349, + 0.5356713917259526, + 0.8921963821030274, + 1.0298179318399876, + 1.2524594472946817, + 0.5444930048968303, + 0.8276916627799221, + 1.2162971880696236, + 1.6194013533720035, + 0.49773181018003887, + 0.9092925416550506, + 1.4080183886330504, + 1.9383923878885982, + 0.6347887641015204, + 1.2149786829606046, + 1.9741746500727624, + 2.7005513648010666 + ], + "population_cv": 0.49185908209808765 + }, + "bits_per_byte": 3.064454978203266, + "branch_output_rms_by_sublayer": [ + 0.011030059307813644, + 0.01687927544116974, + 0.015950003638863564, + 0.013896239921450615, + 0.01350517850369215, + 0.011474287137389183, + 0.01258851308375597, + 0.010206365026533604, + 0.01664685271680355, + 0.008309077471494675, + 0.01638411357998848, + 0.007648966740816832, + 0.024508042261004448, + 0.008688886649906635, + 0.014716755598783493, + 0.00815621018409729, + 0.023891257122159004, + 0.00887411367148161, + 0.01776096411049366, + 0.008233130909502506, + 0.017747292295098305, + 0.00860576517879963, + 0.01586565561592579, + 0.008538740687072277, + 0.024832021445035934, + 0.009623449295759201, + 0.014417353086173534, + 0.009620912373065948, + 0.024172810837626457, + 0.010414699092507362, + 0.022103751078248024, + 0.010040229186415672, + 0.02063768543303013, + 0.01108868420124054, + 0.021260930225253105, + 0.012037678621709347, + 0.036969829350709915, + 0.012793113477528095, + 0.02196773886680603, + 0.01352718099951744, + 0.01941065676510334, + 0.015477209351956844, + 0.013355783186852932, + 0.017857730388641357, + 0.02278650738298893, + 0.02068868651986122, + 0.012330404482781887, + 0.02414606511592865, + 0.013476302847266197, + 0.016907230019569397, + 0.012490835972130299, + 0.018633393570780754, + 0.013263724744319916, + 0.022360114380717278, + 0.01237584836781025, + 0.02444436587393284, + 0.00844587478786707, + 0.026856400072574615, + 0.005500467959791422, + 0.026974599808454514, + 0.013505050912499428, + 0.03141019865870476, + 0.007422533817589283, + 0.031615760177373886 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0006198342260855069, + 0.0005186441869333752, + 0.0004736742025685692, + 0.0004519688001987414, + 0.0004976711823472734, + 0.00036518197323601154, + 0.0003310852368492195, + 0.0003116265537892263, + 0.0003288254475269336, + 0.00024450072302705585, + 0.00022900661460885773, + 0.00023928539744786398, + 0.0002540442376578553, + 0.00021470472716799896, + 0.00023495354274105486, + 0.00021391205552652906, + 0.00020893836869018885, + 0.00018720334490287413, + 0.0002054146048288436, + 0.0002008793076251433, + 0.00021675246724073825, + 0.00016136638641366094, + 0.0001626509134177638, + 0.00013898237065003446, + 0.0001008333231216009, + 9.308022189146143e-05, + 9.599750575032638e-05, + 9.048581684268014e-05, + 9.250484970926273e-05, + 9.079499931726835e-05, + 0.0001017609402056469, + 9.72918718535975e-05 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00044621079525099044, + "first_to_last_ratio": 4.680024343155117, + "imbalance_abs_log_ratio": 1.5433033114448986, + "last_quartile_mean": 9.534369108648053e-05, + "mean": 0.0002429330125054114, + "normalized": [ + 2.5514614901163335, + 2.1349267503189684, + 1.949814056490235, + 1.86046678274602, + 2.0485942903136216, + 1.5032208651670058, + 1.3628663862300086, + 1.2827674204315263, + 1.3535642773935839, + 1.0064532625906886, + 0.942673917583583, + 0.9849850993081224, + 1.0457378148727168, + 0.8838021846175244, + 0.9671536211482217, + 0.880539261915933, + 0.8600657709521249, + 0.7705965647575548, + 0.8455606865051655, + 0.8268917655671382, + 0.8922314221740861, + 0.6642423141649654, + 0.6695298911428957, + 0.5721016226517942, + 0.41506636780933515, + 0.3831518035836651, + 0.39516039734693537, + 0.37247229559080425, + 0.3807833639209581, + 0.3737450022987134, + 0.41888477467993424, + 0.40048847560983625 + ], + "population_cv": 0.5744356336428913 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6574935913085938, + "mean_weights": [ + 0.3934003710746765, + 0.6065995693206787 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6722193956375122, + "mean_weights": [ + 0.4378950297832489, + 0.5621049404144287 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6893110275268555, + "mean_weights": [ + 0.48470139503479004, + 0.51529860496521 + ], + "sources": 2 + }, + { + "entropy_mean": 0.668626070022583, + "mean_weights": [ + 0.43373414874076843, + 0.566265881061554 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6893103122711182, + "mean_weights": [ + 0.514161229133606, + 0.48583877086639404 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6807854175567627, + "mean_weights": [ + 0.4728299677371979, + 0.5271700620651245 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6757797002792358, + "mean_weights": [ + 0.4359714984893799, + 0.5640285015106201 + ], + "sources": 2 + }, + { + "entropy_mean": 0.670562744140625, + "mean_weights": [ + 0.4464735686779022, + 0.5535264611244202 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0913864374160767, + "mean_weights": [ + 0.30253520607948303, + 0.3440452516078949, + 0.35341954231262207 + ], + "sources": 3 + }, + { + "entropy_mean": 1.085841417312622, + "mean_weights": [ + 0.3023490011692047, + 0.32120704650878906, + 0.3764439523220062 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0840845108032227, + "mean_weights": [ + 0.27864623069763184, + 0.3560372591018677, + 0.3653165102005005 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0684943199157715, + "mean_weights": [ + 0.29403436183929443, + 0.2952164113521576, + 0.41074925661087036 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0875171422958374, + "mean_weights": [ + 0.288305401802063, + 0.3345819115638733, + 0.3771126866340637 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0656267404556274, + "mean_weights": [ + 0.2920410931110382, + 0.2975136637687683, + 0.4104451835155487 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0796833038330078, + "mean_weights": [ + 0.2819942831993103, + 0.3191397190093994, + 0.39886602759361267 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0474779605865479, + "mean_weights": [ + 0.2802623212337494, + 0.285836398601532, + 0.43390128016471863 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3790608644485474, + "mean_weights": [ + 0.23497441411018372, + 0.23470337688922882, + 0.2685908079147339, + 0.26173141598701477 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3612701892852783, + "mean_weights": [ + 0.2413691282272339, + 0.22060033679008484, + 0.25590670108795166, + 0.282123863697052 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3824801445007324, + "mean_weights": [ + 0.24027322232723236, + 0.23573172092437744, + 0.25346875190734863, + 0.27052628993988037 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3594305515289307, + "mean_weights": [ + 0.24403870105743408, + 0.2244095504283905, + 0.23220348358154297, + 0.29934826493263245 + ], + "sources": 4 + }, + { + "entropy_mean": 1.378438949584961, + "mean_weights": [ + 0.24557682871818542, + 0.23439225554466248, + 0.24105340242385864, + 0.27897751331329346 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3553731441497803, + "mean_weights": [ + 0.23845693469047546, + 0.22420883178710938, + 0.2327290177345276, + 0.30460524559020996 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3795093297958374, + "mean_weights": [ + 0.2617933750152588, + 0.22338548302650452, + 0.23882907629013062, + 0.2759920656681061 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3442331552505493, + "mean_weights": [ + 0.2454659640789032, + 0.22194916009902954, + 0.21799415349960327, + 0.314590722322464 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6065402030944824, + "mean_weights": [ + 0.1980130970478058, + 0.19166281819343567, + 0.1899465173482895, + 0.20429447293281555, + 0.2160830944776535 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5727713108062744, + "mean_weights": [ + 0.19455748796463013, + 0.1787414848804474, + 0.1701691448688507, + 0.2044031023979187, + 0.2521287798881531 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6030181646347046, + "mean_weights": [ + 0.19228291511535645, + 0.19301250576972961, + 0.1880970001220703, + 0.19615261256694794, + 0.2304549515247345 + ], + "sources": 5 + }, + { + "entropy_mean": 1.572641372680664, + "mean_weights": [ + 0.19314584136009216, + 0.1779472678899765, + 0.17160724103450775, + 0.19405120611190796, + 0.26324841380119324 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6009612083435059, + "mean_weights": [ + 0.19720734655857086, + 0.19277594983577728, + 0.1850610077381134, + 0.1913646161556244, + 0.23359106481075287 + ], + "sources": 5 + }, + { + "entropy_mean": 1.577996015548706, + "mean_weights": [ + 0.19977527856826782, + 0.17792916297912598, + 0.1681613326072693, + 0.19466948509216309, + 0.25946474075317383 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5992729663848877, + "mean_weights": [ + 0.1903986930847168, + 0.19716231524944305, + 0.1842092126607895, + 0.1906658113002777, + 0.23756396770477295 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5842759609222412, + "mean_weights": [ + 0.19851401448249817, + 0.18079298734664917, + 0.17754781246185303, + 0.18667176365852356, + 0.2564734220504761 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7810559272766113, + "mean_weights": [ + 0.15489047765731812, + 0.1641443967819214, + 0.1541292667388916, + 0.15085816383361816, + 0.17898379266262054, + 0.196993887424469 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7728064060211182, + "mean_weights": [ + 0.15748806297779083, + 0.15678821504116058, + 0.14855226874351501, + 0.14670711755752563, + 0.18628007173538208, + 0.20418427884578705 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7790249586105347, + "mean_weights": [ + 0.14998745918273926, + 0.16988207399845123, + 0.15484771132469177, + 0.14880013465881348, + 0.17533092200756073, + 0.20115166902542114 + ], + "sources": 6 + }, + { + "entropy_mean": 1.780612587928772, + "mean_weights": [ + 0.16733241081237793, + 0.16092416644096375, + 0.14877793192863464, + 0.1481637805700302, + 0.17866581678390503, + 0.19613587856292725 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7760759592056274, + "mean_weights": [ + 0.14748865365982056, + 0.17426073551177979, + 0.1526077538728714, + 0.1451883316040039, + 0.17089514434337616, + 0.20955941081047058 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7780911922454834, + "mean_weights": [ + 0.1675359606742859, + 0.16259664297103882, + 0.14989779889583588, + 0.14471940696239471, + 0.17003053426742554, + 0.20521965622901917 + ], + "sources": 6 + }, + { + "entropy_mean": 1.779514193534851, + "mean_weights": [ + 0.14703649282455444, + 0.18676123023033142, + 0.1658312827348709, + 0.14728040993213654, + 0.1593203842639923, + 0.19377018511295319 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7812495231628418, + "mean_weights": [ + 0.1853226125240326, + 0.1571069359779358, + 0.14897850155830383, + 0.15009444952011108, + 0.16374701261520386, + 0.19475045800209045 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9295728206634521, + "mean_weights": [ + 0.11980486661195755, + 0.1521853804588318, + 0.13728220760822296, + 0.12389329075813293, + 0.13301227986812592, + 0.1443169116973877, + 0.18950504064559937 + ], + "sources": 7 + }, + { + "entropy_mean": 1.939403772354126, + "mean_weights": [ + 0.14433100819587708, + 0.1480644941329956, + 0.1359301507472992, + 0.1293383538722992, + 0.13005699217319489, + 0.14529293775558472, + 0.16698604822158813 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9210636615753174, + "mean_weights": [ + 0.11677594482898712, + 0.16504092514514923, + 0.1436513215303421, + 0.122106172144413, + 0.12081696093082428, + 0.13804490864276886, + 0.193563774228096 + ], + "sources": 7 + }, + { + "entropy_mean": 1.936077356338501, + "mean_weights": [ + 0.13597163558006287, + 0.1512579321861267, + 0.14257019758224487, + 0.13186083734035492, + 0.1261359304189682, + 0.13521331548690796, + 0.17699016630649567 + ], + "sources": 7 + }, + { + "entropy_mean": 1.926892876625061, + "mean_weights": [ + 0.11894634366035461, + 0.16807839274406433, + 0.14728626608848572, + 0.12483938038349152, + 0.12647944688796997, + 0.13776510953903198, + 0.17660506069660187 + ], + "sources": 7 + }, + { + "entropy_mean": 1.935590386390686, + "mean_weights": [ + 0.1389310359954834, + 0.15760190784931183, + 0.14319241046905518, + 0.12697544693946838, + 0.12453635036945343, + 0.1408470869064331, + 0.16791579127311707 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9012556076049805, + "mean_weights": [ + 0.10868149995803833, + 0.19076427817344666, + 0.1634586900472641, + 0.12470689415931702, + 0.11844731867313385, + 0.1288575828075409, + 0.16508373618125916 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.056945562362671, + "mean_weights": [ + 0.12425215542316437, + 0.1432928442955017, + 0.13492462038993835, + 0.11158157885074615, + 0.10743287205696106, + 0.12180259078741074, + 0.13476303219795227, + 0.12195030599832535 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1307528018951416, + "mean_weights": [ + 0.08413388580083847, + 0.17738990485668182, + 0.1511024832725525, + 0.11142782866954803, + 0.0973522886633873, + 0.100654736161232, + 0.11285731196403503, + 0.07174938917160034, + 0.09333217144012451 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1892080307006836, + "mean_weights": [ + 0.10430333018302917, + 0.12487784773111343, + 0.11785529553890228, + 0.10954566299915314, + 0.11100621521472931, + 0.11449144780635834, + 0.11575768142938614, + 0.09940418601036072, + 0.10275833308696747 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1215670108795166, + "mean_weights": [ + 0.08588143438100815, + 0.1802678406238556, + 0.1551700383424759, + 0.1130732074379921, + 0.10466247797012329, + 0.10527389496564865, + 0.11408616602420807, + 0.07156369090080261, + 0.07002124190330505 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1661715507507324, + "mean_weights": [ + 0.10998079180717468, + 0.13038715720176697, + 0.1233307272195816, + 0.10191114246845245, + 0.10612759739160538, + 0.11833611130714417, + 0.1095183789730072, + 0.09677169471979141, + 0.10363639146089554 + ], + "sources": 9 + }, + { + "entropy_mean": 2.113847494125366, + "mean_weights": [ + 0.08664848655462265, + 0.1866048276424408, + 0.15658384561538696, + 0.11330428719520569, + 0.10171857476234436, + 0.10515032708644867, + 0.11401651799678802, + 0.07199416309595108, + 0.06397897005081177 + ], + "sources": 9 + }, + { + "entropy_mean": 2.182473659515381, + "mean_weights": [ + 0.10189944505691528, + 0.1302962601184845, + 0.1255553662776947, + 0.11592286825180054, + 0.11095131933689117, + 0.1152336448431015, + 0.11264198273420334, + 0.09484575688838959, + 0.09265334904193878 + ], + "sources": 9 + }, + { + "entropy_mean": 2.106433868408203, + "mean_weights": [ + 0.08660078048706055, + 0.18835332989692688, + 0.157449409365654, + 0.11395236849784851, + 0.10455858707427979, + 0.10647574812173843, + 0.1130424439907074, + 0.0701935887336731, + 0.05937372148036957 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.043994102627038956, + 0.01832600310444832, + 0.02416435442864895, + 0.027104532346129417, + 0.03068346157670021, + 0.03203236684203148, + 0.034910786896944046, + 0.03651289641857147, + 0.038895584642887115, + 0.025965023785829544, + 0.026926059275865555, + 0.02835584245622158, + 0.028757663443684578, + 0.02952437661588192, + 0.0310931745916605, + 0.03088352456688881, + 0.03258504718542099, + 0.023613223806023598, + 0.025326738134026527, + 0.02411509118974209, + 0.025759827345609665, + 0.02486386150121689, + 0.026853490620851517, + 0.025490598753094673, + 0.02807508409023285, + 0.02089438959956169, + 0.022392546758055687, + 0.021244581788778305, + 0.02307868003845215, + 0.022009342908859253, + 0.023753024637699127, + 0.022865574806928635, + 0.02441880665719509, + 0.019287733361124992, + 0.020282816141843796, + 0.0205394234508276, + 0.0212617926299572, + 0.021357320249080658, + 0.022198764607310295, + 0.022119909524917603, + 0.022867528721690178, + 0.01841953583061695, + 0.019446156919002533, + 0.01910795085132122, + 0.020555177703499794, + 0.020677685737609863, + 0.022395195439457893, + 0.02209646999835968, + 0.024376342073082924, + 0.02156461961567402, + 0.022625979036092758, + 0.022691691294312477, + 0.02398044615983963, + 0.024076392874121666, + 0.025595931336283684, + 0.02575858309864998, + 0.02841673232614994, + 0.022702276706695557, + 0.026171032339334488, + 0.024497199803590775, + 0.028925856575369835, + 0.026029502972960472, + 0.03001977689564228, + 0.027346057817339897 + ], + "loss_nats": 2.1241183280944824, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1697804927825928, + "mean_weights": [ + 0.12218696624040604, + 0.09054506570100784, + 0.09749520570039749, + 0.0979107916355133, + 0.10050912201404572, + 0.1113804280757904, + 0.12618403136730194, + 0.124283567070961, + 0.12950481474399567 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.011030059307813644, + 0.021257206797599792, + 0.030371982604265213, + 0.039118532091379166, + 0.04173211753368378, + 0.049036867916584015, + 0.05258767679333687, + 0.058654993772506714, + 0.01664685271680355, + 0.019036665558815002, + 0.02703237533569336, + 0.029960447922348976, + 0.03185733035206795, + 0.0357137955725193, + 0.037442173808813095, + 0.04143182933330536, + 0.023891257122159004, + 0.026369430124759674, + 0.028511347249150276, + 0.03188152238726616, + 0.03345155343413353, + 0.03730445355176926, + 0.0386769101023674, + 0.04281774163246155, + 0.024832021445035934, + 0.027975991368293762, + 0.03410641849040985, + 0.03802811726927757, + 0.03699766471982002, + 0.04204143211245537, + 0.04634200409054756, + 0.05121698975563049, + 0.02063768543303013, + 0.02476053684949875, + 0.03547017648816109, + 0.04124032333493233, + 0.04148983210325241, + 0.04760165512561798, + 0.05035363882780075, + 0.057892896234989166, + 0.01941065676510334, + 0.025168301537632942, + 0.027913866564631462, + 0.03825869783759117, + 0.04175899550318718, + 0.05622135475277901, + 0.05649513006210327, + 0.07485418766736984, + 0.013476302847266197, + 0.02300684154033661, + 0.027908887714147568, + 0.042030565440654755, + 0.04727315157651901, + 0.06508335471153259, + 0.0695502758026123, + 0.08959902822971344, + 0.00844587478786707, + 0.029342075809836388, + 0.031286753714084625, + 0.05616040900349617, + 0.062057800590991974, + 0.09125300496816635, + 0.09534843266010284, + 0.12482858449220657 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.75534174661152e-05, + 7.18602241249755e-05, + 6.714894698234275e-05, + 6.517143629025668e-05, + 7.33894485165365e-05, + 6.863255111966282e-05, + 6.158801261335611e-05, + 5.806105400552042e-05, + 5.946493547526188e-05, + 5.687734665116295e-05, + 5.313277870300226e-05, + 4.949783397023566e-05, + 4.948719288222492e-05, + 4.76040986541193e-05, + 4.6945089707151055e-05, + 4.802063631359488e-05, + 6.637556361965835e-05, + 6.871068762848154e-05, + 6.149089313112199e-05, + 5.571809015236795e-05, + 0.00010889588884310797, + 8.721341873751953e-05, + 6.807410682085901e-05, + 5.299493932398036e-05, + 2.8719190595438704e-05, + 2.7304024115437642e-05, + 2.607300666568335e-05, + 2.5389002985320985e-05, + 3.6188317608321086e-05, + 3.312563057988882e-05, + 3.10283103317488e-05, + 3.066000499529764e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 6.667563638984575e-05, + "first_to_last_ratio": 2.2366166706136106, + "imbalance_abs_log_ratio": 0.804964309135819, + "last_quartile_mean": 2.981093598464213e-05, + "mean": 5.476237748780477e-05, + "normalized": [ + 1.233573496350828, + 1.3122188520938178, + 1.2261875773617825, + 1.1900768242717354, + 1.3401435781870477, + 1.2532792451340677, + 1.1246409567786089, + 1.0602361816458798, + 1.0858720567510851, + 1.0386208426365193, + 0.9702423660264675, + 0.9038656873737542, + 0.9036713735309501, + 0.8692847322912601, + 0.8572507597502579, + 0.8768910064996496, + 1.2120650465630307, + 1.2547060734129551, + 1.1228674858905756, + 1.017451993657069, + 1.9885164567107845, + 1.5925791161448644, + 1.2430816546637093, + 0.9677253208333038, + 0.5244328663749904, + 0.4985909189482884, + 0.4761116639154252, + 0.46362126974810314, + 0.6608244431385396, + 0.6048975975753734, + 0.5665990366955599, + 0.5598735190437162 + ], + "population_cv": 0.341199796595388 + }, + "activation_output_rms_by_block": [ + 0.08486531674861908, + 0.17329199612140656, + 0.20363987982273102, + 0.21725694835186005, + 0.09199090301990509, + 0.10107264667749405, + 0.10942362248897552, + 0.11504421383142471, + 0.029957696795463562, + 0.047157589346170425, + 0.07332460582256317, + 0.10941736400127411, + 0.05922413989901543, + 0.09802009910345078, + 0.13082744181156158, + 0.1853637546300888, + 0.039145275950431824, + 0.06245216727256775, + 0.09436928480863571, + 0.13145026564598083, + 0.061799149960279465, + 0.08572212606668472, + 0.11407312005758286, + 0.15109683573246002, + 0.08768751472234726, + 0.12105754762887955, + 0.16361796855926514, + 0.2166183739900589, + 0.06281252950429916, + 0.11395928263664246, + 0.18981148302555084, + 0.27663207054138184 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.137073190882802, + "first_to_last_ratio": 0.8899435165058517, + "imbalance_abs_log_ratio": 0.11659728287008113, + "last_quartile_mean": 0.15402459632605314, + "mean": 0.11881822545547038, + "normalized": [ + 0.7142449436801607, + 1.4584630889505361, + 1.7138774715925154, + 1.8284816787916232, + 0.7742154258513195, + 0.8506493535823184, + 0.920932980353122, + 0.9682370982264833, + 0.25213048486985556, + 0.3968885174425006, + 0.6171158131800504, + 0.9208803075609016, + 0.4984432284861123, + 0.8249584499996245, + 1.1010721739872467, + 1.5600616312819597, + 0.32945514714072494, + 0.5256110081863915, + 0.7942324037148878, + 1.1063139946930496, + 0.5201150726109774, + 0.7214560370522525, + 0.9600641620450235, + 1.2716637969743685, + 0.7379971749806177, + 1.0188466219287915, + 1.3770443711986289, + 1.8231072982253986, + 0.5286438950213027, + 0.959106081577957, + 1.5974946797760978, + 2.3281956070372005 + ], + "population_cv": 0.48719861447722207 + }, + "bits_per_byte": 2.070141237840908, + "branch_output_rms_by_sublayer": [ + 0.04614673927426338, + 0.06757520884275436, + 0.10059163719415665, + 0.02935902588069439, + 0.0441400371491909, + 0.027569489553570747, + 0.026136217638850212, + 0.01578480750322342, + 0.09152255952358246, + 0.004334695637226105, + 0.03589765354990959, + 0.008940798230469227, + 0.044497180730104446, + 0.014118778519332409, + 0.02492278255522251, + 0.01518768910318613, + 0.024665212258696556, + 0.016177508980035782, + 0.021984105929732323, + 0.01858041249215603, + 0.028709426522254944, + 0.024067549034953117, + 0.03398026525974274, + 0.03344719484448433, + 0.04885389283299446, + 0.026064932346343994, + 0.033774133771657944, + 0.0285946037620306, + 0.030830858275294304, + 0.029918670654296875, + 0.06011629477143288, + 0.029190504923462868, + 0.029006898403167725, + 0.023256491869688034, + 0.03192666172981262, + 0.02065420150756836, + 0.05095710977911949, + 0.025297319516539574, + 0.05602070689201355, + 0.037405598908662796, + 0.04447361081838608, + 0.03998441994190216, + 0.01981385610997677, + 0.0435076542198658, + 0.026326991617679596, + 0.0518617182970047, + 0.032579511404037476, + 0.062188562005758286, + 0.06666523963212967, + 0.04834020137786865, + 0.02888461761176586, + 0.049218423664569855, + 0.03144153580069542, + 0.062090445309877396, + 0.02167421393096447, + 0.08679787814617157, + 0.01256827637553215, + 0.060933083295822144, + 0.00911632552742958, + 0.060767170041799545, + 0.021987028419971466, + 0.08606750518083572, + 0.009766319766640663, + 0.09914495795965195 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00023025988056526322, + 0.00019040619928786272, + 0.0001441906854341128, + 0.00011760812053491448, + 0.00016151012401646462, + 0.00010048529566091257, + 0.00011118485890080037, + 0.00011096545432385721, + 0.00010578555644826673, + 0.00010468298761847836, + 0.00012596723868537883, + 0.0001495844727707013, + 0.00012123300635753196, + 0.00012154383062919504, + 0.00012091446888282359, + 0.00014273902433057623, + 0.0001671190395531384, + 0.00018133004823098025, + 0.00026286395017729953, + 0.0002888645605444862, + 0.0006210099776583573, + 0.0004302889605955639, + 0.00037610688417376715, + 0.0003194297635529056, + 0.00014493952023458926, + 0.00011837337657009581, + 0.00012183797128547915, + 0.0001282958983192414, + 0.00017611097131714273, + 0.00016136686235020573, + 0.00016566687488177103, + 0.0001560060496556608 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0001458263273405235, + "first_to_last_ratio": 0.9948943215687175, + "imbalance_abs_log_ratio": 0.005118756942846249, + "last_quartile_mean": 0.00014657469057677324, + "mean": 0.00018683349729836952, + "normalized": [ + 1.23243360475955, + 1.0191223845892456, + 0.77176035089598, + 0.6294809134097434, + 0.8644602084311254, + 0.5378334030778189, + 0.5951013095004065, + 0.593926977380547, + 0.566202303002064, + 0.5603009585122823, + 0.6742219168772051, + 0.8006298385124042, + 0.6488826046216175, + 0.650546247791379, + 0.6471776776180852, + 0.7639905391409804, + 0.8944811394621195, + 0.9705435634028711, + 1.4069422987758498, + 1.5461069065317288, + 3.323868506655508, + 2.303061104232292, + 2.013059165579562, + 1.7097028539950863, + 0.7757683837771533, + 0.6335768386386077, + 0.6521205942578179, + 0.686685739840085, + 0.9426091887360909, + 0.863693420524618, + 0.8867086324311758, + 0.8350004250389967 + ], + "population_cv": 0.5949621409337901 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.4272422194480896, + "mean_weights": [ + 0.8360251188278198, + 0.16397488117218018 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6394244432449341, + "mean_weights": [ + 0.5483858585357666, + 0.4516141414642334 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5099630355834961, + "mean_weights": [ + 0.7857437133789062, + 0.21425624191761017 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6889077425003052, + "mean_weights": [ + 0.4726368486881256, + 0.5273631811141968 + ], + "sources": 2 + }, + { + "entropy_mean": 0.48204338550567627, + "mean_weights": [ + 0.8057355284690857, + 0.19426444172859192 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6868308782577515, + "mean_weights": [ + 0.5278943777084351, + 0.4721056818962097 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6234793066978455, + "mean_weights": [ + 0.6555083990097046, + 0.3444916009902954 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6733173131942749, + "mean_weights": [ + 0.49166494607925415, + 0.5083350539207458 + ], + "sources": 2 + }, + { + "entropy_mean": 0.8952018618583679, + "mean_weights": [ + 0.22153860330581665, + 0.16111916303634644, + 0.6173421740531921 + ], + "sources": 3 + }, + { + "entropy_mean": 1.094365119934082, + "mean_weights": [ + 0.33062052726745605, + 0.309090793132782, + 0.36028867959976196 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0019135475158691, + "mean_weights": [ + 0.24093851447105408, + 0.24913212656974792, + 0.509929358959198 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0844204425811768, + "mean_weights": [ + 0.3346637487411499, + 0.2689046263694763, + 0.3964316248893738 + ], + "sources": 3 + }, + { + "entropy_mean": 1.028485655784607, + "mean_weights": [ + 0.4040144085884094, + 0.1989833563566208, + 0.3970022201538086 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0878751277923584, + "mean_weights": [ + 0.34510767459869385, + 0.2763848602771759, + 0.37850743532180786 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0230355262756348, + "mean_weights": [ + 0.3256777226924896, + 0.23217403888702393, + 0.44214826822280884 + ], + "sources": 3 + }, + { + "entropy_mean": 1.092297911643982, + "mean_weights": [ + 0.36376672983169556, + 0.29448920488357544, + 0.341744065284729 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3567805290222168, + "mean_weights": [ + 0.289939284324646, + 0.16906873881816864, + 0.2849292755126953, + 0.25606268644332886 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3791608810424805, + "mean_weights": [ + 0.28508156538009644, + 0.2424212545156479, + 0.21972578763961792, + 0.25277137756347656 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3597800731658936, + "mean_weights": [ + 0.29065239429473877, + 0.19744595885276794, + 0.2412797212600708, + 0.2706219255924225 + ], + "sources": 4 + }, + { + "entropy_mean": 1.374577283859253, + "mean_weights": [ + 0.28176671266555786, + 0.21918919682502747, + 0.21482568979263306, + 0.2842184007167816 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3306174278259277, + "mean_weights": [ + 0.32665544748306274, + 0.16931182146072388, + 0.2009648084640503, + 0.30306798219680786 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3698134422302246, + "mean_weights": [ + 0.28411588072776794, + 0.20833143591880798, + 0.2092670202255249, + 0.29828566312789917 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3175115585327148, + "mean_weights": [ + 0.36134999990463257, + 0.17643077671527863, + 0.1699690818786621, + 0.2922501266002655 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3770861625671387, + "mean_weights": [ + 0.2876768410205841, + 0.24400946497917175, + 0.222049742937088, + 0.24626395106315613 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5963144302368164, + "mean_weights": [ + 0.22701337933540344, + 0.17271918058395386, + 0.17705240845680237, + 0.20655202865600586, + 0.21666303277015686 + ], + "sources": 5 + }, + { + "entropy_mean": 1.598343014717102, + "mean_weights": [ + 0.21596191823482513, + 0.19929684698581696, + 0.1717657893896103, + 0.1746840476989746, + 0.2382914125919342 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5866549015045166, + "mean_weights": [ + 0.20038136839866638, + 0.16451019048690796, + 0.17001068592071533, + 0.22104427218437195, + 0.24405348300933838 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5800271034240723, + "mean_weights": [ + 0.19937419891357422, + 0.19696298241615295, + 0.15431061387062073, + 0.16041024029254913, + 0.28894197940826416 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5902726650238037, + "mean_weights": [ + 0.2289389818906784, + 0.17283514142036438, + 0.16605672240257263, + 0.18588173389434814, + 0.24628742039203644 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5538716316223145, + "mean_weights": [ + 0.20080974698066711, + 0.21879449486732483, + 0.13812820613384247, + 0.13342610001564026, + 0.30884143710136414 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5823884010314941, + "mean_weights": [ + 0.18060284852981567, + 0.18559986352920532, + 0.17961221933364868, + 0.22351807355880737, + 0.23066699504852295 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5703144073486328, + "mean_weights": [ + 0.19568157196044922, + 0.1944681704044342, + 0.13765949010849, + 0.17981542646884918, + 0.2923753559589386 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7846708297729492, + "mean_weights": [ + 0.16887447237968445, + 0.15695782005786896, + 0.1638387143611908, + 0.1827341616153717, + 0.16792084276676178, + 0.1596740186214447 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7534986734390259, + "mean_weights": [ + 0.1576918363571167, + 0.1615266054868698, + 0.1132349818944931, + 0.12562187016010284, + 0.19022522866725922, + 0.2516994774341583 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7285370826721191, + "mean_weights": [ + 0.1391056627035141, + 0.13142594695091248, + 0.12644673883914948, + 0.14366388320922852, + 0.16390609741210938, + 0.29545167088508606 + ], + "sources": 6 + }, + { + "entropy_mean": 1.75887131690979, + "mean_weights": [ + 0.16609874367713928, + 0.16340577602386475, + 0.11888011544942856, + 0.12218527495861053, + 0.18685105443000793, + 0.24257901310920715 + ], + "sources": 6 + }, + { + "entropy_mean": 1.675755500793457, + "mean_weights": [ + 0.13635966181755066, + 0.12989377975463867, + 0.10955628752708435, + 0.1512342095375061, + 0.12196142971515656, + 0.35099461674690247 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7734620571136475, + "mean_weights": [ + 0.17241644859313965, + 0.17682810127735138, + 0.1343054175376892, + 0.12303055822849274, + 0.1834457814693451, + 0.20997366309165955 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6871838569641113, + "mean_weights": [ + 0.1398472785949707, + 0.13660696148872375, + 0.1324334740638733, + 0.1545041799545288, + 0.10401631891727448, + 0.3325917720794678 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7801499366760254, + "mean_weights": [ + 0.17806105315685272, + 0.1869724988937378, + 0.13933779299259186, + 0.13054659962654114, + 0.1849755495786667, + 0.1801065057516098 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7390422821044922, + "mean_weights": [ + 0.0914311408996582, + 0.09009712934494019, + 0.09411133825778961, + 0.10237805545330048, + 0.08801177144050598, + 0.14315900206565857, + 0.3908115327358246 + ], + "sources": 7 + }, + { + "entropy_mean": 1.889643669128418, + "mean_weights": [ + 0.1356136053800583, + 0.14138123393058777, + 0.10668636858463287, + 0.10100890696048737, + 0.1309133917093277, + 0.12421204149723053, + 0.26018446683883667 + ], + "sources": 7 + }, + { + "entropy_mean": 1.6817257404327393, + "mean_weights": [ + 0.08877424895763397, + 0.08818034827709198, + 0.08711051940917969, + 0.09243019670248032, + 0.08189001679420471, + 0.13213670253753662, + 0.4294779598712921 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9038465023040771, + "mean_weights": [ + 0.13999512791633606, + 0.14833927154541016, + 0.11340493708848953, + 0.10031044483184814, + 0.14205804467201233, + 0.11917966604232788, + 0.2367125153541565 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7267472743988037, + "mean_weights": [ + 0.10086647421121597, + 0.09752936661243439, + 0.09476756304502487, + 0.09442415833473206, + 0.08333539962768555, + 0.1264602541923523, + 0.40261679887771606 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9142005443572998, + "mean_weights": [ + 0.14507804811000824, + 0.1586056798696518, + 0.1192338764667511, + 0.1008925586938858, + 0.14699265360832214, + 0.1155494898557663, + 0.21364769339561462 + ], + "sources": 7 + }, + { + "entropy_mean": 1.766711950302124, + "mean_weights": [ + 0.10525530576705933, + 0.10558505356311798, + 0.10354176163673401, + 0.1032070443034172, + 0.09395042806863785, + 0.11654135584831238, + 0.37191903591156006 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0250916481018066, + "mean_weights": [ + 0.10501185059547424, + 0.11825795471668243, + 0.12522642314434052, + 0.11810420453548431, + 0.10223683714866638, + 0.10317667573690414, + 0.1963549256324768, + 0.13163113594055176 + ], + "sources": 8 + }, + { + "entropy_mean": 2.087522506713867, + "mean_weights": [ + 0.0886697769165039, + 0.1104993000626564, + 0.09380728006362915, + 0.08890138566493988, + 0.08554239571094513, + 0.07964761555194855, + 0.2380208820104599, + 0.11424270272254944, + 0.10066868364810944 + ], + "sources": 9 + }, + { + "entropy_mean": 2.112476348876953, + "mean_weights": [ + 0.08111660182476044, + 0.10016020387411118, + 0.10466738045215607, + 0.11120539903640747, + 0.08322704583406448, + 0.0931069403886795, + 0.1996506303548813, + 0.10095033794641495, + 0.12591548264026642 + ], + "sources": 9 + }, + { + "entropy_mean": 2.060291290283203, + "mean_weights": [ + 0.07948802411556244, + 0.09962397068738937, + 0.09782242774963379, + 0.08484400808811188, + 0.0861716940999031, + 0.08718602359294891, + 0.23774218559265137, + 0.09835822135210037, + 0.12876343727111816 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1332716941833496, + "mean_weights": [ + 0.10683543980121613, + 0.13522090017795563, + 0.12021824717521667, + 0.08978401124477386, + 0.10303761065006256, + 0.08665332198143005, + 0.12903158366680145, + 0.11247263848781586, + 0.11674625426530838 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0480668544769287, + "mean_weights": [ + 0.08760041743516922, + 0.11633757501840591, + 0.08851546794176102, + 0.08755243569612503, + 0.08085177093744278, + 0.07695268094539642, + 0.22807800769805908, + 0.10828100144863129, + 0.12583065032958984 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1321568489074707, + "mean_weights": [ + 0.08503308147192001, + 0.09791319072246552, + 0.10442496836185455, + 0.1149793416261673, + 0.08077225834131241, + 0.10436949133872986, + 0.2063458263874054, + 0.11004947125911713, + 0.09611237794160843 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0983121395111084, + "mean_weights": [ + 0.08679497987031937, + 0.11847125738859177, + 0.10118316113948822, + 0.10502803325653076, + 0.09095121920108795, + 0.10656492412090302, + 0.20489060878753662, + 0.10370983183383942, + 0.08240596950054169 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05617385730147362, + 0.0483684316277504, + 0.0563882552087307, + 0.06223662570118904, + 0.10121176391839981, + 0.06557638198137283, + 0.10718171298503876, + 0.09055784344673157, + 0.11906259506940842, + 0.08129429072141647, + 0.0919070616364479, + 0.09106305986642838, + 0.08915899693965912, + 0.07935573160648346, + 0.09175078570842743, + 0.08815857023000717, + 0.09255139529705048, + 0.06177081912755966, + 0.07136296480894089, + 0.06460285931825638, + 0.06701613962650299, + 0.05849825590848923, + 0.06552907079458237, + 0.05886770412325859, + 0.07309140264987946, + 0.052307069301605225, + 0.05753219872713089, + 0.05282364413142204, + 0.05908621847629547, + 0.05573583394289017, + 0.0674879178404808, + 0.06553760170936584, + 0.07526855170726776, + 0.05565962195396423, + 0.05646168440580368, + 0.05163586512207985, + 0.057007260620594025, + 0.05146156996488571, + 0.059214167296886444, + 0.055561717599630356, + 0.06149348244071007, + 0.03492685779929161, + 0.045021090656518936, + 0.040231529623270035, + 0.04790741577744484, + 0.04658481478691101, + 0.05053836852312088, + 0.053088560700416565, + 0.05073489621281624, + 0.042606521397829056, + 0.04355289041996002, + 0.04346136003732681, + 0.04509321600198746, + 0.04517792537808418, + 0.04787394776940346, + 0.047962795943021774, + 0.05871318280696869, + 0.053657278418540955, + 0.05217254161834717, + 0.05520017817616463, + 0.05983608216047287, + 0.06309890747070312, + 0.05996966361999512, + 0.0617501474916935 + ], + "loss_nats": 1.4349125623703003, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.9598431587219238, + "mean_weights": [ + 0.11197228729724884, + 0.07924222946166992, + 0.07301962375640869, + 0.06148884445428848, + 0.0683768168091774, + 0.05808756500482559, + 0.1279553771018982, + 0.14163494110107422, + 0.27822232246398926 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.04614673927426338, + 0.08486531674861908, + 0.15898971259593964, + 0.17329199612140656, + 0.19024431705474854, + 0.20363987982273102, + 0.21109889447689056, + 0.21725694835186005, + 0.09152255952358246, + 0.09199090301990509, + 0.09975728392601013, + 0.10107264667749405, + 0.107844278216362, + 0.10942362248897552, + 0.11277434229850769, + 0.11504421383142471, + 0.024665212258696556, + 0.029957696795463562, + 0.03811698406934738, + 0.047157589346170425, + 0.060785479843616486, + 0.07332460582256317, + 0.08899348229169846, + 0.10941736400127411, + 0.04885389283299446, + 0.05922413989901543, + 0.08325673639774323, + 0.09802009910345078, + 0.1148156225681305, + 0.13082744181156158, + 0.16857220232486725, + 0.1853637546300888, + 0.029006898403167725, + 0.039145275950431824, + 0.05470960959792137, + 0.06245216727256775, + 0.08605960011482239, + 0.09436928480863571, + 0.11819671094417572, + 0.13145026564598083, + 0.04447361081838608, + 0.061799149960279465, + 0.06616436690092087, + 0.08572212606668472, + 0.0913865715265274, + 0.11407312005758286, + 0.12277228385210037, + 0.15109683573246002, + 0.06666523963212967, + 0.08768751472234726, + 0.09621186554431915, + 0.12105754762887955, + 0.12993480265140533, + 0.16361796855926514, + 0.16660332679748535, + 0.2166183739900589, + 0.01256827637553215, + 0.06281252950429916, + 0.06412965059280396, + 0.11395928263664246, + 0.11805794388055801, + 0.18981148302555084, + 0.19161006808280945, + 0.27663207054138184 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.422633305191994e-05, + 6.669556751148775e-05, + 6.307693547569215e-05, + 6.212162406882271e-05, + 6.756730726920068e-05, + 6.338968523778021e-05, + 5.897105802432634e-05, + 5.658321242663078e-05, + 6.233588646864519e-05, + 5.9621917898766696e-05, + 5.5369106121361256e-05, + 5.0777889555320144e-05, + 4.796316716237925e-05, + 4.593767152982764e-05, + 4.539819565252401e-05, + 4.6929817472118884e-05, + 7.046038808766752e-05, + 7.202024426078424e-05, + 6.405381282093003e-05, + 5.6841243349481374e-05, + 0.00011249445378780365, + 8.878010703483596e-05, + 6.928577204234898e-05, + 5.257409793557599e-05, + 2.935541488113813e-05, + 2.783649142656941e-05, + 2.6367106329416856e-05, + 2.5619510779506527e-05, + 4.313089812058024e-05, + 3.625801036832854e-05, + 3.1853622203925624e-05, + 3.129153628833592e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 6.282896538323257e-05, + "first_to_last_ratio": 1.996847763044002, + "imbalance_abs_log_ratio": 0.6915698187005687, + "last_quartile_mean": 3.1464073799725156e-05, + "mean": 5.484962764512602e-05, + "normalized": [ + 1.170952945523362, + 1.21597119934896, + 1.149997514728748, + 1.1325805978254966, + 1.2318644659970588, + 1.155699463411272, + 1.0751405352442087, + 1.031606135828687, + 1.1364869579781804, + 1.0870067940026342, + 1.009470957206788, + 0.9257654378959546, + 0.8744483640380974, + 0.8375202075580489, + 0.8276846644474555, + 0.8556086793469618, + 1.2846101443667446, + 1.3130489185223122, + 1.1678076145813527, + 1.036310469001558, + 2.0509611207506526, + 1.6186091108810843, + 1.2631949389086832, + 0.9585133061563776, + 0.5351980704603138, + 0.5075055678895385, + 0.4807162320227685, + 0.46708632090017654, + 0.7863480569026777, + 0.6610438744072396, + 0.5807445478028903, + 0.5704967860637156 + ], + "population_cv": 0.3365634187357017 + }, + "activation_output_rms_by_block": [ + 0.12060720473527908, + 0.2573152482509613, + 0.28323665261268616, + 0.29319411516189575, + 0.09563838690519333, + 0.10590620338916779, + 0.11774174869060516, + 0.12761500477790833, + 0.030837642028927803, + 0.05576188862323761, + 0.09026926010847092, + 0.15066321194171906, + 0.06286879628896713, + 0.10371332615613937, + 0.14144682884216309, + 0.2107442319393158, + 0.04767552763223648, + 0.07339637726545334, + 0.11630241572856903, + 0.16868281364440918, + 0.07512061297893524, + 0.10792030394077301, + 0.14635293185710907, + 0.19755741953849792, + 0.12152406573295593, + 0.16458973288536072, + 0.2178809642791748, + 0.27798205614089966, + 0.07472676783800125, + 0.1334167718887329, + 0.21953867375850677, + 0.32601314783096313 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.1751568205654621, + "first_to_last_ratio": 0.9124698503036889, + "imbalance_abs_log_ratio": 0.0916002348344292, + "last_quartile_mean": 0.1919590225443244, + "mean": 0.147382510418538, + "normalized": [ + 0.8183277947483578, + 1.745900836674823, + 1.9217792654525188, + 1.9893413019582908, + 0.6489127280679281, + 0.7185805363771761, + 0.7988854875401513, + 0.8658761776787913, + 0.2092354238054083, + 0.37834807172767354, + 0.6124828505914546, + 1.022259775015804, + 0.42656890638130557, + 0.7037017205203892, + 0.9597260111832895, + 1.429913436410078, + 0.32348158201978744, + 0.4979992338101837, + 0.7891195189869715, + 1.144523954473176, + 0.5096982862186777, + 0.7322463407245546, + 0.9930142419307072, + 1.3404400493482762, + 0.8245487567544544, + 1.1167521330581052, + 1.4783366334338772, + 1.8861264837427727, + 0.5070260211051609, + 0.9052415480633008, + 1.489584300980212, + 2.212020591216342 + ], + "population_cv": 0.5204095420748207 + }, + "bits_per_byte": 1.9272205592861982, + "branch_output_rms_by_sublayer": [ + 0.07275605201721191, + 0.0915931686758995, + 0.1635962277650833, + 0.026601020246744156, + 0.042792417109012604, + 0.03397384658455849, + 0.026360780000686646, + 0.017556756734848022, + 0.09455645084381104, + 0.004627425689250231, + 0.033148929476737976, + 0.008047145791351795, + 0.03476296365261078, + 0.01755029708147049, + 0.025515727698802948, + 0.018163152039051056, + 0.0242336243391037, + 0.019344858825206757, + 0.025055749341845512, + 0.023276831954717636, + 0.0358181893825531, + 0.030225226655602455, + 0.05326661095023155, + 0.04969140514731407, + 0.0473942868411541, + 0.033756598830223083, + 0.036102235317230225, + 0.038699399679899216, + 0.03342170640826225, + 0.03783123567700386, + 0.07379191368818283, + 0.03680294379591942, + 0.03429211303591728, + 0.029226219281554222, + 0.03589168190956116, + 0.025161370635032654, + 0.0634535625576973, + 0.037599414587020874, + 0.07025575637817383, + 0.06040560454130173, + 0.05036674439907074, + 0.052244365215301514, + 0.027424054220318794, + 0.05997655913233757, + 0.03688060864806175, + 0.07332249730825424, + 0.05082467943429947, + 0.09044640511274338, + 0.09325341880321503, + 0.06612210720777512, + 0.043520405888557434, + 0.06618789583444595, + 0.05466217175126076, + 0.08252561837434769, + 0.03107001818716526, + 0.11511830240488052, + 0.016754506155848503, + 0.0724438950419426, + 0.013077927753329277, + 0.07599451392889023, + 0.02301586978137493, + 0.10536263883113861, + 0.01386463176459074, + 0.12691864371299744 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00027128029391770787, + 0.0001975403308125262, + 0.00014911964651359967, + 0.00010557294228602117, + 0.0001533180018921871, + 8.773323465189516e-05, + 0.00010993899457799429, + 0.00011364083269316236, + 0.00010833261344433474, + 0.00011982783723130222, + 0.0001548857521142111, + 0.00020436345698732848, + 0.0001336206406582627, + 0.00013905794975194562, + 0.00013932433174453112, + 0.00017096312100322667, + 0.00019454152594803353, + 0.0002256334377567118, + 0.0003213947177700571, + 0.0003543524336354052, + 0.0007225144918112082, + 0.000550484400823151, + 0.0005011515689395991, + 0.000442305374149654, + 0.00019265497060143202, + 0.00015424509114698312, + 0.00015875292870757548, + 0.00015838989036593332, + 0.00024545000319233147, + 0.0002126150613087748, + 0.00021297392836658305, + 0.00019258363327050634 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00014851803466813673, + "first_to_last_ratio": 0.7777515902086221, + "imbalance_abs_log_ratio": 0.25134809857951584, + "last_quartile_mean": 0.00019095818837001495, + "mean": 0.000224955107439818, + "normalized": [ + 1.2059308055065308, + 0.8781322329628545, + 0.6628862452189199, + 0.46930671407078406, + 0.6815493261614614, + 0.3900033017715161, + 0.48871526336607474, + 0.5051711605328391, + 0.48157436689147787, + 0.5326744460041195, + 0.6885184954320165, + 0.9084632899121956, + 0.5939880280069217, + 0.6181586687874941, + 0.6193428250203407, + 0.7599877279913027, + 0.8648015515721466, + 1.0030154030491407, + 1.428706026850444, + 1.575213995664467, + 3.2118163487552813, + 2.4470855856003246, + 2.227784799567939, + 1.9661939628020382, + 0.8564151878746421, + 0.685670545125312, + 0.7057093769255561, + 0.7040955511903956, + 1.0911066033830614, + 0.9451444056039292, + 0.9467396886001344, + 0.8560980697983398 + ], + "population_cv": 0.6331120880926366 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.24165314435958862, + "mean_weights": [ + 0.9284138083457947, + 0.07158620655536652 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5771778225898743, + "mean_weights": [ + 0.6417506337165833, + 0.35824936628341675 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5607964992523193, + "mean_weights": [ + 0.740641713142395, + 0.25935834646224976 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6847844123840332, + "mean_weights": [ + 0.4396772086620331, + 0.5603227615356445 + ], + "sources": 2 + }, + { + "entropy_mean": 0.46502435207366943, + "mean_weights": [ + 0.8168871998786926, + 0.18311277031898499 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6869450807571411, + "mean_weights": [ + 0.5358831286430359, + 0.4641168713569641 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6456354856491089, + "mean_weights": [ + 0.6021647453308105, + 0.39783525466918945 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6693196296691895, + "mean_weights": [ + 0.47989413142204285, + 0.5201058387756348 + ], + "sources": 2 + }, + { + "entropy_mean": 0.46101945638656616, + "mean_weights": [ + 0.07870025932788849, + 0.062126897275447845, + 0.8591728210449219 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0897139310836792, + "mean_weights": [ + 0.30999815464019775, + 0.3011889159679413, + 0.3888128995895386 + ], + "sources": 3 + }, + { + "entropy_mean": 0.913651704788208, + "mean_weights": [ + 0.1749667525291443, + 0.22547858953475952, + 0.5995546579360962 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0892587900161743, + "mean_weights": [ + 0.3452480137348175, + 0.276547372341156, + 0.3782046437263489 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0203359127044678, + "mean_weights": [ + 0.41242194175720215, + 0.19152171909809113, + 0.39605632424354553 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0860936641693115, + "mean_weights": [ + 0.3446395993232727, + 0.2708646357059479, + 0.3844957947731018 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0171257257461548, + "mean_weights": [ + 0.3257474899291992, + 0.22652524709701538, + 0.4477272629737854 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0872398614883423, + "mean_weights": [ + 0.3929253816604614, + 0.2865680158138275, + 0.32050663232803345 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3475291728973389, + "mean_weights": [ + 0.28426051139831543, + 0.1534726619720459, + 0.28338101506233215, + 0.2788858115673065 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3713825941085815, + "mean_weights": [ + 0.2825669050216675, + 0.22910134494304657, + 0.20437976717948914, + 0.28395193815231323 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3452380895614624, + "mean_weights": [ + 0.2726133465766907, + 0.18858514726161957, + 0.24317678809165955, + 0.2956247329711914 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3624985218048096, + "mean_weights": [ + 0.29694390296936035, + 0.2065664827823639, + 0.19368112087249756, + 0.3028084933757782 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3124669790267944, + "mean_weights": [ + 0.3300153613090515, + 0.15998056530952454, + 0.19470375776290894, + 0.3153003454208374 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3601500988006592, + "mean_weights": [ + 0.29828065633773804, + 0.2015766054391861, + 0.1924743950366974, + 0.3076683282852173 + ], + "sources": 4 + }, + { + "entropy_mean": 1.305145025253296, + "mean_weights": [ + 0.37038660049438477, + 0.1684686243534088, + 0.17173582315444946, + 0.2894088923931122 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3764257431030273, + "mean_weights": [ + 0.291445791721344, + 0.23688730597496033, + 0.21587741374969482, + 0.25578948855400085 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5886735916137695, + "mean_weights": [ + 0.23008812963962555, + 0.16205063462257385, + 0.17811012268066406, + 0.2101232409477234, + 0.21962788701057434 + ], + "sources": 5 + }, + { + "entropy_mean": 1.586238145828247, + "mean_weights": [ + 0.21088337898254395, + 0.1916535198688507, + 0.16390381753444672, + 0.1622629314661026, + 0.27129632234573364 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5744354724884033, + "mean_weights": [ + 0.206439808011055, + 0.15598373115062714, + 0.15800794959068298, + 0.24066726863384247, + 0.23890122771263123 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5702533721923828, + "mean_weights": [ + 0.1951810121536255, + 0.18971338868141174, + 0.15757259726524353, + 0.14954689145088196, + 0.3079860806465149 + ], + "sources": 5 + }, + { + "entropy_mean": 1.570713758468628, + "mean_weights": [ + 0.23311907052993774, + 0.15482673048973083, + 0.15546178817749023, + 0.17419752478599548, + 0.2823948860168457 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5437917709350586, + "mean_weights": [ + 0.19975396990776062, + 0.21470393240451813, + 0.14538854360580444, + 0.11585079878568649, + 0.3243027329444885 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5694549083709717, + "mean_weights": [ + 0.18292103707790375, + 0.17510700225830078, + 0.1714228093624115, + 0.24560697376728058, + 0.224942147731781 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5775599479675293, + "mean_weights": [ + 0.20127585530281067, + 0.19775623083114624, + 0.1430359035730362, + 0.17229337990283966, + 0.28563863039016724 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7800664901733398, + "mean_weights": [ + 0.16920313239097595, + 0.14216600358486176, + 0.16181352734565735, + 0.18052005767822266, + 0.1572553813457489, + 0.18904191255569458 + ], + "sources": 6 + }, + { + "entropy_mean": 1.748520851135254, + "mean_weights": [ + 0.16022704541683197, + 0.16041651368141174, + 0.12097706645727158, + 0.11627625674009323, + 0.176894873380661, + 0.26520824432373047 + ], + "sources": 6 + }, + { + "entropy_mean": 1.660531997680664, + "mean_weights": [ + 0.13148245215415955, + 0.10716301947832108, + 0.11324919760227203, + 0.1423710733652115, + 0.13752585649490356, + 0.3682084083557129 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7625548839569092, + "mean_weights": [ + 0.1663871705532074, + 0.1725061535835266, + 0.12895390391349792, + 0.11700935661792755, + 0.17722275853157043, + 0.2379206418991089 + ], + "sources": 6 + }, + { + "entropy_mean": 1.644116997718811, + "mean_weights": [ + 0.14135517179965973, + 0.11779841780662537, + 0.10928981751203537, + 0.15594087541103363, + 0.09924115985631943, + 0.37637460231781006 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7725964784622192, + "mean_weights": [ + 0.17268413305282593, + 0.18088045716285706, + 0.13958394527435303, + 0.1191333755850792, + 0.17517143487930298, + 0.21254664659500122 + ], + "sources": 6 + }, + { + "entropy_mean": 1.655699372291565, + "mean_weights": [ + 0.14339840412139893, + 0.12976261973381042, + 0.12792503833770752, + 0.15223471820354462, + 0.08974117040634155, + 0.35693806409835815 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7807013988494873, + "mean_weights": [ + 0.17964020371437073, + 0.19494879245758057, + 0.15304836630821228, + 0.12730863690376282, + 0.1718367338180542, + 0.1732172667980194 + ], + "sources": 6 + }, + { + "entropy_mean": 1.747307300567627, + "mean_weights": [ + 0.0962090790271759, + 0.090702123939991, + 0.0918470174074173, + 0.10531884431838989, + 0.0826181098818779, + 0.1527005136013031, + 0.3806043267250061 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8726190328598022, + "mean_weights": [ + 0.13168329000473022, + 0.14015334844589233, + 0.10547423362731934, + 0.09640335291624069, + 0.12434421479701996, + 0.1218581274151802, + 0.28008341789245605 + ], + "sources": 7 + }, + { + "entropy_mean": 1.6716570854187012, + "mean_weights": [ + 0.09257283806800842, + 0.08803431689739227, + 0.0836150050163269, + 0.09234962612390518, + 0.07381278276443481, + 0.13743700087070465, + 0.43217843770980835 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8947217464447021, + "mean_weights": [ + 0.1394326388835907, + 0.15343080461025238, + 0.11989420652389526, + 0.09237080812454224, + 0.13990652561187744, + 0.11003051698207855, + 0.24493449926376343 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7137484550476074, + "mean_weights": [ + 0.10278474539518356, + 0.09255793690681458, + 0.08860480785369873, + 0.10369215905666351, + 0.075224369764328, + 0.12985029816627502, + 0.4072856903076172 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9018497467041016, + "mean_weights": [ + 0.1455693244934082, + 0.16475746035575867, + 0.12093567848205566, + 0.0925828069448471, + 0.14651423692703247, + 0.10433352738618851, + 0.22530697286128998 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7591365575790405, + "mean_weights": [ + 0.11013476550579071, + 0.10883849859237671, + 0.10023228824138641, + 0.1035059317946434, + 0.08450647443532944, + 0.11719217896461487, + 0.37558984756469727 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0172319412231445, + "mean_weights": [ + 0.09959844499826431, + 0.12341248244047165, + 0.12437080591917038, + 0.1249062567949295, + 0.10500459372997284, + 0.09868454933166504, + 0.19231930375099182, + 0.13170355558395386 + ], + "sources": 8 + }, + { + "entropy_mean": 2.034546136856079, + "mean_weights": [ + 0.08726541697978973, + 0.09959165751934052, + 0.08329164236783981, + 0.08240707218647003, + 0.07135353237390518, + 0.0697166919708252, + 0.21859842538833618, + 0.12381041049957275, + 0.1639651358127594 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0888147354125977, + "mean_weights": [ + 0.07617583125829697, + 0.099439337849617, + 0.09872502833604813, + 0.1190575510263443, + 0.07568623125553131, + 0.09469741582870483, + 0.18636906147003174, + 0.09555765241384506, + 0.15429189801216125 + ], + "sources": 9 + }, + { + "entropy_mean": 2.020714521408081, + "mean_weights": [ + 0.07701840251684189, + 0.09260682761669159, + 0.08840936422348022, + 0.08414323627948761, + 0.07246369123458862, + 0.07894633710384369, + 0.22243204712867737, + 0.10568448156118393, + 0.17829561233520508 + ], + "sources": 9 + }, + { + "entropy_mean": 2.138826370239258, + "mean_weights": [ + 0.1031472384929657, + 0.13061866164207458, + 0.11922028660774231, + 0.09511415660381317, + 0.11113704741001129, + 0.08113724738359451, + 0.12610481679439545, + 0.10975348204374313, + 0.12376706302165985 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0196757316589355, + "mean_weights": [ + 0.08713073283433914, + 0.11073944717645645, + 0.08367414772510529, + 0.08517413586378098, + 0.07308490574359894, + 0.07473974674940109, + 0.20730704069137573, + 0.1178179681301117, + 0.1603318601846695 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1012635231018066, + "mean_weights": [ + 0.07784530520439148, + 0.10391807556152344, + 0.11372596770524979, + 0.130476176738739, + 0.07206802815198898, + 0.09977149963378906, + 0.20828433334827423, + 0.10122931003570557, + 0.09268133342266083 + ], + "sources": 9 + }, + { + "entropy_mean": 2.092306613922119, + "mean_weights": [ + 0.08883365988731384, + 0.11712762713432312, + 0.09774035215377808, + 0.108831487596035, + 0.08085808902978897, + 0.10952344536781311, + 0.18719899654388428, + 0.11508475244045258, + 0.09480157494544983 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05932779982686043, + 0.055799033492803574, + 0.0662136971950531, + 0.08507908135652542, + 0.15218402445316315, + 0.07682093232870102, + 0.1424744874238968, + 0.1285623162984848, + 0.15947651863098145, + 0.09444322437047958, + 0.11992523819208145, + 0.11550913751125336, + 0.11470400542020798, + 0.0961972326040268, + 0.11591855436563492, + 0.1105724573135376, + 0.1160203143954277, + 0.07375656813383102, + 0.08760861307382584, + 0.0797935277223587, + 0.08050386607646942, + 0.06967583298683167, + 0.0788898766040802, + 0.06891992688179016, + 0.08732876181602478, + 0.062183890491724014, + 0.06928366422653198, + 0.06255658715963364, + 0.07143163681030273, + 0.06459178775548935, + 0.08290247619152069, + 0.07782828062772751, + 0.0893530622124672, + 0.06204934045672417, + 0.0649322122335434, + 0.05308152362704277, + 0.06773758679628372, + 0.05756285786628723, + 0.07093854993581772, + 0.06636505573987961, + 0.07485510408878326, + 0.04315124452114105, + 0.054709821939468384, + 0.05043099820613861, + 0.060126665979623795, + 0.0598531998693943, + 0.06411325931549072, + 0.06925592571496964, + 0.06228240206837654, + 0.05212033912539482, + 0.05325160548090935, + 0.05331920459866524, + 0.05528927966952324, + 0.05580088123679161, + 0.058924708515405655, + 0.05906222388148308, + 0.07412761449813843, + 0.06625261157751083, + 0.064215287566185, + 0.06934450566768646, + 0.07083795964717865, + 0.08039923757314682, + 0.07262717187404633, + 0.07725618034601212 + ], + "loss_nats": 1.3358474969863892, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.8999089002609253, + "mean_weights": [ + 0.10558189451694489, + 0.07969948649406433, + 0.0799170732498169, + 0.05718890577554703, + 0.05987335741519928, + 0.05149075761437416, + 0.10265408456325531, + 0.13197818398475647, + 0.33161628246307373 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.07275605201721191, + 0.12060720473527908, + 0.24681445956230164, + 0.2573152482509613, + 0.2694833278656006, + 0.28323665261268616, + 0.28791290521621704, + 0.29319411516189575, + 0.09455645084381104, + 0.09563838690519333, + 0.10445866733789444, + 0.10590620338916779, + 0.11606099456548691, + 0.11774174869060516, + 0.12493877112865448, + 0.12761500477790833, + 0.0242336243391037, + 0.030837642028927803, + 0.0435737743973732, + 0.05576188862323761, + 0.07435832917690277, + 0.09026926010847092, + 0.1190997064113617, + 0.15066321194171906, + 0.0473942868411541, + 0.06286879628896713, + 0.08106257021427155, + 0.10371332615613937, + 0.11923341453075409, + 0.14144682884216309, + 0.18915845453739166, + 0.2107442319393158, + 0.03429211303591728, + 0.04767552763223648, + 0.06270946562290192, + 0.07339637726545334, + 0.10349760204553604, + 0.11630241572856903, + 0.1461056023836136, + 0.16868281364440918, + 0.05036674439907074, + 0.07512061297893524, + 0.08057750016450882, + 0.10792030394077301, + 0.11609549075365067, + 0.14635293185710907, + 0.15996946394443512, + 0.19755741953849792, + 0.09325341880321503, + 0.12152406573295593, + 0.134757399559021, + 0.16458973288536072, + 0.17996180057525635, + 0.2178809642791748, + 0.2225029021501541, + 0.27798205614089966, + 0.016754506155848503, + 0.07472676783800125, + 0.07638624310493469, + 0.1334167718887329, + 0.1364293098449707, + 0.21953867375850677, + 0.22130025923252106, + 0.32601314783096313 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00010430111433379352, + 0.00010754061077022925, + 0.00010278610716341063, + 0.00010147228022105992, + 0.00011407036072341725, + 0.0001072324303095229, + 9.987767407437786e-05, + 9.626782411942258e-05, + 0.00010618739179335535, + 0.00010225731966784224, + 9.589397086529061e-05, + 8.911445911508054e-05, + 8.166737097781152e-05, + 7.793214899720624e-05, + 7.684857700951397e-05, + 8.047092705965042e-05, + 0.0001293971436098218, + 0.0001289034989895299, + 0.00011100608389824629, + 9.808179311221465e-05, + 0.00019314092060085386, + 0.0001478801859775558, + 0.00011168832133989781, + 8.24130474939011e-05, + 4.944952161167748e-05, + 4.673326475312933e-05, + 4.401550177135505e-05, + 4.241442002239637e-05, + 8.204829646274447e-05, + 6.386488530552015e-05, + 4.887451359536499e-05, + 4.6167227992555127e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00010419355021440424, + "first_to_last_ratio": 1.967922805466359, + "imbalance_abs_log_ratio": 0.6769785729960915, + "last_quartile_mean": 5.294595393934287e-05, + "mean": 9.281247480430466e-05, + "normalized": [ + 1.1237833551331617, + 1.158687029916818, + 1.1074600411220088, + 1.0933043261157995, + 1.2290412572656306, + 1.1553665661391181, + 1.0761233798039562, + 1.0372293629967688, + 1.1441068888341974, + 1.1017626658857236, + 1.033201313373911, + 0.9601560490976951, + 0.8799180406514036, + 0.8396732137735404, + 0.8279983609051412, + 0.8670270589090915, + 1.3941783567635277, + 1.3888596254040555, + 1.1960254710619764, + 1.0567738153628634, + 2.0809801808225887, + 1.593322301655694, + 1.2033761794995008, + 0.8879522679216262, + 0.5327896030780552, + 0.5035235279704214, + 0.4742412252680671, + 0.4569905081383433, + 0.8840222894146883, + 0.6881066951417837, + 0.5265942288298747, + 0.4974248137429677 + ], + "population_cv": 0.3451413926042897 + }, + "activation_output_rms_by_block": [ + 0.10948990285396576, + 0.2626326084136963, + 0.28264743089675903, + 0.28982052206993103, + 0.07678548991680145, + 0.08437556773424149, + 0.09489941596984863, + 0.10230697691440582, + 0.025471342727541924, + 0.047026216983795166, + 0.07730481773614883, + 0.13858282566070557, + 0.05319376289844513, + 0.08911364525556564, + 0.11975770443677902, + 0.18381209671497345, + 0.03967120870947838, + 0.06193102151155472, + 0.10077209025621414, + 0.14859046041965485, + 0.06707533448934555, + 0.0977855920791626, + 0.13274765014648438, + 0.18068167567253113, + 0.11654389649629593, + 0.15409494936466217, + 0.19554980099201202, + 0.23872582614421844, + 0.06386543065309525, + 0.11006071418523788, + 0.1780913770198822, + 0.25628867745399475 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.1628697393462062, + "first_to_last_ratio": 0.9921850472230982, + "imbalance_abs_log_ratio": 0.00784564955418727, + "last_quartile_mean": 0.16415258403867483, + "mean": 0.13061550102429464, + "normalized": [ + 0.838261171111693, + 2.0107307812175086, + 2.163965445756597, + 2.2188830559707005, + 0.5878742516366358, + 0.6459843362584318, + 0.7265555406949534, + 0.7832682653445291, + 0.19501010621093298, + 0.36003549819900954, + 0.5918502561328463, + 1.0609983085769352, + 0.40725459444932977, + 0.6822593379555341, + 0.9168720672326934, + 1.4072762824741925, + 0.3037251199005812, + 0.4741475630831556, + 0.7715170823214191, + 1.137617352109049, + 0.5135327274583547, + 0.7486522756665333, + 1.0163238597675566, + 1.3833095938507645, + 0.8922669635866467, + 1.1797600449888435, + 1.497140840547245, + 1.8276990423963166, + 0.48895751386518965, + 0.8426313364197596, + 1.363478114184602, + 1.96216127063146 + ], + "population_cv": 0.5521985510612447 + }, + "bits_per_byte": 1.8516758040335026, + "branch_output_rms_by_sublayer": [ + 0.06820283830165863, + 0.08090997487306595, + 0.1809755116701126, + 0.020453093573451042, + 0.03570745885372162, + 0.02561795525252819, + 0.018777042627334595, + 0.013531037606298923, + 0.0755426213145256, + 0.005655187182128429, + 0.02343769371509552, + 0.00643240986391902, + 0.028086941689252853, + 0.013076604343950748, + 0.021053144708275795, + 0.014737329445779324, + 0.01945621147751808, + 0.01630411669611931, + 0.020461415871977806, + 0.01901024580001831, + 0.03094385378062725, + 0.0255262553691864, + 0.05226128548383713, + 0.04450787976384163, + 0.03883286938071251, + 0.029658861458301544, + 0.028961921110749245, + 0.03657185286283493, + 0.026584293693304062, + 0.03356487676501274, + 0.06814584881067276, + 0.03172357380390167, + 0.026314711198210716, + 0.02643820084631443, + 0.029159776866436005, + 0.024399247020483017, + 0.05293852463364601, + 0.03850069269537926, + 0.05838865786790848, + 0.059986162930727005, + 0.04273597151041031, + 0.048990398645401, + 0.02424224093556404, + 0.05808926746249199, + 0.03226586431264877, + 0.07184388488531113, + 0.04726063460111618, + 0.08994103968143463, + 0.09045237302780151, + 0.06303001940250397, + 0.042153000831604004, + 0.06376723200082779, + 0.04783050715923309, + 0.07607632875442505, + 0.02927425503730774, + 0.10138995200395584, + 0.013994279317557812, + 0.062000904232263565, + 0.012239223346114159, + 0.06553959101438522, + 0.01865031197667122, + 0.08940920978784561, + 0.015221481211483479, + 0.09842391312122345 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00035111741146770563, + 0.00025902548265502507, + 0.00018620685693891004, + 0.0001333512165747529, + 0.00018515338480069277, + 0.0001230557032091345, + 0.0001663276293820351, + 0.00017063175242936545, + 0.00016105811439522938, + 0.00016907699291556183, + 0.00022478671092279016, + 0.00031975601554814164, + 0.0002041658222050231, + 0.0002310970884368777, + 0.00022340376551087992, + 0.0002728161516085498, + 0.00032773482270883844, + 0.00038802914337115763, + 0.000557456242251675, + 0.0006025627442029069, + 0.0011594674683432322, + 0.00091227430589559, + 0.000805688922792764, + 0.0007107155549657487, + 0.0003299877704093532, + 0.000264082614725284, + 0.0002629495704413282, + 0.0002533431982080855, + 0.00042885190498764727, + 0.000356776698324927, + 0.0003321383609003025, + 0.00027606964378155345 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00019685867968220267, + "first_to_last_ratio": 0.6288912975293752, + "imbalance_abs_log_ratio": 0.46379685513475605, + "last_quartile_mean": 0.0003130249702223101, + "mean": 0.0003546612207909709, + "normalized": [ + 0.9900079029915879, + 0.7303462218884333, + 0.5250273952241766, + 0.37599604568368367, + 0.5220570334353467, + 0.3469668968450901, + 0.46897608092333487, + 0.4811119525524102, + 0.45411819774385004, + 0.4767281648060753, + 0.6338068493123308, + 0.9015815567177513, + 0.5756643530118386, + 0.6515995403204259, + 0.6299075072618354, + 0.7692302840443369, + 0.924078538888236, + 1.0940839331285475, + 1.5717992539709515, + 1.6989811935431292, + 3.269225391367486, + 2.572241486850527, + 2.271714175561411, + 2.0039280115844074, + 0.9304309325767542, + 0.7446052718600666, + 0.7414105490724191, + 0.7143244971724724, + 1.2091874720084006, + 1.0059647838837247, + 0.9364947206789691, + 0.7784038050899917 + ], + "population_cv": 0.6729885573692121 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.2297750860452652, + "mean_weights": [ + 0.9342420101165771, + 0.06575798988342285 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5617177486419678, + "mean_weights": [ + 0.6770219802856445, + 0.32297801971435547 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6259181499481201, + "mean_weights": [ + 0.6623371839523315, + 0.3376627564430237 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6780073642730713, + "mean_weights": [ + 0.4160298705101013, + 0.5839701294898987 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5229959487915039, + "mean_weights": [ + 0.7747018933296204, + 0.22529810667037964 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6889395117759705, + "mean_weights": [ + 0.5119822025299072, + 0.4880177974700928 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6652290225028992, + "mean_weights": [ + 0.5563431978225708, + 0.4436568021774292 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6795815229415894, + "mean_weights": [ + 0.4889851212501526, + 0.5110149383544922 + ], + "sources": 2 + }, + { + "entropy_mean": 0.34434884786605835, + "mean_weights": [ + 0.055761903524398804, + 0.04292435944080353, + 0.9013137817382812 + ], + "sources": 3 + }, + { + "entropy_mean": 1.08120858669281, + "mean_weights": [ + 0.2822376787662506, + 0.3003927171230316, + 0.4173696041107178 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9031982421875, + "mean_weights": [ + 0.15619418025016785, + 0.23383793234825134, + 0.6099678874015808 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0832421779632568, + "mean_weights": [ + 0.32599377632141113, + 0.2692318260669708, + 0.40477436780929565 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0103168487548828, + "mean_weights": [ + 0.34918153285980225, + 0.1856410801410675, + 0.46517735719680786 + ], + "sources": 3 + }, + { + "entropy_mean": 1.083648920059204, + "mean_weights": [ + 0.32806581258773804, + 0.2715883255004883, + 0.4003458619117737 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0123062133789062, + "mean_weights": [ + 0.29575595259666443, + 0.22443537414073944, + 0.4798086881637573 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0900384187698364, + "mean_weights": [ + 0.371364027261734, + 0.28760212659835815, + 0.3410338759422302 + ], + "sources": 3 + }, + { + "entropy_mean": 1.346717119216919, + "mean_weights": [ + 0.252036988735199, + 0.15496578812599182, + 0.3008464276790619, + 0.2921508550643921 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3694984912872314, + "mean_weights": [ + 0.2764948606491089, + 0.2209404706954956, + 0.20124787092208862, + 0.3013167977333069 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3397115468978882, + "mean_weights": [ + 0.2501073181629181, + 0.18987098336219788, + 0.25304114818573, + 0.30698060989379883 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3640000820159912, + "mean_weights": [ + 0.28623706102371216, + 0.2060294896364212, + 0.19822579622268677, + 0.30950766801834106 + ], + "sources": 4 + }, + { + "entropy_mean": 1.322603464126587, + "mean_weights": [ + 0.3028380572795868, + 0.16069075465202332, + 0.2121516764163971, + 0.3243195116519928 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3637139797210693, + "mean_weights": [ + 0.2874491214752197, + 0.2012495994567871, + 0.20179373025894165, + 0.3095075190067291 + ], + "sources": 4 + }, + { + "entropy_mean": 1.321109414100647, + "mean_weights": [ + 0.3375937044620514, + 0.17029225826263428, + 0.1850890815258026, + 0.3070249557495117 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3781479597091675, + "mean_weights": [ + 0.2801607549190521, + 0.22943861782550812, + 0.2231782078742981, + 0.26722243428230286 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5841326713562012, + "mean_weights": [ + 0.2102087140083313, + 0.15246617794036865, + 0.18049006164073944, + 0.2226225584745407, + 0.2342124879360199 + ], + "sources": 5 + }, + { + "entropy_mean": 1.585011601448059, + "mean_weights": [ + 0.20242321491241455, + 0.18982183933258057, + 0.1644069254398346, + 0.16460540890693665, + 0.27874264121055603 + ], + "sources": 5 + }, + { + "entropy_mean": 1.563858985900879, + "mean_weights": [ + 0.19176200032234192, + 0.14140060544013977, + 0.1587054431438446, + 0.25370848178863525, + 0.25442349910736084 + ], + "sources": 5 + }, + { + "entropy_mean": 1.56727135181427, + "mean_weights": [ + 0.18896782398223877, + 0.1815582811832428, + 0.1578895002603531, + 0.155012309551239, + 0.3165720999240875 + ], + "sources": 5 + }, + { + "entropy_mean": 1.560135841369629, + "mean_weights": [ + 0.22008571028709412, + 0.14016731083393097, + 0.1550605595111847, + 0.17754825949668884, + 0.3071381449699402 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5404894351959229, + "mean_weights": [ + 0.19314616918563843, + 0.2199089527130127, + 0.14857767522335052, + 0.11053609848022461, + 0.32783108949661255 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5632984638214111, + "mean_weights": [ + 0.17721346020698547, + 0.1596105694770813, + 0.17957568168640137, + 0.2437225878238678, + 0.23987770080566406 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5873030424118042, + "mean_weights": [ + 0.20342105627059937, + 0.2030828595161438, + 0.15223635733127594, + 0.1706695854663849, + 0.2705901265144348 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7768934965133667, + "mean_weights": [ + 0.1681121289730072, + 0.13532255589962006, + 0.16692501306533813, + 0.19713614881038666, + 0.15416869521141052, + 0.17833545804023743 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7485562562942505, + "mean_weights": [ + 0.1546163260936737, + 0.16004210710525513, + 0.12417291104793549, + 0.11629535257816315, + 0.17760780453681946, + 0.2672654986381531 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6414597034454346, + "mean_weights": [ + 0.12850892543792725, + 0.09477660059928894, + 0.11211013793945312, + 0.15507088601589203, + 0.12727181613445282, + 0.38226163387298584 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7658894062042236, + "mean_weights": [ + 0.16306474804878235, + 0.1769276112318039, + 0.1359354704618454, + 0.11684098839759827, + 0.174740731716156, + 0.2324904501438141 + ], + "sources": 6 + }, + { + "entropy_mean": 1.60666823387146, + "mean_weights": [ + 0.13450534641742706, + 0.10427224636077881, + 0.10384270548820496, + 0.16184298694133759, + 0.09028864651918411, + 0.4052480459213257 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7733936309814453, + "mean_weights": [ + 0.16830414533615112, + 0.18477420508861542, + 0.14625763893127441, + 0.11769066751003265, + 0.1719978153705597, + 0.2109755277633667 + ], + "sources": 6 + }, + { + "entropy_mean": 1.647918939590454, + "mean_weights": [ + 0.1405443549156189, + 0.1221591904759407, + 0.12497685104608536, + 0.15805450081825256, + 0.08781267702579498, + 0.3664524257183075 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7795891761779785, + "mean_weights": [ + 0.17636838555335999, + 0.20160673558712006, + 0.15837456285953522, + 0.12470819056034088, + 0.1714586466550827, + 0.16748350858688354 + ], + "sources": 6 + }, + { + "entropy_mean": 1.743251085281372, + "mean_weights": [ + 0.09491100907325745, + 0.08660215139389038, + 0.09464608877897263, + 0.10725297033786774, + 0.08001378923654556, + 0.15247948467731476, + 0.3840944766998291 + ], + "sources": 7 + }, + { + "entropy_mean": 1.874605655670166, + "mean_weights": [ + 0.1307772994041443, + 0.13972724974155426, + 0.11382026225328445, + 0.09566846489906311, + 0.12381766736507416, + 0.11732564121484756, + 0.27886343002319336 + ], + "sources": 7 + }, + { + "entropy_mean": 1.67841637134552, + "mean_weights": [ + 0.09184464812278748, + 0.08378953486680984, + 0.08575505018234253, + 0.10182991623878479, + 0.07102746516466141, + 0.13755403459072113, + 0.4281993508338928 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8973462581634521, + "mean_weights": [ + 0.1369848996400833, + 0.15450161695480347, + 0.12527018785476685, + 0.09363152086734772, + 0.13999822735786438, + 0.10804674029350281, + 0.2415667623281479 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7149667739868164, + "mean_weights": [ + 0.10129009932279587, + 0.08960527181625366, + 0.09035854041576385, + 0.11120589822530746, + 0.0718621164560318, + 0.1295810490846634, + 0.40609702467918396 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9010921716690063, + "mean_weights": [ + 0.14319470524787903, + 0.1676906794309616, + 0.12502367794513702, + 0.0911751464009285, + 0.14979220926761627, + 0.10053727775812149, + 0.22258628904819489 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7564094066619873, + "mean_weights": [ + 0.10970132052898407, + 0.10525630414485931, + 0.09957294911146164, + 0.11123809963464737, + 0.07975444942712784, + 0.11702363938093185, + 0.3774532675743103 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0291786193847656, + "mean_weights": [ + 0.10173255950212479, + 0.12468422949314117, + 0.1323830783367157, + 0.12290939688682556, + 0.11930540204048157, + 0.09622782468795776, + 0.17337492108345032, + 0.12938258051872253 + ], + "sources": 8 + }, + { + "entropy_mean": 2.000657081604004, + "mean_weights": [ + 0.08407881110906601, + 0.09390538930892944, + 0.07879739254713058, + 0.07974676787853241, + 0.06416118144989014, + 0.06447971612215042, + 0.2076210230588913, + 0.13026511669158936, + 0.19694459438323975 + ], + "sources": 9 + }, + { + "entropy_mean": 2.111039638519287, + "mean_weights": [ + 0.0761786699295044, + 0.10082964599132538, + 0.11319905519485474, + 0.12796571850776672, + 0.08496169000864029, + 0.09271451085805893, + 0.15376174449920654, + 0.09040819853544235, + 0.15998075902462006 + ], + "sources": 9 + }, + { + "entropy_mean": 1.9949814081192017, + "mean_weights": [ + 0.07406981289386749, + 0.08625134825706482, + 0.08813680708408356, + 0.07981229573488235, + 0.06944070756435394, + 0.0715879425406456, + 0.20321771502494812, + 0.10909592360258102, + 0.2183874547481537 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1417574882507324, + "mean_weights": [ + 0.10147816687822342, + 0.131428062915802, + 0.1290806084871292, + 0.09297427535057068, + 0.11756201833486557, + 0.0782734826207161, + 0.11685051769018173, + 0.10428153723478317, + 0.1280713528394699 + ], + "sources": 9 + }, + { + "entropy_mean": 1.9728388786315918, + "mean_weights": [ + 0.0842263326048851, + 0.1015067845582962, + 0.07838370651006699, + 0.08320709317922592, + 0.06520986557006836, + 0.06523947417736053, + 0.19323685765266418, + 0.12399101257324219, + 0.20499888062477112 + ], + "sources": 9 + }, + { + "entropy_mean": 2.111469268798828, + "mean_weights": [ + 0.07709462940692902, + 0.09934261441230774, + 0.12177299708127975, + 0.13947510719299316, + 0.07755875587463379, + 0.09407972544431686, + 0.1861572265625, + 0.09887102246284485, + 0.10564793646335602 + ], + "sources": 9 + }, + { + "entropy_mean": 2.082412004470825, + "mean_weights": [ + 0.08912709355354309, + 0.11207807064056396, + 0.0984480082988739, + 0.09966367483139038, + 0.07740098237991333, + 0.10569732636213303, + 0.17289775609970093, + 0.12048183381557465, + 0.12420523166656494 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06215258687734604, + 0.058602508157491684, + 0.06213507056236267, + 0.10184644162654877, + 0.16069963574409485, + 0.08485730737447739, + 0.14774948358535767, + 0.1375991851091385, + 0.15486319363117218, + 0.07774228602647781, + 0.11468139290809631, + 0.10723253339529037, + 0.10692181438207626, + 0.09051667153835297, + 0.10881098359823227, + 0.10183381289243698, + 0.11029750853776932, + 0.06815624982118607, + 0.08005055785179138, + 0.07420902699232101, + 0.07460519671440125, + 0.0639740377664566, + 0.07246750593185425, + 0.060992322862148285, + 0.07624820619821548, + 0.05172989144921303, + 0.06137692928314209, + 0.051088910549879074, + 0.06176261603832245, + 0.05370519310235977, + 0.07504400610923767, + 0.06572356075048447, + 0.0777428075671196, + 0.052333567291498184, + 0.057081419974565506, + 0.042210403829813004, + 0.060358330607414246, + 0.047470901161432266, + 0.06285852193832397, + 0.05563334375619888, + 0.06698674708604813, + 0.0358138233423233, + 0.04760207608342171, + 0.04306061193346977, + 0.05306747928261757, + 0.05196848884224892, + 0.05752786248922348, + 0.06134935840964317, + 0.053416959941387177, + 0.04408494010567665, + 0.044951580464839935, + 0.04511101543903351, + 0.04671613872051239, + 0.04714162275195122, + 0.0496322326362133, + 0.049752429127693176, + 0.061137303709983826, + 0.05715201422572136, + 0.05183509364724159, + 0.05826110020279884, + 0.057258833199739456, + 0.0694499984383583, + 0.06018918380141258, + 0.06646695733070374 + ], + "loss_nats": 1.283483862876892, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.797360897064209, + "mean_weights": [ + 0.10202459990978241, + 0.07120659947395325, + 0.07440070807933807, + 0.054721247404813766, + 0.05339593440294266, + 0.043382428586483, + 0.08416780829429626, + 0.12605667114257812, + 0.39064404368400574 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.06820283830165863, + 0.10948990285396576, + 0.2553703188896179, + 0.2626326084136963, + 0.27410775423049927, + 0.28264743089675903, + 0.2862999737262726, + 0.28982052206993103, + 0.0755426213145256, + 0.07678548991680145, + 0.08286485821008682, + 0.08437556773424149, + 0.09355434030294418, + 0.09489941596984863, + 0.10012844204902649, + 0.10230697691440582, + 0.01945621147751808, + 0.025471342727541924, + 0.03668670356273651, + 0.047026216983795166, + 0.06381506472826004, + 0.07730481773614883, + 0.10937535017728806, + 0.13858282566070557, + 0.03883286938071251, + 0.05319376289844513, + 0.06728078424930573, + 0.08911364525556564, + 0.09941593557596207, + 0.11975770443677902, + 0.16529875993728638, + 0.18381209671497345, + 0.026314711198210716, + 0.03967120870947838, + 0.05123560130596161, + 0.06193102151155472, + 0.0871388167142868, + 0.10077209025621414, + 0.12466661632061005, + 0.14859046041965485, + 0.04273597151041031, + 0.06707533448934555, + 0.07142169773578644, + 0.0977855920791626, + 0.10372500866651535, + 0.13274765014648438, + 0.14468589425086975, + 0.18068167567253113, + 0.09045237302780151, + 0.11654389649629593, + 0.12876485288143158, + 0.15409494936466217, + 0.1657731682062149, + 0.19554980099201202, + 0.19888760149478912, + 0.23872582614421844, + 0.013994279317557812, + 0.06386543065309525, + 0.06539583951234818, + 0.11006071418523788, + 0.11213613301515579, + 0.1780913770198822, + 0.17916782200336456, + 0.25628867745399475 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 8.034013546791773, + "cross_entropy_nats": 5.5687538385391235, + "step": 0 + }, + { + "bits_per_byte": 5.132769201348395, + "cross_entropy_nats": 3.5577645003795624, + "step": 100 + }, + { + "bits_per_byte": 3.090294726560462, + "cross_entropy_nats": 2.1420290768146515, + "step": 500 + }, + { + "bits_per_byte": 1.9592917980277693, + "cross_entropy_nats": 1.358077585697174, + "step": 2000 + }, + { + "bits_per_byte": 1.798009563736343, + "cross_entropy_nats": 1.2462852597236633, + "step": 4000 + }, + { + "bits_per_byte": 1.7150680711461828, + "cross_entropy_nats": 1.1887945979833603, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.48.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.48.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "48": true, + "49": true, + "50": true, + "51": true, + "52": true, + "53": true, + "54": true, + "55": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_group_7_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "0695d9627784a176678ce2545746c3e9d15fccb00bd852bcbf0a473f878800cf", + "final_model_state": "e10332275d8102ee43a6b19fc51dfc805f29261e119c83a4b8fb42164b8ed5b1", + "final_optimizer_state": "a434e781357c00cf5966ce4484f34d66cb5a13e4604bba24fd277cc844b2aa46", + "final_public_parameters": "f619aee8395b6c72c9ac3f0eddcd16e20acfbc4a5812199f504935007e98c54a", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "bbf2cfc5c0019fb54bf6927274ab9b185b564a77b4b8ef2dc6f5d81575f835c2" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "add6249175ac3bb3fa911069fbe8f25d17d9ab05df34b66ee89996983522ce83", + "1": "619b5ee4f70e975312ead9adf5df4f9d8584ce2839369c749442e429923405cf", + "7999": "fe22994f7750c2f8323fdb5ac51c9be1e208733abaf6a337f5b1ae1df1cd2730" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "b533586129c36c6094c9e1e537557a106f510f9958b1029e80152cc45bfb544d", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073002, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 209.1048789904587, + "measured_steps": 7980, + "median_ms": 209.48190649505705, + "p95_ms": 211.4643468768918, + "peak_allocated_bytes": 11621543936, + "peak_reserved_bytes": 12083789824, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 8.018870646365954, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.558257579803467, + "step": 1, + "unclipped_grad_norm": 26.833614349365234 + }, + { + "bits_per_byte": 7.214333741837924, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.0005950927734375, + "step": 10, + "unclipped_grad_norm": 12.741315841674805 + }, + { + "bits_per_byte": 6.684427677305473, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.633292198181152, + "step": 20, + "unclipped_grad_norm": 3.454456090927124 + }, + { + "bits_per_byte": 6.541226667029857, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.534032821655273, + "step": 30, + "unclipped_grad_norm": 3.0606353282928467 + }, + { + "bits_per_byte": 6.411199527810332, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.443904876708984, + "step": 40, + "unclipped_grad_norm": 3.000077724456787 + }, + { + "bits_per_byte": 6.085839997185657, + "learning_rate": 3.75e-05, + "loss_nats": 4.218382835388184, + "step": 50, + "unclipped_grad_norm": 2.9765207767486572 + }, + { + "bits_per_byte": 5.879723605640805, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.07551383972168, + "step": 60, + "unclipped_grad_norm": 2.7090725898742676 + }, + { + "bits_per_byte": 5.710479606510336, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.958202838897705, + "step": 70, + "unclipped_grad_norm": 2.5605835914611816 + }, + { + "bits_per_byte": 5.512398932344397, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.820903778076172, + "step": 80, + "unclipped_grad_norm": 2.552492380142212 + }, + { + "bits_per_byte": 5.341720941871034, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.702598810195923, + "step": 90, + "unclipped_grad_norm": 2.3851852416992188 + }, + { + "bits_per_byte": 5.1700182787638305, + "learning_rate": 7.5e-05, + "loss_nats": 3.5835835933685303, + "step": 100, + "unclipped_grad_norm": 2.2464096546173096 + }, + { + "bits_per_byte": 5.001330471526251, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.466658115386963, + "step": 110, + "unclipped_grad_norm": 2.1938729286193848 + }, + { + "bits_per_byte": 4.734189475975749, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.2814900875091553, + "step": 120, + "unclipped_grad_norm": 1.882785677909851 + }, + { + "bits_per_byte": 4.5854998113804974, + "learning_rate": 9.75e-05, + "loss_nats": 3.1784262657165527, + "step": 130, + "unclipped_grad_norm": 1.664886474609375 + }, + { + "bits_per_byte": 4.392287622095184, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.044501781463623, + "step": 140, + "unclipped_grad_norm": 1.4583598375320435 + }, + { + "bits_per_byte": 4.260272019466383, + "learning_rate": 0.0001125, + "loss_nats": 2.952995538711548, + "step": 150, + "unclipped_grad_norm": 1.233046054840088 + }, + { + "bits_per_byte": 4.1527501859734635, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.878467082977295, + "step": 160, + "unclipped_grad_norm": 1.0419447422027588 + }, + { + "bits_per_byte": 3.9965011959421886, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.7701635360717773, + "step": 170, + "unclipped_grad_norm": 0.9107012152671814 + }, + { + "bits_per_byte": 3.856395873382103, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.6730499267578125, + "step": 180, + "unclipped_grad_norm": 1.0191558599472046 + }, + { + "bits_per_byte": 3.873603769496832, + "learning_rate": 0.0001425, + "loss_nats": 2.6849775314331055, + "step": 190, + "unclipped_grad_norm": 0.9363032579421997 + }, + { + "bits_per_byte": 3.744209870397784, + "learning_rate": 0.00015, + "loss_nats": 2.5952885150909424, + "step": 200, + "unclipped_grad_norm": 0.7475519776344299 + }, + { + "bits_per_byte": 3.6893518763214708, + "learning_rate": 0.0001575, + "loss_nats": 2.5572638511657715, + "step": 210, + "unclipped_grad_norm": 0.7676230669021606 + }, + { + "bits_per_byte": 3.6586082576532353, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.535953998565674, + "step": 220, + "unclipped_grad_norm": 0.8576812148094177 + }, + { + "bits_per_byte": 3.672157394853509, + "learning_rate": 0.0001725, + "loss_nats": 2.5453455448150635, + "step": 230, + "unclipped_grad_norm": 0.6714520454406738 + }, + { + "bits_per_byte": 3.6609334630929005, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.5375657081604004, + "step": 240, + "unclipped_grad_norm": 0.8629013895988464 + }, + { + "bits_per_byte": 3.521832175248552, + "learning_rate": 0.0001875, + "loss_nats": 2.441148042678833, + "step": 250, + "unclipped_grad_norm": 0.9965693354606628 + }, + { + "bits_per_byte": 3.524218950477226, + "learning_rate": 0.000195, + "loss_nats": 2.4428024291992188, + "step": 260, + "unclipped_grad_norm": 1.4807337522506714 + }, + { + "bits_per_byte": 3.4849972750520575, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.415616035461426, + "step": 270, + "unclipped_grad_norm": 0.8415420651435852 + }, + { + "bits_per_byte": 3.4979042290340097, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.424562454223633, + "step": 280, + "unclipped_grad_norm": 0.938925564289093 + }, + { + "bits_per_byte": 3.391529863787647, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.3508293628692627, + "step": 290, + "unclipped_grad_norm": 0.8912168741226196 + }, + { + "bits_per_byte": 3.488251874736985, + "learning_rate": 0.000225, + "loss_nats": 2.4178719520568848, + "step": 300, + "unclipped_grad_norm": 1.450618028640747 + }, + { + "bits_per_byte": 3.382848523537394, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.3448119163513184, + "step": 310, + "unclipped_grad_norm": 1.0444859266281128 + }, + { + "bits_per_byte": 3.3669628300424983, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.333800792694092, + "step": 320, + "unclipped_grad_norm": 1.2737646102905273 + }, + { + "bits_per_byte": 3.350645116128635, + "learning_rate": 0.0002475, + "loss_nats": 2.3224902153015137, + "step": 330, + "unclipped_grad_norm": 1.0999655723571777 + }, + { + "bits_per_byte": 3.3492740704359565, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.321539878845215, + "step": 340, + "unclipped_grad_norm": 0.8235205411911011 + }, + { + "bits_per_byte": 3.331560201362389, + "learning_rate": 0.0002625, + "loss_nats": 2.3092615604400635, + "step": 350, + "unclipped_grad_norm": 0.8199692368507385 + }, + { + "bits_per_byte": 3.30965821077501, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.2940802574157715, + "step": 360, + "unclipped_grad_norm": 1.1285293102264404 + }, + { + "bits_per_byte": 3.315466408859949, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.2981061935424805, + "step": 370, + "unclipped_grad_norm": 1.0061535835266113 + }, + { + "bits_per_byte": 3.349600837472596, + "learning_rate": 0.000285, + "loss_nats": 2.3217663764953613, + "step": 380, + "unclipped_grad_norm": 0.9340975284576416 + }, + { + "bits_per_byte": 3.299866550530787, + "learning_rate": 0.0002925, + "loss_nats": 2.2872931957244873, + "step": 390, + "unclipped_grad_norm": 0.8931443691253662 + }, + { + "bits_per_byte": 3.2367840021249124, + "learning_rate": 0.0003, + "loss_nats": 2.243567705154419, + "step": 400, + "unclipped_grad_norm": 1.2228831052780151 + }, + { + "bits_per_byte": 3.1942103848344274, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.2140579223632812, + "step": 410, + "unclipped_grad_norm": 0.9069884419441223 + }, + { + "bits_per_byte": 3.223513820784338, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.2343695163726807, + "step": 420, + "unclipped_grad_norm": 1.3727779388427734 + }, + { + "bits_per_byte": 3.1941945624305483, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.2140469551086426, + "step": 430, + "unclipped_grad_norm": 0.9241280555725098 + }, + { + "bits_per_byte": 3.210989356217905, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.2256882190704346, + "step": 440, + "unclipped_grad_norm": 1.1470668315887451 + }, + { + "bits_per_byte": 3.1761460151184013, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.2015366554260254, + "step": 450, + "unclipped_grad_norm": 1.2246326208114624 + }, + { + "bits_per_byte": 3.1324280252690757, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.171233654022217, + "step": 460, + "unclipped_grad_norm": 1.0958805084228516 + }, + { + "bits_per_byte": 3.155777077880814, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.187417984008789, + "step": 470, + "unclipped_grad_norm": 1.1011202335357666 + }, + { + "bits_per_byte": 3.1172495244345275, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.160712718963623, + "step": 480, + "unclipped_grad_norm": 1.0157102346420288 + }, + { + "bits_per_byte": 3.090977110723423, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.1425020694732666, + "step": 490, + "unclipped_grad_norm": 1.0723968744277954 + }, + { + "bits_per_byte": 3.1139928609577896, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.1584553718566895, + "step": 500, + "unclipped_grad_norm": 1.0149942636489868 + }, + { + "bits_per_byte": 3.095357508840899, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.145538330078125, + "step": 510, + "unclipped_grad_norm": 1.0867739915847778 + }, + { + "bits_per_byte": 2.9719764671813165, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.0600171089172363, + "step": 520, + "unclipped_grad_norm": 1.104459285736084 + }, + { + "bits_per_byte": 3.0612526612442004, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.121898651123047, + "step": 530, + "unclipped_grad_norm": 0.8698751330375671 + }, + { + "bits_per_byte": 3.004157172880162, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.0823230743408203, + "step": 540, + "unclipped_grad_norm": 0.941727340221405 + }, + { + "bits_per_byte": 2.962734119523949, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.0536108016967773, + "step": 550, + "unclipped_grad_norm": 1.0719168186187744 + }, + { + "bits_per_byte": 2.972056267131317, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.060072422027588, + "step": 560, + "unclipped_grad_norm": 1.06044340133667 + }, + { + "bits_per_byte": 2.9569561903855623, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.0496058464050293, + "step": 570, + "unclipped_grad_norm": 1.0216634273529053 + }, + { + "bits_per_byte": 2.849740142045877, + "learning_rate": 0.000299626473885547, + "loss_nats": 1.9752893447875977, + "step": 580, + "unclipped_grad_norm": 0.8320088386535645 + }, + { + "bits_per_byte": 2.9075655247801775, + "learning_rate": 0.00029958384005397226, + "loss_nats": 2.0153708457946777, + "step": 590, + "unclipped_grad_norm": 1.0326138734817505 + }, + { + "bits_per_byte": 2.824461100126156, + "learning_rate": 0.0002995389065559004, + "loss_nats": 1.9577672481536865, + "step": 600, + "unclipped_grad_norm": 0.9783652424812317 + }, + { + "bits_per_byte": 2.8831694417899865, + "learning_rate": 0.000299491674159121, + "loss_nats": 1.9984607696533203, + "step": 610, + "unclipped_grad_norm": 0.9224134087562561 + }, + { + "bits_per_byte": 2.774689664931761, + "learning_rate": 0.00029944214367070535, + "loss_nats": 1.9232683181762695, + "step": 620, + "unclipped_grad_norm": 1.0137561559677124 + }, + { + "bits_per_byte": 2.7800847606893275, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.9270079135894775, + "step": 630, + "unclipped_grad_norm": 0.9512865543365479 + }, + { + "bits_per_byte": 2.779256836208074, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.92643404006958, + "step": 640, + "unclipped_grad_norm": 0.9055118560791016 + }, + { + "bits_per_byte": 2.7097476401047578, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.8782539367675781, + "step": 650, + "unclipped_grad_norm": 0.9712872505187988 + }, + { + "bits_per_byte": 2.6894005446638296, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.8641504049301147, + "step": 660, + "unclipped_grad_norm": 1.3138395547866821 + }, + { + "bits_per_byte": 2.696550895356102, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.869106650352478, + "step": 670, + "unclipped_grad_norm": 0.9996857643127441 + }, + { + "bits_per_byte": 2.7083721228631568, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.877300500869751, + "step": 680, + "unclipped_grad_norm": 1.0686421394348145 + }, + { + "bits_per_byte": 2.6112352897599376, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.8099703788757324, + "step": 690, + "unclipped_grad_norm": 1.0986355543136597 + }, + { + "bits_per_byte": 2.5454563973538358, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.764375925064087, + "step": 700, + "unclipped_grad_norm": 0.9605162143707275 + }, + { + "bits_per_byte": 2.6479955494859575, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.8354506492614746, + "step": 710, + "unclipped_grad_norm": 1.5879634618759155 + }, + { + "bits_per_byte": 2.689204656424497, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.8640146255493164, + "step": 720, + "unclipped_grad_norm": 0.8233228921890259 + }, + { + "bits_per_byte": 2.495427332148448, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.7296984195709229, + "step": 730, + "unclipped_grad_norm": 0.87166827917099 + }, + { + "bits_per_byte": 2.5372468055148993, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.7586854696273804, + "step": 740, + "unclipped_grad_norm": 1.2335482835769653 + }, + { + "bits_per_byte": 2.5404658047910984, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.7609167098999023, + "step": 750, + "unclipped_grad_norm": 0.870635986328125 + }, + { + "bits_per_byte": 2.5220898025117093, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.7481794357299805, + "step": 760, + "unclipped_grad_norm": 1.3706663846969604 + }, + { + "bits_per_byte": 2.496485541399207, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.7304319143295288, + "step": 770, + "unclipped_grad_norm": 1.0551260709762573 + }, + { + "bits_per_byte": 2.427846061561304, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.6828546524047852, + "step": 780, + "unclipped_grad_norm": 1.0029908418655396 + }, + { + "bits_per_byte": 2.501523773156233, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.733924150466919, + "step": 790, + "unclipped_grad_norm": 1.1062501668930054 + }, + { + "bits_per_byte": 2.446479693851686, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.695770502090454, + "step": 800, + "unclipped_grad_norm": 1.0662990808486938 + }, + { + "bits_per_byte": 2.488364176660163, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7248026132583618, + "step": 810, + "unclipped_grad_norm": 1.2042149305343628 + }, + { + "bits_per_byte": 2.4472537877632194, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.6963070631027222, + "step": 820, + "unclipped_grad_norm": 0.9396573305130005 + }, + { + "bits_per_byte": 2.434023334414994, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.6871364116668701, + "step": 830, + "unclipped_grad_norm": 1.0756558179855347 + }, + { + "bits_per_byte": 2.4110442164852617, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.6712085008621216, + "step": 840, + "unclipped_grad_norm": 1.093858242034912 + }, + { + "bits_per_byte": 2.3413303610274188, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.6228865385055542, + "step": 850, + "unclipped_grad_norm": 0.8437754511833191 + }, + { + "bits_per_byte": 2.412108445128801, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.6719461679458618, + "step": 860, + "unclipped_grad_norm": 1.1316680908203125 + }, + { + "bits_per_byte": 2.346336947976686, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.626356840133667, + "step": 870, + "unclipped_grad_norm": 0.9768064022064209 + }, + { + "bits_per_byte": 2.3935282994428375, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.6590673923492432, + "step": 880, + "unclipped_grad_norm": 0.9414467811584473 + }, + { + "bits_per_byte": 2.372501012617747, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.6444923877716064, + "step": 890, + "unclipped_grad_norm": 0.8692553043365479 + }, + { + "bits_per_byte": 2.3103862105882773, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.6014376878738403, + "step": 900, + "unclipped_grad_norm": 1.0080275535583496 + }, + { + "bits_per_byte": 2.3553295768250004, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.6325900554656982, + "step": 910, + "unclipped_grad_norm": 1.2455649375915527 + }, + { + "bits_per_byte": 2.290456861006295, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.5876237154006958, + "step": 920, + "unclipped_grad_norm": 1.0591238737106323 + }, + { + "bits_per_byte": 2.3198868761872404, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.6080230474472046, + "step": 930, + "unclipped_grad_norm": 0.947547435760498 + }, + { + "bits_per_byte": 2.261559820060373, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.5675938129425049, + "step": 940, + "unclipped_grad_norm": 1.405505657196045 + }, + { + "bits_per_byte": 2.2492379510566565, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.5590529441833496, + "step": 950, + "unclipped_grad_norm": 1.0640307664871216 + }, + { + "bits_per_byte": 2.262126330912315, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.5679864883422852, + "step": 960, + "unclipped_grad_norm": 1.2646914720535278 + }, + { + "bits_per_byte": 2.2733535303432904, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5757685899734497, + "step": 970, + "unclipped_grad_norm": 0.9794231057167053 + }, + { + "bits_per_byte": 2.2450408864450004, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.5561437606811523, + "step": 980, + "unclipped_grad_norm": 0.9477900862693787 + }, + { + "bits_per_byte": 2.1799691866992466, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.5110394954681396, + "step": 990, + "unclipped_grad_norm": 0.9894558191299438 + }, + { + "bits_per_byte": 2.2379091098790216, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.5512003898620605, + "step": 1000, + "unclipped_grad_norm": 1.1689600944519043 + }, + { + "bits_per_byte": 2.2155228161468132, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.5356833934783936, + "step": 1010, + "unclipped_grad_norm": 0.8594710826873779 + }, + { + "bits_per_byte": 2.2282065366479453, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.5444750785827637, + "step": 1020, + "unclipped_grad_norm": 0.9485281109809875 + }, + { + "bits_per_byte": 2.142923435772794, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.4853613376617432, + "step": 1030, + "unclipped_grad_norm": 0.9571923613548279 + }, + { + "bits_per_byte": 2.2326815252233954, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.547576904296875, + "step": 1040, + "unclipped_grad_norm": 1.0019253492355347 + }, + { + "bits_per_byte": 2.203188564392234, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5271339416503906, + "step": 1050, + "unclipped_grad_norm": 1.0869957208633423 + }, + { + "bits_per_byte": 2.1524289168859814, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.4919500350952148, + "step": 1060, + "unclipped_grad_norm": 0.997162401676178 + }, + { + "bits_per_byte": 2.2603824268325665, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.5667777061462402, + "step": 1070, + "unclipped_grad_norm": 0.8555520176887512 + }, + { + "bits_per_byte": 2.193970294305988, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.5207443237304688, + "step": 1080, + "unclipped_grad_norm": 1.00409734249115 + }, + { + "bits_per_byte": 2.1135825076049954, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.4650237560272217, + "step": 1090, + "unclipped_grad_norm": 0.8757327198982239 + }, + { + "bits_per_byte": 2.2096807374796543, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.531633973121643, + "step": 1100, + "unclipped_grad_norm": 0.9500272870063782 + }, + { + "bits_per_byte": 2.228721968652581, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.5448323488235474, + "step": 1110, + "unclipped_grad_norm": 0.9051864743232727 + }, + { + "bits_per_byte": 2.0999171101327385, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.4555516242980957, + "step": 1120, + "unclipped_grad_norm": 0.8793370723724365 + }, + { + "bits_per_byte": 2.1174923611897105, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.4677338600158691, + "step": 1130, + "unclipped_grad_norm": 0.8832807540893555 + }, + { + "bits_per_byte": 2.121381920821624, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.4704298973083496, + "step": 1140, + "unclipped_grad_norm": 1.124153971672058 + }, + { + "bits_per_byte": 2.1060974786741444, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.4598355293273926, + "step": 1150, + "unclipped_grad_norm": 0.8464806079864502 + }, + { + "bits_per_byte": 2.0499162500820924, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.420893669128418, + "step": 1160, + "unclipped_grad_norm": 0.8455079197883606 + }, + { + "bits_per_byte": 2.1718032784536296, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.505379319190979, + "step": 1170, + "unclipped_grad_norm": 0.8364704251289368 + }, + { + "bits_per_byte": 2.0832922352044343, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.4440281391143799, + "step": 1180, + "unclipped_grad_norm": 0.9065543413162231 + }, + { + "bits_per_byte": 2.111357740033434, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.4634816646575928, + "step": 1190, + "unclipped_grad_norm": 0.9885900616645813 + }, + { + "bits_per_byte": 2.07246386354081, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.4365224838256836, + "step": 1200, + "unclipped_grad_norm": 0.8652650713920593 + }, + { + "bits_per_byte": 2.2129238143269743, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.5338819026947021, + "step": 1210, + "unclipped_grad_norm": 1.0179673433303833 + }, + { + "bits_per_byte": 2.065404835636146, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4316295385360718, + "step": 1220, + "unclipped_grad_norm": 0.8472744226455688 + }, + { + "bits_per_byte": 2.0120852263718003, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.3946712017059326, + "step": 1230, + "unclipped_grad_norm": 0.8921005129814148 + }, + { + "bits_per_byte": 2.010883411607571, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.3938381671905518, + "step": 1240, + "unclipped_grad_norm": 0.9352678060531616 + }, + { + "bits_per_byte": 2.1729849712477076, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.5061984062194824, + "step": 1250, + "unclipped_grad_norm": 0.9334508180618286 + }, + { + "bits_per_byte": 2.1138903565500398, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.4652371406555176, + "step": 1260, + "unclipped_grad_norm": 0.9316595196723938 + }, + { + "bits_per_byte": 1.995035382313214, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.3828531503677368, + "step": 1270, + "unclipped_grad_norm": 0.9936383366584778 + }, + { + "bits_per_byte": 2.137490503832038, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.481595516204834, + "step": 1280, + "unclipped_grad_norm": 0.919409453868866 + }, + { + "bits_per_byte": 2.020972257872485, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4008312225341797, + "step": 1290, + "unclipped_grad_norm": 0.8115113973617554 + }, + { + "bits_per_byte": 2.0664358716280686, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.4323441982269287, + "step": 1300, + "unclipped_grad_norm": 0.874910831451416 + }, + { + "bits_per_byte": 2.1354817464699605, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.4802031517028809, + "step": 1310, + "unclipped_grad_norm": 0.8457413911819458 + }, + { + "bits_per_byte": 2.068599929323875, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.4338442087173462, + "step": 1320, + "unclipped_grad_norm": 0.927187979221344 + }, + { + "bits_per_byte": 1.9624942009781603, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.3602973222732544, + "step": 1330, + "unclipped_grad_norm": 0.8361653685569763 + }, + { + "bits_per_byte": 1.9964213904965173, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.3838138580322266, + "step": 1340, + "unclipped_grad_norm": 0.8345865607261658 + }, + { + "bits_per_byte": 2.1564326729980676, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.494725227355957, + "step": 1350, + "unclipped_grad_norm": 0.8373364806175232 + }, + { + "bits_per_byte": 2.0113718423360214, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.394176721572876, + "step": 1360, + "unclipped_grad_norm": 0.9000253677368164 + }, + { + "bits_per_byte": 2.0690928316012482, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4341858625411987, + "step": 1370, + "unclipped_grad_norm": 0.8617348670959473 + }, + { + "bits_per_byte": 2.053337844921008, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.4232653379440308, + "step": 1380, + "unclipped_grad_norm": 0.7746536731719971 + }, + { + "bits_per_byte": 2.0770642274687394, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.4397112131118774, + "step": 1390, + "unclipped_grad_norm": 0.8930739760398865 + }, + { + "bits_per_byte": 2.0603043461595076, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.4280941486358643, + "step": 1400, + "unclipped_grad_norm": 0.8241319060325623 + }, + { + "bits_per_byte": 1.9907524983587774, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.3798844814300537, + "step": 1410, + "unclipped_grad_norm": 0.8087075352668762 + }, + { + "bits_per_byte": 2.0346235527673713, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.4102935791015625, + "step": 1420, + "unclipped_grad_norm": 0.8896597623825073 + }, + { + "bits_per_byte": 2.0308808663192948, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.4076993465423584, + "step": 1430, + "unclipped_grad_norm": 0.794548749923706 + }, + { + "bits_per_byte": 2.0100977948584298, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.3932936191558838, + "step": 1440, + "unclipped_grad_norm": 0.8715466856956482 + }, + { + "bits_per_byte": 2.05626447369074, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.4252939224243164, + "step": 1450, + "unclipped_grad_norm": 0.9222073554992676 + }, + { + "bits_per_byte": 1.9953391036746377, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.3830636739730835, + "step": 1460, + "unclipped_grad_norm": 0.8260480761528015 + }, + { + "bits_per_byte": 1.9909874266598562, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.38004732131958, + "step": 1470, + "unclipped_grad_norm": 0.8825197815895081 + }, + { + "bits_per_byte": 1.8950214514437105, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.3135287761688232, + "step": 1480, + "unclipped_grad_norm": 0.807368278503418 + }, + { + "bits_per_byte": 1.9967474696025533, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.3840398788452148, + "step": 1490, + "unclipped_grad_norm": 0.7884242534637451 + }, + { + "bits_per_byte": 1.9069929917876198, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.3218268156051636, + "step": 1500, + "unclipped_grad_norm": 0.7282363772392273 + }, + { + "bits_per_byte": 1.9927103488561997, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.3812415599822998, + "step": 1510, + "unclipped_grad_norm": 0.879385769367218 + }, + { + "bits_per_byte": 1.9797700302399803, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.37227201461792, + "step": 1520, + "unclipped_grad_norm": 0.7807854413986206 + }, + { + "bits_per_byte": 1.9256206046852216, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.3347384929656982, + "step": 1530, + "unclipped_grad_norm": 0.7780346274375916 + }, + { + "bits_per_byte": 2.013980991132261, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3959852457046509, + "step": 1540, + "unclipped_grad_norm": 0.7615655064582825 + }, + { + "bits_per_byte": 1.9894151612656683, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.3789575099945068, + "step": 1550, + "unclipped_grad_norm": 0.7858165502548218 + }, + { + "bits_per_byte": 2.0495932666637717, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.4206697940826416, + "step": 1560, + "unclipped_grad_norm": 0.8555475473403931 + }, + { + "bits_per_byte": 2.0134487048278404, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.3956162929534912, + "step": 1570, + "unclipped_grad_norm": 0.839836835861206 + }, + { + "bits_per_byte": 1.9926367402816305, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.381190538406372, + "step": 1580, + "unclipped_grad_norm": 0.799673318862915 + }, + { + "bits_per_byte": 2.0038273074079718, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3889472484588623, + "step": 1590, + "unclipped_grad_norm": 0.8408812880516052 + }, + { + "bits_per_byte": 1.9927292669477945, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.3812546730041504, + "step": 1600, + "unclipped_grad_norm": 0.8364356756210327 + }, + { + "bits_per_byte": 2.0163957995330257, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.3976590633392334, + "step": 1610, + "unclipped_grad_norm": 0.8611925840377808 + }, + { + "bits_per_byte": 2.0411272486923995, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.4148015975952148, + "step": 1620, + "unclipped_grad_norm": 0.8479971289634705 + }, + { + "bits_per_byte": 1.9166272879063035, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.3285048007965088, + "step": 1630, + "unclipped_grad_norm": 0.7962436079978943 + }, + { + "bits_per_byte": 2.017438358362556, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.3983817100524902, + "step": 1640, + "unclipped_grad_norm": 0.8221224546432495 + }, + { + "bits_per_byte": 1.9654077590664272, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.3623168468475342, + "step": 1650, + "unclipped_grad_norm": 0.7991349697113037 + }, + { + "bits_per_byte": 1.9400169284236455, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.344717264175415, + "step": 1660, + "unclipped_grad_norm": 0.7683122754096985 + }, + { + "bits_per_byte": 2.0956333662650475, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.4525823593139648, + "step": 1670, + "unclipped_grad_norm": 0.9254200458526611 + }, + { + "bits_per_byte": 2.0023885005508535, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.3879499435424805, + "step": 1680, + "unclipped_grad_norm": 0.8094135522842407 + }, + { + "bits_per_byte": 1.8932421189378839, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.3122954368591309, + "step": 1690, + "unclipped_grad_norm": 0.8024735450744629 + }, + { + "bits_per_byte": 1.930969093144404, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.338445782661438, + "step": 1700, + "unclipped_grad_norm": 0.8164986371994019 + }, + { + "bits_per_byte": 1.9992095732323052, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3857464790344238, + "step": 1710, + "unclipped_grad_norm": 0.8078791499137878 + }, + { + "bits_per_byte": 1.9968109312007216, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.384083867073059, + "step": 1720, + "unclipped_grad_norm": 0.7626392841339111 + }, + { + "bits_per_byte": 1.9019544160652921, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.3183343410491943, + "step": 1730, + "unclipped_grad_norm": 0.8557855486869812 + }, + { + "bits_per_byte": 1.950854931115718, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.3522295951843262, + "step": 1740, + "unclipped_grad_norm": 0.8147823214530945 + }, + { + "bits_per_byte": 2.055611455565414, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.424841284751892, + "step": 1750, + "unclipped_grad_norm": 0.8082402348518372 + }, + { + "bits_per_byte": 1.9132956399937893, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.326195478439331, + "step": 1760, + "unclipped_grad_norm": 0.8262012600898743 + }, + { + "bits_per_byte": 1.9723888788302502, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.3671557903289795, + "step": 1770, + "unclipped_grad_norm": 0.8193358182907104 + }, + { + "bits_per_byte": 1.9061896608254396, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.3212699890136719, + "step": 1780, + "unclipped_grad_norm": 0.8364750146865845 + }, + { + "bits_per_byte": 1.8828732849173653, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3051083087921143, + "step": 1790, + "unclipped_grad_norm": 0.8194389939308167 + }, + { + "bits_per_byte": 1.9609494528081104, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.3592265844345093, + "step": 1800, + "unclipped_grad_norm": 0.788703203201294 + }, + { + "bits_per_byte": 1.9596294859627383, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.358311653137207, + "step": 1810, + "unclipped_grad_norm": 0.7769515514373779 + }, + { + "bits_per_byte": 2.0159249110349635, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.3973326683044434, + "step": 1820, + "unclipped_grad_norm": 0.7803637981414795 + }, + { + "bits_per_byte": 1.8619116954649044, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.290578842163086, + "step": 1830, + "unclipped_grad_norm": 0.7515676617622375 + }, + { + "bits_per_byte": 1.9549832026670289, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.3550910949707031, + "step": 1840, + "unclipped_grad_norm": 0.7675890326499939 + }, + { + "bits_per_byte": 1.845600516891774, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.2792727947235107, + "step": 1850, + "unclipped_grad_norm": 0.7334516644477844 + }, + { + "bits_per_byte": 1.9955637130166646, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.3832193613052368, + "step": 1860, + "unclipped_grad_norm": 0.8520342707633972 + }, + { + "bits_per_byte": 2.0064283730196206, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.3907501697540283, + "step": 1870, + "unclipped_grad_norm": 0.7900378704071045 + }, + { + "bits_per_byte": 1.8610653688400085, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.2899922132492065, + "step": 1880, + "unclipped_grad_norm": 0.7505924105644226 + }, + { + "bits_per_byte": 1.9472825074919957, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3497533798217773, + "step": 1890, + "unclipped_grad_norm": 0.7831321358680725 + }, + { + "bits_per_byte": 1.8680425330028618, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.2948284149169922, + "step": 1900, + "unclipped_grad_norm": 0.7896786332130432 + }, + { + "bits_per_byte": 1.8564060148628332, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.2867625951766968, + "step": 1910, + "unclipped_grad_norm": 0.7299009561538696 + }, + { + "bits_per_byte": 1.9245498407009496, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.333996295928955, + "step": 1920, + "unclipped_grad_norm": 0.7537475824356079 + }, + { + "bits_per_byte": 1.8869117815249257, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.3079075813293457, + "step": 1930, + "unclipped_grad_norm": 0.7813870310783386 + }, + { + "bits_per_byte": 1.8543097183314665, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.2853095531463623, + "step": 1940, + "unclipped_grad_norm": 0.8040772080421448 + }, + { + "bits_per_byte": 1.91063678821145, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.324352502822876, + "step": 1950, + "unclipped_grad_norm": 0.9114683866500854 + }, + { + "bits_per_byte": 1.8794511741304971, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.3027362823486328, + "step": 1960, + "unclipped_grad_norm": 0.7730908393859863 + }, + { + "bits_per_byte": 1.8330727846549744, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.2705892324447632, + "step": 1970, + "unclipped_grad_norm": 0.7381971478462219 + }, + { + "bits_per_byte": 1.8138487359241804, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.2572641372680664, + "step": 1980, + "unclipped_grad_norm": 0.7530549168586731 + }, + { + "bits_per_byte": 1.9272045648996678, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.335836410522461, + "step": 1990, + "unclipped_grad_norm": 0.8273325562477112 + }, + { + "bits_per_byte": 1.8725211332139802, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.297932744026184, + "step": 2000, + "unclipped_grad_norm": 0.7813737988471985 + }, + { + "bits_per_byte": 1.9373781986114573, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.3428882360458374, + "step": 2010, + "unclipped_grad_norm": 0.7935295701026917 + }, + { + "bits_per_byte": 1.9137467504870023, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.3265081644058228, + "step": 2020, + "unclipped_grad_norm": 0.8239653706550598 + }, + { + "bits_per_byte": 1.9106557063030452, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.3243656158447266, + "step": 2030, + "unclipped_grad_norm": 0.8195171356201172 + }, + { + "bits_per_byte": 1.928864197480494, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.336986780166626, + "step": 2040, + "unclipped_grad_norm": 0.7979855537414551 + }, + { + "bits_per_byte": 1.8535075912478425, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.2847535610198975, + "step": 2050, + "unclipped_grad_norm": 0.7699401378631592 + }, + { + "bits_per_byte": 1.7680721137440267, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.225534200668335, + "step": 2060, + "unclipped_grad_norm": 0.7582734227180481 + }, + { + "bits_per_byte": 1.8705871883050278, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.2965922355651855, + "step": 2070, + "unclipped_grad_norm": 0.8142459392547607 + }, + { + "bits_per_byte": 1.948319046928746, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.3504718542099, + "step": 2080, + "unclipped_grad_norm": 0.7283441424369812 + }, + { + "bits_per_byte": 1.830672422796882, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.268925428390503, + "step": 2090, + "unclipped_grad_norm": 0.7204533815383911 + }, + { + "bits_per_byte": 1.9008731611393177, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.3175848722457886, + "step": 2100, + "unclipped_grad_norm": 0.7506802082061768 + }, + { + "bits_per_byte": 1.8253227304591435, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.2652173042297363, + "step": 2110, + "unclipped_grad_norm": 0.7168958187103271 + }, + { + "bits_per_byte": 1.8695023217433853, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.2958402633666992, + "step": 2120, + "unclipped_grad_norm": 0.7972914576530457 + }, + { + "bits_per_byte": 1.8823679998891303, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.304758071899414, + "step": 2130, + "unclipped_grad_norm": 0.7355724573135376 + }, + { + "bits_per_byte": 1.7812361817890185, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.2346588373184204, + "step": 2140, + "unclipped_grad_norm": 0.6865952610969543 + }, + { + "bits_per_byte": 1.8303987983993593, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2687357664108276, + "step": 2150, + "unclipped_grad_norm": 0.7372509241104126 + }, + { + "bits_per_byte": 1.8737392863300415, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.2987771034240723, + "step": 2160, + "unclipped_grad_norm": 0.753890335559845 + }, + { + "bits_per_byte": 1.847122391369259, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.2803276777267456, + "step": 2170, + "unclipped_grad_norm": 0.7440467476844788 + }, + { + "bits_per_byte": 1.7862572152809584, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.2381391525268555, + "step": 2180, + "unclipped_grad_norm": 0.7136292457580566 + }, + { + "bits_per_byte": 1.9008721292434125, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.3175841569900513, + "step": 2190, + "unclipped_grad_norm": 0.8136576414108276 + }, + { + "bits_per_byte": 1.946590105339622, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.349273443222046, + "step": 2200, + "unclipped_grad_norm": 0.7835270166397095 + }, + { + "bits_per_byte": 1.9543465228935346, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.3546497821807861, + "step": 2210, + "unclipped_grad_norm": 0.7719687819480896 + }, + { + "bits_per_byte": 1.8449313323972667, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2788089513778687, + "step": 2220, + "unclipped_grad_norm": 0.8071997165679932 + }, + { + "bits_per_byte": 1.8616636964823603, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.2904069423675537, + "step": 2230, + "unclipped_grad_norm": 0.7589682936668396 + }, + { + "bits_per_byte": 1.8205526196548127, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.2619109153747559, + "step": 2240, + "unclipped_grad_norm": 0.7538968920707703 + }, + { + "bits_per_byte": 1.811641510583007, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2557342052459717, + "step": 2250, + "unclipped_grad_norm": 0.7343966364860535 + }, + { + "bits_per_byte": 1.8584512325468938, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.2881802320480347, + "step": 2260, + "unclipped_grad_norm": 0.7512363195419312 + }, + { + "bits_per_byte": 1.893525374363855, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.312491774559021, + "step": 2270, + "unclipped_grad_norm": 0.7310166358947754 + }, + { + "bits_per_byte": 1.7853028835513205, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.2374776601791382, + "step": 2280, + "unclipped_grad_norm": 0.7221986651420593 + }, + { + "bits_per_byte": 1.8272737016505314, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.2665696144104004, + "step": 2290, + "unclipped_grad_norm": 0.7432793974876404 + }, + { + "bits_per_byte": 1.8754620365437342, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.2999712228775024, + "step": 2300, + "unclipped_grad_norm": 0.7622602581977844 + }, + { + "bits_per_byte": 1.8406768255802226, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.2758599519729614, + "step": 2310, + "unclipped_grad_norm": 0.7708092331886292 + }, + { + "bits_per_byte": 1.8417070016588908, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.2765740156173706, + "step": 2320, + "unclipped_grad_norm": 0.7121317386627197 + }, + { + "bits_per_byte": 1.791599340382059, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.2418420314788818, + "step": 2330, + "unclipped_grad_norm": 0.7635183334350586 + }, + { + "bits_per_byte": 1.8205443644875714, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.2619051933288574, + "step": 2340, + "unclipped_grad_norm": 0.8071432113647461 + }, + { + "bits_per_byte": 1.740452387946063, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2063896656036377, + "step": 2350, + "unclipped_grad_norm": 0.6944671869277954 + }, + { + "bits_per_byte": 1.8400327505527414, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.2754135131835938, + "step": 2360, + "unclipped_grad_norm": 0.7582014203071594 + }, + { + "bits_per_byte": 1.8758464177684127, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.3002376556396484, + "step": 2370, + "unclipped_grad_norm": 0.919369101524353 + }, + { + "bits_per_byte": 1.7745310941798302, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.230011224746704, + "step": 2380, + "unclipped_grad_norm": 0.7276220321655273 + }, + { + "bits_per_byte": 1.768817486552866, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.226050853729248, + "step": 2390, + "unclipped_grad_norm": 0.7200958132743835 + }, + { + "bits_per_byte": 1.8718060293516927, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.297437071800232, + "step": 2400, + "unclipped_grad_norm": 0.7491564750671387 + }, + { + "bits_per_byte": 1.8435730134207522, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2778674364089966, + "step": 2410, + "unclipped_grad_norm": 0.8167338371276855 + }, + { + "bits_per_byte": 1.9406400215677215, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.3451491594314575, + "step": 2420, + "unclipped_grad_norm": 0.7389746308326721 + }, + { + "bits_per_byte": 1.832556492737084, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.2702313661575317, + "step": 2430, + "unclipped_grad_norm": 0.7736530900001526 + }, + { + "bits_per_byte": 1.760901125133651, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.2205636501312256, + "step": 2440, + "unclipped_grad_norm": 0.7612105011940002 + }, + { + "bits_per_byte": 1.8632951239084448, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.2915377616882324, + "step": 2450, + "unclipped_grad_norm": 0.7563976049423218 + }, + { + "bits_per_byte": 1.8796537696932134, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.3028767108917236, + "step": 2460, + "unclipped_grad_norm": 0.7965136766433716 + }, + { + "bits_per_byte": 1.8059736503411725, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.2518055438995361, + "step": 2470, + "unclipped_grad_norm": 0.7263591885566711 + }, + { + "bits_per_byte": 1.8197415494733438, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.2613487243652344, + "step": 2480, + "unclipped_grad_norm": 0.7056093215942383 + }, + { + "bits_per_byte": 1.7492658108721786, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.212498664855957, + "step": 2490, + "unclipped_grad_norm": 0.7084574699401855 + }, + { + "bits_per_byte": 1.809534379144636, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2542736530303955, + "step": 2500, + "unclipped_grad_norm": 0.756045401096344 + }, + { + "bits_per_byte": 1.7856576837600506, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.2377235889434814, + "step": 2510, + "unclipped_grad_norm": 0.7166275382041931 + }, + { + "bits_per_byte": 1.7799100235682155, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.2337396144866943, + "step": 2520, + "unclipped_grad_norm": 0.7529211640357971 + }, + { + "bits_per_byte": 1.777542682379089, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.2320986986160278, + "step": 2530, + "unclipped_grad_norm": 0.7156926393508911 + }, + { + "bits_per_byte": 1.8142602904076952, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.2575494050979614, + "step": 2540, + "unclipped_grad_norm": 0.779152512550354 + }, + { + "bits_per_byte": 1.8414756849934804, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.2764136791229248, + "step": 2550, + "unclipped_grad_norm": 0.7701292634010315 + }, + { + "bits_per_byte": 1.794007269476789, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.2435110807418823, + "step": 2560, + "unclipped_grad_norm": 0.7429668307304382 + }, + { + "bits_per_byte": 1.7801779725382598, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2339253425598145, + "step": 2570, + "unclipped_grad_norm": 0.7433871030807495 + }, + { + "bits_per_byte": 1.8142921071981049, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.2575714588165283, + "step": 2580, + "unclipped_grad_norm": 0.8072794079780579 + }, + { + "bits_per_byte": 1.7761052513831777, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.2311023473739624, + "step": 2590, + "unclipped_grad_norm": 0.7563872337341309 + }, + { + "bits_per_byte": 1.8168768344579218, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.259363055229187, + "step": 2600, + "unclipped_grad_norm": 0.6932308673858643 + }, + { + "bits_per_byte": 1.7279317069979487, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.1977109909057617, + "step": 2610, + "unclipped_grad_norm": 0.7474331855773926 + }, + { + "bits_per_byte": 1.7718242592379005, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.2281349897384644, + "step": 2620, + "unclipped_grad_norm": 0.7322432994842529 + }, + { + "bits_per_byte": 1.79140224826417, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.2417054176330566, + "step": 2630, + "unclipped_grad_norm": 0.7361857891082764 + }, + { + "bits_per_byte": 1.8099839417939914, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.2545852661132812, + "step": 2640, + "unclipped_grad_norm": 0.7318217754364014 + }, + { + "bits_per_byte": 1.7552884713227432, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2166732549667358, + "step": 2650, + "unclipped_grad_norm": 0.6905237436294556 + }, + { + "bits_per_byte": 1.7514881706866277, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.2140390872955322, + "step": 2660, + "unclipped_grad_norm": 0.7271956205368042 + }, + { + "bits_per_byte": 1.7884833587137265, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.2396821975708008, + "step": 2670, + "unclipped_grad_norm": 0.6943170428276062 + }, + { + "bits_per_byte": 1.7662112614616912, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.2242443561553955, + "step": 2680, + "unclipped_grad_norm": 0.7118848562240601 + }, + { + "bits_per_byte": 1.845755817225503, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.279380440711975, + "step": 2690, + "unclipped_grad_norm": 0.7350090742111206 + }, + { + "bits_per_byte": 1.811656645056283, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.2557446956634521, + "step": 2700, + "unclipped_grad_norm": 0.7478894591331482 + }, + { + "bits_per_byte": 1.7139670597131904, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.1880314350128174, + "step": 2710, + "unclipped_grad_norm": 0.7357944846153259 + }, + { + "bits_per_byte": 1.8133080224698677, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.2568893432617188, + "step": 2720, + "unclipped_grad_norm": 0.7097733020782471 + }, + { + "bits_per_byte": 1.746889698567858, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.2108516693115234, + "step": 2730, + "unclipped_grad_norm": 0.6692879796028137 + }, + { + "bits_per_byte": 1.7687051818818527, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.2259730100631714, + "step": 2740, + "unclipped_grad_norm": 0.7475840449333191 + }, + { + "bits_per_byte": 1.7781879612851261, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.2325459718704224, + "step": 2750, + "unclipped_grad_norm": 0.7285085320472717 + }, + { + "bits_per_byte": 1.8651222675912114, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.29280424118042, + "step": 2760, + "unclipped_grad_norm": 0.7806845903396606 + }, + { + "bits_per_byte": 1.7645860254110375, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.2231178283691406, + "step": 2770, + "unclipped_grad_norm": 0.6795276999473572 + }, + { + "bits_per_byte": 1.7774769850064593, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.2320531606674194, + "step": 2780, + "unclipped_grad_norm": 0.7761561274528503 + }, + { + "bits_per_byte": 1.7782003440359884, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.23255455493927, + "step": 2790, + "unclipped_grad_norm": 0.7932373881340027 + }, + { + "bits_per_byte": 1.7165449076669732, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.189818263053894, + "step": 2800, + "unclipped_grad_norm": 0.7099766135215759 + }, + { + "bits_per_byte": 1.7811985175884797, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.2346327304840088, + "step": 2810, + "unclipped_grad_norm": 0.7101942896842957 + }, + { + "bits_per_byte": 1.7819846502855736, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2351776361465454, + "step": 2820, + "unclipped_grad_norm": 0.6840137243270874 + }, + { + "bits_per_byte": 1.715528318217723, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.1891136169433594, + "step": 2830, + "unclipped_grad_norm": 0.6831706166267395 + }, + { + "bits_per_byte": 1.7928673684668703, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.2427209615707397, + "step": 2840, + "unclipped_grad_norm": 0.7265968918800354 + }, + { + "bits_per_byte": 1.720763470109987, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.1927423477172852, + "step": 2850, + "unclipped_grad_norm": 0.674518346786499 + }, + { + "bits_per_byte": 1.7033779159169165, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.1806915998458862, + "step": 2860, + "unclipped_grad_norm": 0.7654244303703308 + }, + { + "bits_per_byte": 1.8090689940914013, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.253951072692871, + "step": 2870, + "unclipped_grad_norm": 0.711654782295227 + }, + { + "bits_per_byte": 1.7371888451632904, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.204127550125122, + "step": 2880, + "unclipped_grad_norm": 0.7070475220680237 + }, + { + "bits_per_byte": 1.794491228656317, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.2438465356826782, + "step": 2890, + "unclipped_grad_norm": 0.7585119009017944 + }, + { + "bits_per_byte": 1.69685856957066, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.1761727333068848, + "step": 2900, + "unclipped_grad_norm": 0.7276758551597595 + }, + { + "bits_per_byte": 1.7345860597251326, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.2023234367370605, + "step": 2910, + "unclipped_grad_norm": 0.7163064479827881 + }, + { + "bits_per_byte": 1.8229538414261592, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.2635753154754639, + "step": 2920, + "unclipped_grad_norm": 0.7330783009529114 + }, + { + "bits_per_byte": 1.771378824172166, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.2278262376785278, + "step": 2930, + "unclipped_grad_norm": 0.7703059315681458 + }, + { + "bits_per_byte": 1.793198779035083, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.242950677871704, + "step": 2940, + "unclipped_grad_norm": 0.7444400787353516 + }, + { + "bits_per_byte": 1.78159923716499, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.234910488128662, + "step": 2950, + "unclipped_grad_norm": 0.7506009936332703 + }, + { + "bits_per_byte": 1.75274639576034, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.2149112224578857, + "step": 2960, + "unclipped_grad_norm": 0.7135187387466431 + }, + { + "bits_per_byte": 1.8037172379618525, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.2502415180206299, + "step": 2970, + "unclipped_grad_norm": 0.7579538226127625 + }, + { + "bits_per_byte": 1.743725561757284, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.2086584568023682, + "step": 2980, + "unclipped_grad_norm": 0.7373576760292053 + }, + { + "bits_per_byte": 1.8504586828133465, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.2826402187347412, + "step": 2990, + "unclipped_grad_norm": 0.7562762498855591 + }, + { + "bits_per_byte": 1.7858227871048789, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.2378380298614502, + "step": 3000, + "unclipped_grad_norm": 0.6897611021995544 + }, + { + "bits_per_byte": 1.7614212006698602, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.2209241390228271, + "step": 3010, + "unclipped_grad_norm": 0.7554614543914795 + }, + { + "bits_per_byte": 1.725351623269705, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.195922613143921, + "step": 3020, + "unclipped_grad_norm": 0.7368059754371643 + }, + { + "bits_per_byte": 1.7570442422054016, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.2178902626037598, + "step": 3030, + "unclipped_grad_norm": 0.7938402891159058 + }, + { + "bits_per_byte": 1.7386587808802147, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.2051464319229126, + "step": 3040, + "unclipped_grad_norm": 0.7106590270996094 + }, + { + "bits_per_byte": 1.746893310203526, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.210854172706604, + "step": 3050, + "unclipped_grad_norm": 0.7691935300827026 + }, + { + "bits_per_byte": 1.7417082052626633, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.2072601318359375, + "step": 3060, + "unclipped_grad_norm": 0.8065770864486694 + }, + { + "bits_per_byte": 1.732928490936117, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.2011744976043701, + "step": 3070, + "unclipped_grad_norm": 0.7525816559791565 + }, + { + "bits_per_byte": 1.7368562307165216, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.2038969993591309, + "step": 3080, + "unclipped_grad_norm": 0.7258161902427673 + }, + { + "bits_per_byte": 1.7037829350596985, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.1809723377227783, + "step": 3090, + "unclipped_grad_norm": 0.7290193438529968 + }, + { + "bits_per_byte": 1.8014883428066704, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.2486965656280518, + "step": 3100, + "unclipped_grad_norm": 0.7526745796203613 + }, + { + "bits_per_byte": 1.7740380199198067, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.229669451713562, + "step": 3110, + "unclipped_grad_norm": 0.7630355954170227 + }, + { + "bits_per_byte": 1.651798083148699, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.1449391841888428, + "step": 3120, + "unclipped_grad_norm": 0.7035436034202576 + }, + { + "bits_per_byte": 1.760444855160912, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.2202473878860474, + "step": 3130, + "unclipped_grad_norm": 0.7679929733276367 + }, + { + "bits_per_byte": 1.7000084318212123, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.1783560514450073, + "step": 3140, + "unclipped_grad_norm": 0.7096188068389893 + }, + { + "bits_per_byte": 1.70021326315839, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.1784980297088623, + "step": 3150, + "unclipped_grad_norm": 0.7140545845031738 + }, + { + "bits_per_byte": 1.766801849884754, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.224653720855713, + "step": 3160, + "unclipped_grad_norm": 0.7189685702323914 + }, + { + "bits_per_byte": 1.794254408546079, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.2436823844909668, + "step": 3170, + "unclipped_grad_norm": 0.7592320442199707 + }, + { + "bits_per_byte": 1.6862627184510024, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.1688282489776611, + "step": 3180, + "unclipped_grad_norm": 0.7602659463882446 + }, + { + "bits_per_byte": 1.768633293133792, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.2259231805801392, + "step": 3190, + "unclipped_grad_norm": 0.7015324831008911 + }, + { + "bits_per_byte": 1.6582421010938775, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.149405837059021, + "step": 3200, + "unclipped_grad_norm": 0.672649085521698 + }, + { + "bits_per_byte": 1.7129356797559663, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.1873165369033813, + "step": 3210, + "unclipped_grad_norm": 0.7425183653831482 + }, + { + "bits_per_byte": 1.70026107433533, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.1785311698913574, + "step": 3220, + "unclipped_grad_norm": 0.6799039840698242 + }, + { + "bits_per_byte": 1.6848039616063841, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.1678171157836914, + "step": 3230, + "unclipped_grad_norm": 0.7309718728065491 + }, + { + "bits_per_byte": 1.7241479166963163, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.195088267326355, + "step": 3240, + "unclipped_grad_norm": 0.7472179532051086 + }, + { + "bits_per_byte": 1.7337502240419393, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.2017440795898438, + "step": 3250, + "unclipped_grad_norm": 0.726871907711029 + }, + { + "bits_per_byte": 1.8372738048676, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.2735011577606201, + "step": 3260, + "unclipped_grad_norm": 0.7614314556121826 + }, + { + "bits_per_byte": 1.6988687027939446, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.1775660514831543, + "step": 3270, + "unclipped_grad_norm": 0.7264232039451599 + }, + { + "bits_per_byte": 1.6351205815292307, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.1333792209625244, + "step": 3280, + "unclipped_grad_norm": 0.699049711227417 + }, + { + "bits_per_byte": 1.6743682263346797, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.1605836153030396, + "step": 3290, + "unclipped_grad_norm": 0.6957393288612366 + }, + { + "bits_per_byte": 1.7439493111860565, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.2088135480880737, + "step": 3300, + "unclipped_grad_norm": 0.6974862813949585 + }, + { + "bits_per_byte": 1.7411403185495142, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.2068665027618408, + "step": 3310, + "unclipped_grad_norm": 0.7476986050605774 + }, + { + "bits_per_byte": 1.7567350173991503, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.2176759243011475, + "step": 3320, + "unclipped_grad_norm": 0.7615799307823181 + }, + { + "bits_per_byte": 1.7088663982539014, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.1844959259033203, + "step": 3330, + "unclipped_grad_norm": 0.7350987792015076 + }, + { + "bits_per_byte": 1.7122492969963727, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.1868407726287842, + "step": 3340, + "unclipped_grad_norm": 0.7334772944450378 + }, + { + "bits_per_byte": 1.710089194901536, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.1853435039520264, + "step": 3350, + "unclipped_grad_norm": 0.7237264513969421 + }, + { + "bits_per_byte": 1.7073164906043257, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.1834216117858887, + "step": 3360, + "unclipped_grad_norm": 0.7319026589393616 + }, + { + "bits_per_byte": 1.642050106497795, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.1381824016571045, + "step": 3370, + "unclipped_grad_norm": 0.7280613780021667 + }, + { + "bits_per_byte": 1.757205561931911, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.2180020809173584, + "step": 3380, + "unclipped_grad_norm": 0.8463512063026428 + }, + { + "bits_per_byte": 1.7246451185399605, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.1954329013824463, + "step": 3390, + "unclipped_grad_norm": 0.7450569272041321 + }, + { + "bits_per_byte": 1.7166613399216073, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.18989896774292, + "step": 3400, + "unclipped_grad_norm": 0.7346100211143494 + }, + { + "bits_per_byte": 1.7498677501501985, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.2129158973693848, + "step": 3410, + "unclipped_grad_norm": 0.7586395740509033 + }, + { + "bits_per_byte": 1.717769940089069, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.1906673908233643, + "step": 3420, + "unclipped_grad_norm": 0.7326739430427551 + }, + { + "bits_per_byte": 1.6693836531647228, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1571285724639893, + "step": 3430, + "unclipped_grad_norm": 0.7360373139381409 + }, + { + "bits_per_byte": 1.6658869019073803, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.1547048091888428, + "step": 3440, + "unclipped_grad_norm": 0.6685504913330078 + }, + { + "bits_per_byte": 1.6662052417941273, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.1549254655838013, + "step": 3450, + "unclipped_grad_norm": 0.7631767392158508 + }, + { + "bits_per_byte": 1.6663135908641709, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.1550005674362183, + "step": 3460, + "unclipped_grad_norm": 0.7326028943061829 + }, + { + "bits_per_byte": 1.7228891756746514, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.1942157745361328, + "step": 3470, + "unclipped_grad_norm": 0.7883577942848206 + }, + { + "bits_per_byte": 1.7792774713783421, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.2333011627197266, + "step": 3480, + "unclipped_grad_norm": 0.7605118751525879 + }, + { + "bits_per_byte": 1.7247985270645303, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.1955392360687256, + "step": 3490, + "unclipped_grad_norm": 0.739758312702179 + }, + { + "bits_per_byte": 1.6670271468826008, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1554951667785645, + "step": 3500, + "unclipped_grad_norm": 0.7004515528678894 + }, + { + "bits_per_byte": 1.6755079553619474, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.1613736152648926, + "step": 3510, + "unclipped_grad_norm": 0.6788433790206909 + }, + { + "bits_per_byte": 1.7205484917964085, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.1925933361053467, + "step": 3520, + "unclipped_grad_norm": 0.7589633464813232 + }, + { + "bits_per_byte": 1.6869193482119966, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.169283390045166, + "step": 3530, + "unclipped_grad_norm": 0.7391977310180664 + }, + { + "bits_per_byte": 1.6831914522718943, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.1666994094848633, + "step": 3540, + "unclipped_grad_norm": 0.7568037509918213 + }, + { + "bits_per_byte": 1.7356833090376373, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.2030839920043945, + "step": 3550, + "unclipped_grad_norm": 0.7387335300445557 + }, + { + "bits_per_byte": 1.6630999230501484, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.1527730226516724, + "step": 3560, + "unclipped_grad_norm": 0.7604349255561829 + }, + { + "bits_per_byte": 1.7031775561286613, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.1805527210235596, + "step": 3570, + "unclipped_grad_norm": 0.723289966583252 + }, + { + "bits_per_byte": 1.7307549741945127, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.1996679306030273, + "step": 3580, + "unclipped_grad_norm": 0.7507236003875732 + }, + { + "bits_per_byte": 1.7099024217426988, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.1852140426635742, + "step": 3590, + "unclipped_grad_norm": 0.7191433310508728 + }, + { + "bits_per_byte": 1.7598348326983015, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.2198245525360107, + "step": 3600, + "unclipped_grad_norm": 0.7742589712142944 + }, + { + "bits_per_byte": 1.7170397017535055, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.1901612281799316, + "step": 3610, + "unclipped_grad_norm": 0.7492689490318298 + }, + { + "bits_per_byte": 1.6647831172541427, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.1539397239685059, + "step": 3620, + "unclipped_grad_norm": 0.6780882477760315 + }, + { + "bits_per_byte": 1.6812962034593861, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.1653857231140137, + "step": 3630, + "unclipped_grad_norm": 0.8111791610717773 + }, + { + "bits_per_byte": 1.6162136688066278, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.1202739477157593, + "step": 3640, + "unclipped_grad_norm": 0.6700810194015503 + }, + { + "bits_per_byte": 1.6940698708869195, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.1742397546768188, + "step": 3650, + "unclipped_grad_norm": 0.7609620690345764 + }, + { + "bits_per_byte": 1.691939349808031, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.1727629899978638, + "step": 3660, + "unclipped_grad_norm": 0.7344038486480713 + }, + { + "bits_per_byte": 1.7084398812797617, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.1842002868652344, + "step": 3670, + "unclipped_grad_norm": 0.777010977268219 + }, + { + "bits_per_byte": 1.6420332521980103, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.1381707191467285, + "step": 3680, + "unclipped_grad_norm": 0.7007532119750977 + }, + { + "bits_per_byte": 1.7237969001059052, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.1948449611663818, + "step": 3690, + "unclipped_grad_norm": 0.7735820412635803 + }, + { + "bits_per_byte": 1.6208267594507209, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.1234714984893799, + "step": 3700, + "unclipped_grad_norm": 0.7354447841644287 + }, + { + "bits_per_byte": 1.683411246099697, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.1668517589569092, + "step": 3710, + "unclipped_grad_norm": 0.7224906086921692 + }, + { + "bits_per_byte": 1.7113298777448602, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.1862034797668457, + "step": 3720, + "unclipped_grad_norm": 0.7525572776794434 + }, + { + "bits_per_byte": 1.6810134639813676, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.1651897430419922, + "step": 3730, + "unclipped_grad_norm": 0.7857299447059631 + }, + { + "bits_per_byte": 1.6696806672027629, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.1573344469070435, + "step": 3740, + "unclipped_grad_norm": 0.7745850086212158 + }, + { + "bits_per_byte": 1.6853979896824642, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1682288646697998, + "step": 3750, + "unclipped_grad_norm": 0.7253222465515137 + }, + { + "bits_per_byte": 1.7132295981062908, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.1875202655792236, + "step": 3760, + "unclipped_grad_norm": 0.7334293723106384 + }, + { + "bits_per_byte": 1.7158002227887372, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.1893020868301392, + "step": 3770, + "unclipped_grad_norm": 0.7110471725463867 + }, + { + "bits_per_byte": 1.754358561129528, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.2160286903381348, + "step": 3780, + "unclipped_grad_norm": 0.7414421439170837 + }, + { + "bits_per_byte": 1.6659691096144926, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.154761791229248, + "step": 3790, + "unclipped_grad_norm": 0.6979415416717529 + }, + { + "bits_per_byte": 1.696525095210637, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.1759415864944458, + "step": 3800, + "unclipped_grad_norm": 0.7420910596847534 + }, + { + "bits_per_byte": 1.6300749545182176, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.1298818588256836, + "step": 3810, + "unclipped_grad_norm": 0.7547963261604309 + }, + { + "bits_per_byte": 1.626499091241478, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1274032592773438, + "step": 3820, + "unclipped_grad_norm": 0.7493447661399841 + }, + { + "bits_per_byte": 1.6966757520127929, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.1760460138320923, + "step": 3830, + "unclipped_grad_norm": 0.7342293858528137 + }, + { + "bits_per_byte": 1.6390436777780621, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.1360985040664673, + "step": 3840, + "unclipped_grad_norm": 0.6855485439300537 + }, + { + "bits_per_byte": 1.6170943919616962, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1208844184875488, + "step": 3850, + "unclipped_grad_norm": 0.6862776875495911 + }, + { + "bits_per_byte": 1.6102295324698563, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.1161260604858398, + "step": 3860, + "unclipped_grad_norm": 0.762079656124115 + }, + { + "bits_per_byte": 1.560827000061562, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.0818828344345093, + "step": 3870, + "unclipped_grad_norm": 0.7250840663909912 + }, + { + "bits_per_byte": 1.660220761492054, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1507773399353027, + "step": 3880, + "unclipped_grad_norm": 0.7003803253173828 + }, + { + "bits_per_byte": 1.6392378461908863, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.1362330913543701, + "step": 3890, + "unclipped_grad_norm": 0.7212581634521484 + }, + { + "bits_per_byte": 1.6391527147787093, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.1361740827560425, + "step": 3900, + "unclipped_grad_norm": 0.7184483408927917 + }, + { + "bits_per_byte": 1.656392427683848, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1481237411499023, + "step": 3910, + "unclipped_grad_norm": 0.7179060578346252 + }, + { + "bits_per_byte": 1.6858616549091903, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.1685502529144287, + "step": 3920, + "unclipped_grad_norm": 0.7784472107887268 + }, + { + "bits_per_byte": 1.678337929881895, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.1633352041244507, + "step": 3930, + "unclipped_grad_norm": 0.7443260550498962 + }, + { + "bits_per_byte": 1.6086877080048714, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.115057349205017, + "step": 3940, + "unclipped_grad_norm": 0.7205680012702942 + }, + { + "bits_per_byte": 1.730125173727053, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1992313861846924, + "step": 3950, + "unclipped_grad_norm": 0.7901617288589478 + }, + { + "bits_per_byte": 1.6953611166295977, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.1751347780227661, + "step": 3960, + "unclipped_grad_norm": 0.7302133440971375 + }, + { + "bits_per_byte": 1.651679071154302, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1448566913604736, + "step": 3970, + "unclipped_grad_norm": 0.7738355398178101 + }, + { + "bits_per_byte": 1.633614529455625, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.1323353052139282, + "step": 3980, + "unclipped_grad_norm": 0.7410290837287903 + }, + { + "bits_per_byte": 1.6806216875027022, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1649181842803955, + "step": 3990, + "unclipped_grad_norm": 0.7142926454544067 + }, + { + "bits_per_byte": 1.5794174647065773, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.094768762588501, + "step": 4000, + "unclipped_grad_norm": 0.7205081582069397 + }, + { + "bits_per_byte": 1.6672507243287222, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.1556501388549805, + "step": 4010, + "unclipped_grad_norm": 0.8078667521476746 + }, + { + "bits_per_byte": 1.68056940477684, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.164881944656372, + "step": 4020, + "unclipped_grad_norm": 0.8188139796257019 + }, + { + "bits_per_byte": 1.6298911050644453, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.1297544240951538, + "step": 4030, + "unclipped_grad_norm": 0.7587766051292419 + }, + { + "bits_per_byte": 1.6220925517610711, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1243488788604736, + "step": 4040, + "unclipped_grad_norm": 0.7564710974693298 + }, + { + "bits_per_byte": 1.6506616217917975, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.1441514492034912, + "step": 4050, + "unclipped_grad_norm": 0.7635183334350586 + }, + { + "bits_per_byte": 1.6219026829145187, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1242172718048096, + "step": 4060, + "unclipped_grad_norm": 0.7427014112472534 + }, + { + "bits_per_byte": 1.7537997894968747, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.2156413793563843, + "step": 4070, + "unclipped_grad_norm": 0.7789963483810425 + }, + { + "bits_per_byte": 1.5922702441535077, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.1036776304244995, + "step": 4080, + "unclipped_grad_norm": 0.7233743071556091 + }, + { + "bits_per_byte": 1.6121278769700802, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.1174418926239014, + "step": 4090, + "unclipped_grad_norm": 0.7508261203765869 + }, + { + "bits_per_byte": 1.730570264827486, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.1995398998260498, + "step": 4100, + "unclipped_grad_norm": 0.7926138639450073 + }, + { + "bits_per_byte": 1.6268710897152943, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.127661108970642, + "step": 4110, + "unclipped_grad_norm": 0.7293549180030823 + }, + { + "bits_per_byte": 1.6331204232996963, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.1319928169250488, + "step": 4120, + "unclipped_grad_norm": 0.757316529750824 + }, + { + "bits_per_byte": 1.6949371793952208, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.1748409271240234, + "step": 4130, + "unclipped_grad_norm": 0.7389926314353943 + }, + { + "bits_per_byte": 1.704332591611856, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.1813533306121826, + "step": 4140, + "unclipped_grad_norm": 0.8232205510139465 + }, + { + "bits_per_byte": 1.6256795939101167, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1268352270126343, + "step": 4150, + "unclipped_grad_norm": 0.7450867891311646 + }, + { + "bits_per_byte": 1.6333955955410768, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.13218355178833, + "step": 4160, + "unclipped_grad_norm": 0.7731286287307739 + }, + { + "bits_per_byte": 1.6517733176469747, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.1449220180511475, + "step": 4170, + "unclipped_grad_norm": 0.7551148533821106 + }, + { + "bits_per_byte": 1.6326828994359013, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.1316895484924316, + "step": 4180, + "unclipped_grad_norm": 0.7393075823783875 + }, + { + "bits_per_byte": 1.667416515604154, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.1557650566101074, + "step": 4190, + "unclipped_grad_norm": 0.7493076920509338 + }, + { + "bits_per_byte": 1.6599129125470098, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1505639553070068, + "step": 4200, + "unclipped_grad_norm": 0.8006278872489929 + }, + { + "bits_per_byte": 1.677859130181893, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.1630033254623413, + "step": 4210, + "unclipped_grad_norm": 0.7366514801979065 + }, + { + "bits_per_byte": 1.6454061759467318, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.1405086517333984, + "step": 4220, + "unclipped_grad_norm": 0.728699266910553 + }, + { + "bits_per_byte": 1.6522063699618472, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.1452221870422363, + "step": 4230, + "unclipped_grad_norm": 0.7545228004455566 + }, + { + "bits_per_byte": 1.683407462481378, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.166849136352539, + "step": 4240, + "unclipped_grad_norm": 0.7422810196876526 + }, + { + "bits_per_byte": 1.6078241831148892, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1144587993621826, + "step": 4250, + "unclipped_grad_norm": 0.7206326723098755 + }, + { + "bits_per_byte": 1.6576331105271724, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1489837169647217, + "step": 4260, + "unclipped_grad_norm": 0.7456027865409851 + }, + { + "bits_per_byte": 1.5690563699053475, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.0875869989395142, + "step": 4270, + "unclipped_grad_norm": 0.7475060224533081 + }, + { + "bits_per_byte": 1.6429423524904712, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.138800859451294, + "step": 4280, + "unclipped_grad_norm": 0.7600163817405701 + }, + { + "bits_per_byte": 1.610052046374166, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.1160030364990234, + "step": 4290, + "unclipped_grad_norm": 0.7834687829017639 + }, + { + "bits_per_byte": 1.5916227294730092, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.1032288074493408, + "step": 4300, + "unclipped_grad_norm": 0.7749201059341431 + }, + { + "bits_per_byte": 1.5730694130805802, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.090368628501892, + "step": 4310, + "unclipped_grad_norm": 0.7362476587295532 + }, + { + "bits_per_byte": 1.707772416611763, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.1837376356124878, + "step": 4320, + "unclipped_grad_norm": 0.8086820840835571 + }, + { + "bits_per_byte": 1.644883348688109, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.140146255493164, + "step": 4330, + "unclipped_grad_norm": 0.7674968242645264 + }, + { + "bits_per_byte": 1.5845465033032589, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.098323941230774, + "step": 4340, + "unclipped_grad_norm": 0.7907853722572327 + }, + { + "bits_per_byte": 1.548560681454074, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.073380470275879, + "step": 4350, + "unclipped_grad_norm": 0.7505454421043396 + }, + { + "bits_per_byte": 1.645202892453412, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.1403677463531494, + "step": 4360, + "unclipped_grad_norm": 0.7696559429168701 + }, + { + "bits_per_byte": 1.615986479724838, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.1201164722442627, + "step": 4370, + "unclipped_grad_norm": 0.7597289681434631 + }, + { + "bits_per_byte": 1.561606769400574, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.0824233293533325, + "step": 4380, + "unclipped_grad_norm": 0.7038122415542603 + }, + { + "bits_per_byte": 1.6928398509679488, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1733871698379517, + "step": 4390, + "unclipped_grad_norm": 0.7434208393096924 + }, + { + "bits_per_byte": 1.6496006608186249, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.1434160470962524, + "step": 4400, + "unclipped_grad_norm": 0.7787129878997803 + }, + { + "bits_per_byte": 1.6500163428857604, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.1437041759490967, + "step": 4410, + "unclipped_grad_norm": 0.7854211926460266 + }, + { + "bits_per_byte": 1.6810537079216696, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.165217638015747, + "step": 4420, + "unclipped_grad_norm": 0.7670703530311584 + }, + { + "bits_per_byte": 1.613851487097027, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.1186366081237793, + "step": 4430, + "unclipped_grad_norm": 0.7906524538993835 + }, + { + "bits_per_byte": 1.6346070413337543, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.1330232620239258, + "step": 4440, + "unclipped_grad_norm": 0.7283197045326233 + }, + { + "bits_per_byte": 1.6547135330461253, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.1469600200653076, + "step": 4450, + "unclipped_grad_norm": 0.7324498891830444 + }, + { + "bits_per_byte": 1.634302632041727, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.132812261581421, + "step": 4460, + "unclipped_grad_norm": 0.8062254190444946 + }, + { + "bits_per_byte": 1.6172010212052312, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1209583282470703, + "step": 4470, + "unclipped_grad_norm": 0.7936866879463196 + }, + { + "bits_per_byte": 1.6558871426556132, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.1477735042572021, + "step": 4480, + "unclipped_grad_norm": 0.7719366550445557 + }, + { + "bits_per_byte": 1.702131213680812, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.1798274517059326, + "step": 4490, + "unclipped_grad_norm": 0.812337338924408 + }, + { + "bits_per_byte": 1.5975956869374746, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.1073689460754395, + "step": 4500, + "unclipped_grad_norm": 0.7559470534324646 + }, + { + "bits_per_byte": 1.5918301405499498, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.103372573852539, + "step": 4510, + "unclipped_grad_norm": 0.7325426936149597 + }, + { + "bits_per_byte": 1.6708895332556777, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.158172369003296, + "step": 4520, + "unclipped_grad_norm": 0.8050368428230286 + }, + { + "bits_per_byte": 1.6257181180239102, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.1268619298934937, + "step": 4530, + "unclipped_grad_norm": 0.8098328709602356 + }, + { + "bits_per_byte": 1.6361610765669508, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.1341004371643066, + "step": 4540, + "unclipped_grad_norm": 0.7618226408958435 + }, + { + "bits_per_byte": 1.653153306437498, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.145878553390503, + "step": 4550, + "unclipped_grad_norm": 0.7616913914680481 + }, + { + "bits_per_byte": 1.6499458299989065, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.1436553001403809, + "step": 4560, + "unclipped_grad_norm": 0.8365893959999084 + }, + { + "bits_per_byte": 1.634003382229226, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.1326048374176025, + "step": 4570, + "unclipped_grad_norm": 0.779934287071228 + }, + { + "bits_per_byte": 1.6037027908696129, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.1116020679473877, + "step": 4580, + "unclipped_grad_norm": 0.7435479760169983 + }, + { + "bits_per_byte": 1.587256433932904, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.1002023220062256, + "step": 4590, + "unclipped_grad_norm": 0.7966163158416748 + }, + { + "bits_per_byte": 1.5935126468233405, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.1045387983322144, + "step": 4600, + "unclipped_grad_norm": 0.7649639248847961 + }, + { + "bits_per_byte": 1.5980572883723902, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.1076889038085938, + "step": 4610, + "unclipped_grad_norm": 0.7703146934509277 + }, + { + "bits_per_byte": 1.6559168956542123, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.1477941274642944, + "step": 4620, + "unclipped_grad_norm": 0.756706178188324 + }, + { + "bits_per_byte": 1.6444052369187103, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.139814853668213, + "step": 4630, + "unclipped_grad_norm": 0.8557617664337158 + }, + { + "bits_per_byte": 1.6524189405183136, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.145369529724121, + "step": 4640, + "unclipped_grad_norm": 0.8033635020256042 + }, + { + "bits_per_byte": 1.6805250332529174, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.164851188659668, + "step": 4650, + "unclipped_grad_norm": 0.7698349952697754 + }, + { + "bits_per_byte": 1.6069549827974285, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.113856315612793, + "step": 4660, + "unclipped_grad_norm": 0.759970486164093 + }, + { + "bits_per_byte": 1.6100126623471185, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.1159757375717163, + "step": 4670, + "unclipped_grad_norm": 0.7600313425064087 + }, + { + "bits_per_byte": 1.597988839277347, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.1076414585113525, + "step": 4680, + "unclipped_grad_norm": 0.8227591514587402 + }, + { + "bits_per_byte": 1.5931459798117011, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.104284644126892, + "step": 4690, + "unclipped_grad_norm": 0.7564940452575684 + }, + { + "bits_per_byte": 1.6313747993934387, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.1307828426361084, + "step": 4700, + "unclipped_grad_norm": 0.7582079172134399 + }, + { + "bits_per_byte": 1.6163297570959603, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.120354413986206, + "step": 4710, + "unclipped_grad_norm": 0.8014265298843384 + }, + { + "bits_per_byte": 1.6186002720526511, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.1219282150268555, + "step": 4720, + "unclipped_grad_norm": 0.7675052285194397 + }, + { + "bits_per_byte": 1.654904605771234, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.1470924615859985, + "step": 4730, + "unclipped_grad_norm": 0.8101351261138916 + }, + { + "bits_per_byte": 1.6189717545785147, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.1221857070922852, + "step": 4740, + "unclipped_grad_norm": 0.8598103523254395 + }, + { + "bits_per_byte": 1.580895827573394, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.0957934856414795, + "step": 4750, + "unclipped_grad_norm": 0.8005276322364807 + }, + { + "bits_per_byte": 1.5436019056817465, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.0699433088302612, + "step": 4760, + "unclipped_grad_norm": 0.7416074275970459 + }, + { + "bits_per_byte": 1.5626405571149105, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.0831398963928223, + "step": 4770, + "unclipped_grad_norm": 0.7465143799781799 + }, + { + "bits_per_byte": 1.5869711147151229, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.1000045537948608, + "step": 4780, + "unclipped_grad_norm": 0.783750593662262 + }, + { + "bits_per_byte": 1.5971492199758346, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.1070594787597656, + "step": 4790, + "unclipped_grad_norm": 0.7763316631317139 + }, + { + "bits_per_byte": 1.6268268901740226, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.1276304721832275, + "step": 4800, + "unclipped_grad_norm": 0.8211584091186523 + }, + { + "bits_per_byte": 1.5932545008643955, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.1043598651885986, + "step": 4810, + "unclipped_grad_norm": 0.8034570217132568 + }, + { + "bits_per_byte": 1.5974343672109652, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.1072571277618408, + "step": 4820, + "unclipped_grad_norm": 0.7744666934013367 + }, + { + "bits_per_byte": 1.5939462150861656, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.1048393249511719, + "step": 4830, + "unclipped_grad_norm": 0.7603051662445068 + }, + { + "bits_per_byte": 1.5381509155626498, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.0661649703979492, + "step": 4840, + "unclipped_grad_norm": 0.7635394930839539 + }, + { + "bits_per_byte": 1.5984607596713145, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.1079685688018799, + "step": 4850, + "unclipped_grad_norm": 0.7812595367431641 + }, + { + "bits_per_byte": 1.5820200781620841, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.096572756767273, + "step": 4860, + "unclipped_grad_norm": 0.7616010308265686 + }, + { + "bits_per_byte": 1.6032098885922401, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.1112604141235352, + "step": 4870, + "unclipped_grad_norm": 0.7853467464447021 + }, + { + "bits_per_byte": 1.5611388046075763, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.0820989608764648, + "step": 4880, + "unclipped_grad_norm": 0.7643629312515259 + }, + { + "bits_per_byte": 1.5278708245899764, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.0590393543243408, + "step": 4890, + "unclipped_grad_norm": 0.8007761240005493 + }, + { + "bits_per_byte": 1.5153095557362586, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.0503325462341309, + "step": 4900, + "unclipped_grad_norm": 0.7629372477531433 + }, + { + "bits_per_byte": 1.5863199883989563, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.0995532274246216, + "step": 4910, + "unclipped_grad_norm": 0.7892634272575378 + }, + { + "bits_per_byte": 1.585380103211991, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.0989017486572266, + "step": 4920, + "unclipped_grad_norm": 0.7887189984321594 + }, + { + "bits_per_byte": 1.5684195181492024, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.0871455669403076, + "step": 4930, + "unclipped_grad_norm": 0.7403048276901245 + }, + { + "bits_per_byte": 1.6554664730916027, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.147481918334961, + "step": 4940, + "unclipped_grad_norm": 0.8125011920928955 + }, + { + "bits_per_byte": 1.51878016563067, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.0527381896972656, + "step": 4950, + "unclipped_grad_norm": 0.7984678745269775 + }, + { + "bits_per_byte": 1.5966829750093456, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.1067363023757935, + "step": 4960, + "unclipped_grad_norm": 0.8314551115036011 + }, + { + "bits_per_byte": 1.679989135312829, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.1644797325134277, + "step": 4970, + "unclipped_grad_norm": 0.8750884532928467 + }, + { + "bits_per_byte": 1.554043832328882, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.077181100845337, + "step": 4980, + "unclipped_grad_norm": 0.834908127784729 + }, + { + "bits_per_byte": 1.6029044474043077, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.111048698425293, + "step": 4990, + "unclipped_grad_norm": 0.7933679223060608 + }, + { + "bits_per_byte": 1.6180771008287265, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.121565580368042, + "step": 5000, + "unclipped_grad_norm": 0.8222954273223877 + }, + { + "bits_per_byte": 1.5906635822291473, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.1025639772415161, + "step": 5010, + "unclipped_grad_norm": 0.8256110548973083 + }, + { + "bits_per_byte": 1.5900814209559766, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.1021604537963867, + "step": 5020, + "unclipped_grad_norm": 0.7704684734344482 + }, + { + "bits_per_byte": 1.575425059449448, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.0920014381408691, + "step": 5030, + "unclipped_grad_norm": 0.8217195272445679 + }, + { + "bits_per_byte": 1.5616463254102724, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.0824507474899292, + "step": 5040, + "unclipped_grad_norm": 0.7686264514923096 + }, + { + "bits_per_byte": 1.6032394696081884, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.111280918121338, + "step": 5050, + "unclipped_grad_norm": 0.8206285238265991 + }, + { + "bits_per_byte": 1.5503986600438449, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.074654459953308, + "step": 5060, + "unclipped_grad_norm": 0.7991617918014526 + }, + { + "bits_per_byte": 1.624417069270133, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.125960111618042, + "step": 5070, + "unclipped_grad_norm": 0.83602374792099 + }, + { + "bits_per_byte": 1.6334912178949565, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.1322498321533203, + "step": 5080, + "unclipped_grad_norm": 0.9215573072433472 + }, + { + "bits_per_byte": 1.6244679761347884, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.125995397567749, + "step": 5090, + "unclipped_grad_norm": 0.880305826663971 + }, + { + "bits_per_byte": 1.5500395602688433, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.074405550956726, + "step": 5100, + "unclipped_grad_norm": 0.8300489783287048 + }, + { + "bits_per_byte": 1.567405852405017, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.0864429473876953, + "step": 5110, + "unclipped_grad_norm": 0.7976618409156799 + }, + { + "bits_per_byte": 1.5667330562748418, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0859766006469727, + "step": 5120, + "unclipped_grad_norm": 0.8788276314735413 + }, + { + "bits_per_byte": 1.6067348450043242, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.113703727722168, + "step": 5130, + "unclipped_grad_norm": 0.8028310537338257 + }, + { + "bits_per_byte": 1.5504239414935217, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.074671983718872, + "step": 5140, + "unclipped_grad_norm": 0.8123137354850769 + }, + { + "bits_per_byte": 1.545032973319576, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.0709352493286133, + "step": 5150, + "unclipped_grad_norm": 0.849959135055542 + }, + { + "bits_per_byte": 1.5732883469951287, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.0905203819274902, + "step": 5160, + "unclipped_grad_norm": 0.8103107810020447 + }, + { + "bits_per_byte": 1.5891399879251535, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.1015079021453857, + "step": 5170, + "unclipped_grad_norm": 0.806939959526062 + }, + { + "bits_per_byte": 1.6119741244802088, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.117335319519043, + "step": 5180, + "unclipped_grad_norm": 0.8405307531356812 + }, + { + "bits_per_byte": 1.5732463832283181, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.0904912948608398, + "step": 5190, + "unclipped_grad_norm": 0.791869580745697 + }, + { + "bits_per_byte": 1.5171222528763526, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.051589012145996, + "step": 5200, + "unclipped_grad_norm": 0.8294087648391724 + }, + { + "bits_per_byte": 1.6001295073326363, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.1091252565383911, + "step": 5210, + "unclipped_grad_norm": 0.8102724552154541 + }, + { + "bits_per_byte": 1.5914727606014567, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.1031248569488525, + "step": 5220, + "unclipped_grad_norm": 0.828036904335022 + }, + { + "bits_per_byte": 1.6165311487801206, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.12049400806427, + "step": 5230, + "unclipped_grad_norm": 0.8456141352653503 + }, + { + "bits_per_byte": 1.565167154238736, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0848912000656128, + "step": 5240, + "unclipped_grad_norm": 0.8292003870010376 + }, + { + "bits_per_byte": 1.5262173833845813, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0578932762145996, + "step": 5250, + "unclipped_grad_norm": 0.7859673500061035 + }, + { + "bits_per_byte": 1.5082104558739435, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.0454118251800537, + "step": 5260, + "unclipped_grad_norm": 0.7717975378036499 + }, + { + "bits_per_byte": 1.5983742523979305, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.1079086065292358, + "step": 5270, + "unclipped_grad_norm": 0.8473073244094849 + }, + { + "bits_per_byte": 1.6047603121897682, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.1123350858688354, + "step": 5280, + "unclipped_grad_norm": 0.8248690962791443 + }, + { + "bits_per_byte": 1.575782611380592, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.0922492742538452, + "step": 5290, + "unclipped_grad_norm": 0.8321958184242249 + }, + { + "bits_per_byte": 1.5183407499577155, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.0524336099624634, + "step": 5300, + "unclipped_grad_norm": 0.8376712203025818 + }, + { + "bits_per_byte": 1.593326733577758, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.10440993309021, + "step": 5310, + "unclipped_grad_norm": 0.8155526518821716 + }, + { + "bits_per_byte": 1.5617190740715874, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0825011730194092, + "step": 5320, + "unclipped_grad_norm": 0.8000097274780273 + }, + { + "bits_per_byte": 1.6330275526682303, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.1319284439086914, + "step": 5330, + "unclipped_grad_norm": 0.8233504295349121 + }, + { + "bits_per_byte": 1.5384372666763364, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.0663634538650513, + "step": 5340, + "unclipped_grad_norm": 0.8107735514640808 + }, + { + "bits_per_byte": 1.5931053919060973, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.104256510734558, + "step": 5350, + "unclipped_grad_norm": 0.8740422129631042 + }, + { + "bits_per_byte": 1.4994637622163627, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.03934907913208, + "step": 5360, + "unclipped_grad_norm": 0.7547010183334351 + }, + { + "bits_per_byte": 1.536192721099926, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.064807653427124, + "step": 5370, + "unclipped_grad_norm": 0.8179121017456055 + }, + { + "bits_per_byte": 1.6001040539003086, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.1091076135635376, + "step": 5380, + "unclipped_grad_norm": 0.8696821331977844 + }, + { + "bits_per_byte": 1.5054580455295352, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.043503999710083, + "step": 5390, + "unclipped_grad_norm": 0.8188030123710632 + }, + { + "bits_per_byte": 1.5759031712188467, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.0923328399658203, + "step": 5400, + "unclipped_grad_norm": 0.8273021578788757 + }, + { + "bits_per_byte": 1.531432929254647, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.0615084171295166, + "step": 5410, + "unclipped_grad_norm": 0.7996953725814819 + }, + { + "bits_per_byte": 1.4907026220161095, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.0332763195037842, + "step": 5420, + "unclipped_grad_norm": 0.8094907402992249 + }, + { + "bits_per_byte": 1.5658236120170792, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.0853462219238281, + "step": 5430, + "unclipped_grad_norm": 0.8032275438308716 + }, + { + "bits_per_byte": 1.5244272159717502, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.0566524267196655, + "step": 5440, + "unclipped_grad_norm": 0.8245100378990173 + }, + { + "bits_per_byte": 1.5534439568426726, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.0767652988433838, + "step": 5450, + "unclipped_grad_norm": 0.8273851871490479 + }, + { + "bits_per_byte": 1.5365442536382894, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.0650513172149658, + "step": 5460, + "unclipped_grad_norm": 0.7948191165924072 + }, + { + "bits_per_byte": 1.5624176675993922, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0829854011535645, + "step": 5470, + "unclipped_grad_norm": 0.8186808228492737 + }, + { + "bits_per_byte": 1.5249908031186277, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.0570430755615234, + "step": 5480, + "unclipped_grad_norm": 0.8306441307067871 + }, + { + "bits_per_byte": 1.5993870582288614, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.1086106300354004, + "step": 5490, + "unclipped_grad_norm": 0.9400379061698914 + }, + { + "bits_per_byte": 1.5728931308634457, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.0902464389801025, + "step": 5500, + "unclipped_grad_norm": 0.8489547967910767 + }, + { + "bits_per_byte": 1.524312331560974, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.0565727949142456, + "step": 5510, + "unclipped_grad_norm": 0.8289045691490173 + }, + { + "bits_per_byte": 1.491928170386158, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.034125804901123, + "step": 5520, + "unclipped_grad_norm": 0.7798097133636475 + }, + { + "bits_per_byte": 1.4998407481870542, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.0396103858947754, + "step": 5530, + "unclipped_grad_norm": 0.8041956424713135 + }, + { + "bits_per_byte": 1.6251579705300498, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.1264736652374268, + "step": 5540, + "unclipped_grad_norm": 0.8368581533432007 + }, + { + "bits_per_byte": 1.5591094093273952, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.0806922912597656, + "step": 5550, + "unclipped_grad_norm": 0.8318032026290894 + }, + { + "bits_per_byte": 1.608697167050669, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.1150639057159424, + "step": 5560, + "unclipped_grad_norm": 0.9183136820793152 + }, + { + "bits_per_byte": 1.5361999443712622, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.0648126602172852, + "step": 5570, + "unclipped_grad_norm": 0.7968618273735046 + }, + { + "bits_per_byte": 1.576336567499021, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.0926332473754883, + "step": 5580, + "unclipped_grad_norm": 0.8440650701522827 + }, + { + "bits_per_byte": 1.5774926348781209, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.0934345722198486, + "step": 5590, + "unclipped_grad_norm": 0.8763058185577393 + }, + { + "bits_per_byte": 1.5551637833513006, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.0779573917388916, + "step": 5600, + "unclipped_grad_norm": 0.8399657607078552 + }, + { + "bits_per_byte": 1.4888436615429335, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.0319877862930298, + "step": 5610, + "unclipped_grad_norm": 0.8241370320320129 + }, + { + "bits_per_byte": 1.5594860513327846, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.0809533596038818, + "step": 5620, + "unclipped_grad_norm": 0.8907544016838074 + }, + { + "bits_per_byte": 1.4908742607016707, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0333952903747559, + "step": 5630, + "unclipped_grad_norm": 0.8095570802688599 + }, + { + "bits_per_byte": 1.4701137189680684, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.0190051794052124, + "step": 5640, + "unclipped_grad_norm": 0.7788687944412231 + }, + { + "bits_per_byte": 1.546012070550938, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.0716139078140259, + "step": 5650, + "unclipped_grad_norm": 0.8717238903045654 + }, + { + "bits_per_byte": 1.497651924989523, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.0380932092666626, + "step": 5660, + "unclipped_grad_norm": 0.887111246585846 + }, + { + "bits_per_byte": 1.5673062744501676, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.0863739252090454, + "step": 5670, + "unclipped_grad_norm": 0.854918897151947 + }, + { + "bits_per_byte": 1.5629083341023038, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0833255052566528, + "step": 5680, + "unclipped_grad_norm": 0.8803378939628601 + }, + { + "bits_per_byte": 1.5827074928175828, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.0970492362976074, + "step": 5690, + "unclipped_grad_norm": 0.8571926951408386 + }, + { + "bits_per_byte": 1.5838535852029327, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.0978436470031738, + "step": 5700, + "unclipped_grad_norm": 0.8622198700904846 + }, + { + "bits_per_byte": 1.5674087761100817, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.0864449739456177, + "step": 5710, + "unclipped_grad_norm": 0.8891540765762329 + }, + { + "bits_per_byte": 1.5804761899052888, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0955026149749756, + "step": 5720, + "unclipped_grad_norm": 0.8605297803878784 + }, + { + "bits_per_byte": 1.5148931857385197, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.0500439405441284, + "step": 5730, + "unclipped_grad_norm": 0.8846258521080017 + }, + { + "bits_per_byte": 1.5794986405177847, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.094825029373169, + "step": 5740, + "unclipped_grad_norm": 0.8216924071311951 + }, + { + "bits_per_byte": 1.4878905336918518, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.0313271284103394, + "step": 5750, + "unclipped_grad_norm": 0.8151376843452454 + }, + { + "bits_per_byte": 1.4962136340803573, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.0370962619781494, + "step": 5760, + "unclipped_grad_norm": 0.8453561067581177 + }, + { + "bits_per_byte": 1.5253337365244481, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.0572807788848877, + "step": 5770, + "unclipped_grad_norm": 0.887503445148468 + }, + { + "bits_per_byte": 1.6049470853486052, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.1124645471572876, + "step": 5780, + "unclipped_grad_norm": 0.8942475914955139 + }, + { + "bits_per_byte": 1.4426902253747393, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 0.9999966621398926, + "step": 5790, + "unclipped_grad_norm": 0.7910417318344116 + }, + { + "bits_per_byte": 1.5562142533827708, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.0786855220794678, + "step": 5800, + "unclipped_grad_norm": 0.8394567370414734 + }, + { + "bits_per_byte": 1.4757829550711101, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.0229347944259644, + "step": 5810, + "unclipped_grad_norm": 0.8100422024726868 + }, + { + "bits_per_byte": 1.4892896125566208, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.032296895980835, + "step": 5820, + "unclipped_grad_norm": 0.8988341689109802 + }, + { + "bits_per_byte": 1.5528347942933165, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.076343059539795, + "step": 5830, + "unclipped_grad_norm": 0.8739818334579468 + }, + { + "bits_per_byte": 1.528282551056142, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.0593247413635254, + "step": 5840, + "unclipped_grad_norm": 0.8891842365264893 + }, + { + "bits_per_byte": 1.4910875191887405, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.0335431098937988, + "step": 5850, + "unclipped_grad_norm": 0.8053832054138184 + }, + { + "bits_per_byte": 1.504225789836103, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.0426498651504517, + "step": 5860, + "unclipped_grad_norm": 0.8241029381752014 + }, + { + "bits_per_byte": 1.5601206673144685, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0813932418823242, + "step": 5870, + "unclipped_grad_norm": 0.8466597199440002 + }, + { + "bits_per_byte": 1.5101533438807406, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.0467585325241089, + "step": 5880, + "unclipped_grad_norm": 0.8218726515769958 + }, + { + "bits_per_byte": 1.5450690896762571, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.070960283279419, + "step": 5890, + "unclipped_grad_norm": 0.9049459099769592 + }, + { + "bits_per_byte": 1.5844802899826766, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.0982780456542969, + "step": 5900, + "unclipped_grad_norm": 0.8854564428329468 + }, + { + "bits_per_byte": 1.5130032683881882, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.0487339496612549, + "step": 5910, + "unclipped_grad_norm": 0.839859664440155 + }, + { + "bits_per_byte": 1.4849007872892528, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.0292547941207886, + "step": 5920, + "unclipped_grad_norm": 0.853774905204773 + }, + { + "bits_per_byte": 1.6007457211706777, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.1095523834228516, + "step": 5930, + "unclipped_grad_norm": 0.9440886974334717 + }, + { + "bits_per_byte": 1.5326942500160747, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.062382698059082, + "step": 5940, + "unclipped_grad_norm": 0.9063394665718079 + }, + { + "bits_per_byte": 1.5398812330129805, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.0673643350601196, + "step": 5950, + "unclipped_grad_norm": 0.8717015385627747 + }, + { + "bits_per_byte": 1.4713537138807893, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.0198646783828735, + "step": 5960, + "unclipped_grad_norm": 0.8385668992996216 + }, + { + "bits_per_byte": 1.5036370932221996, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.0422418117523193, + "step": 5970, + "unclipped_grad_norm": 0.8743377923965454 + }, + { + "bits_per_byte": 1.490322368375052, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.0330127477645874, + "step": 5980, + "unclipped_grad_norm": 0.8565140962600708 + }, + { + "bits_per_byte": 1.491100933835508, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0335524082183838, + "step": 5990, + "unclipped_grad_norm": 0.866962730884552 + }, + { + "bits_per_byte": 1.5721176610907055, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.089708924293518, + "step": 6000, + "unclipped_grad_norm": 0.8703925609588623 + }, + { + "bits_per_byte": 1.5456141026968415, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0713380575180054, + "step": 6010, + "unclipped_grad_norm": 0.9116437435150146 + }, + { + "bits_per_byte": 1.5061998067027065, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0440181493759155, + "step": 6020, + "unclipped_grad_norm": 0.8624663949012756 + }, + { + "bits_per_byte": 1.4859792904928133, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.0300023555755615, + "step": 6030, + "unclipped_grad_norm": 0.8753954172134399 + }, + { + "bits_per_byte": 1.5236682565334927, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.056126356124878, + "step": 6040, + "unclipped_grad_norm": 0.8407245874404907 + }, + { + "bits_per_byte": 1.5360929711624256, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.0647385120391846, + "step": 6050, + "unclipped_grad_norm": 0.8646782040596008 + }, + { + "bits_per_byte": 1.5189576517263603, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.052861213684082, + "step": 6060, + "unclipped_grad_norm": 0.8866371512413025 + }, + { + "bits_per_byte": 1.4134779402297861, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 0.9797482490539551, + "step": 6070, + "unclipped_grad_norm": 0.8272168040275574 + }, + { + "bits_per_byte": 1.4889160662389467, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.0320379734039307, + "step": 6080, + "unclipped_grad_norm": 0.8825421333312988 + }, + { + "bits_per_byte": 1.4834005826257766, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.028214931488037, + "step": 6090, + "unclipped_grad_norm": 0.9016297459602356 + }, + { + "bits_per_byte": 1.5961228275154853, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.1063480377197266, + "step": 6100, + "unclipped_grad_norm": 0.9122457504272461 + }, + { + "bits_per_byte": 1.5235870807222855, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.05607008934021, + "step": 6110, + "unclipped_grad_norm": 0.8806039094924927 + }, + { + "bits_per_byte": 1.482743092951528, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0277591943740845, + "step": 6120, + "unclipped_grad_norm": 0.8462417125701904 + }, + { + "bits_per_byte": 1.5472673719195855, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.072484016418457, + "step": 6130, + "unclipped_grad_norm": 0.8564061522483826 + }, + { + "bits_per_byte": 1.4962796754182888, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.037142038345337, + "step": 6140, + "unclipped_grad_norm": 0.8453916907310486 + }, + { + "bits_per_byte": 1.466229490798332, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.016312837600708, + "step": 6150, + "unclipped_grad_norm": 0.8858425617218018 + }, + { + "bits_per_byte": 1.5397856106591008, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 1.0672980546951294, + "step": 6160, + "unclipped_grad_norm": 0.8804157376289368 + }, + { + "bits_per_byte": 1.4780109903130378, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 1.0244791507720947, + "step": 6170, + "unclipped_grad_norm": 0.8190087080001831 + }, + { + "bits_per_byte": 1.4770709331434217, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0238275527954102, + "step": 6180, + "unclipped_grad_norm": 0.8309900164604187 + }, + { + "bits_per_byte": 1.4993727833940564, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.0392860174179077, + "step": 6190, + "unclipped_grad_norm": 0.8653017282485962 + }, + { + "bits_per_byte": 1.4938571277982353, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.0354628562927246, + "step": 6200, + "unclipped_grad_norm": 0.8370429873466492 + }, + { + "bits_per_byte": 1.5110282196256797, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.0473649501800537, + "step": 6210, + "unclipped_grad_norm": 0.8925408720970154 + }, + { + "bits_per_byte": 1.5417909283681612, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0686880350112915, + "step": 6220, + "unclipped_grad_norm": 0.9388988614082336 + }, + { + "bits_per_byte": 1.477624545296549, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.0242112874984741, + "step": 6230, + "unclipped_grad_norm": 0.8727648854255676 + }, + { + "bits_per_byte": 1.5422649125539392, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.0690165758132935, + "step": 6240, + "unclipped_grad_norm": 0.9395257234573364 + }, + { + "bits_per_byte": 1.4772477313085086, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.0239500999450684, + "step": 6250, + "unclipped_grad_norm": 0.9107094407081604 + }, + { + "bits_per_byte": 1.439288408540673, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 0.9976387023925781, + "step": 6260, + "unclipped_grad_norm": 0.8751688599586487 + }, + { + "bits_per_byte": 1.5074798735730783, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.044905424118042, + "step": 6270, + "unclipped_grad_norm": 0.8716090321540833 + }, + { + "bits_per_byte": 1.5183749745052373, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.052457332611084, + "step": 6280, + "unclipped_grad_norm": 0.8638784885406494 + }, + { + "bits_per_byte": 1.4950828481509344, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 1.036312460899353, + "step": 6290, + "unclipped_grad_norm": 0.8782238960266113 + }, + { + "bits_per_byte": 1.4698543691305674, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.0188254117965698, + "step": 6300, + "unclipped_grad_norm": 0.8900427222251892 + }, + { + "bits_per_byte": 1.5384663317443323, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.0663836002349854, + "step": 6310, + "unclipped_grad_norm": 0.9044233560562134 + }, + { + "bits_per_byte": 1.5066746508017386, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 1.0443472862243652, + "step": 6320, + "unclipped_grad_norm": 0.9048281908035278 + }, + { + "bits_per_byte": 1.4939795794456496, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 1.0355477333068848, + "step": 6330, + "unclipped_grad_norm": 0.90269935131073 + }, + { + "bits_per_byte": 1.4334408264286866, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 0.993585467338562, + "step": 6340, + "unclipped_grad_norm": 0.8604462146759033 + }, + { + "bits_per_byte": 1.543729172843385, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.0700315237045288, + "step": 6350, + "unclipped_grad_norm": 0.8941039443016052 + }, + { + "bits_per_byte": 1.5526413138110957, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.0762089490890503, + "step": 6360, + "unclipped_grad_norm": 0.9167100787162781 + }, + { + "bits_per_byte": 1.4650109937169689, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 1.0154682397842407, + "step": 6370, + "unclipped_grad_norm": 0.8579038381576538 + }, + { + "bits_per_byte": 1.4610012182121026, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.0126888751983643, + "step": 6380, + "unclipped_grad_norm": 0.8909633159637451 + }, + { + "bits_per_byte": 1.575549230923371, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0920875072479248, + "step": 6390, + "unclipped_grad_norm": 0.908513069152832 + }, + { + "bits_per_byte": 1.5363303072206163, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.0649030208587646, + "step": 6400, + "unclipped_grad_norm": 0.928503692150116 + }, + { + "bits_per_byte": 1.5396418331629795, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.067198395729065, + "step": 6410, + "unclipped_grad_norm": 0.9374822974205017 + }, + { + "bits_per_byte": 1.4725835618171093, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0207171440124512, + "step": 6420, + "unclipped_grad_norm": 0.8952345848083496 + }, + { + "bits_per_byte": 1.5375078724310738, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.0657192468643188, + "step": 6430, + "unclipped_grad_norm": 0.9057692885398865 + }, + { + "bits_per_byte": 1.4531674084653017, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.0072588920593262, + "step": 6440, + "unclipped_grad_norm": 0.9215906262397766 + }, + { + "bits_per_byte": 1.4564670676047555, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0095460414886475, + "step": 6450, + "unclipped_grad_norm": 0.8423585295677185 + }, + { + "bits_per_byte": 1.4918280764833558, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.0340564250946045, + "step": 6460, + "unclipped_grad_norm": 0.91697096824646 + }, + { + "bits_per_byte": 1.4929280775182743, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.0348188877105713, + "step": 6470, + "unclipped_grad_norm": 0.9495468735694885 + }, + { + "bits_per_byte": 1.458121196740754, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 1.0106925964355469, + "step": 6480, + "unclipped_grad_norm": 0.8924557566642761 + }, + { + "bits_per_byte": 1.5293368047059308, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0600554943084717, + "step": 6490, + "unclipped_grad_norm": 0.9207810759544373 + }, + { + "bits_per_byte": 1.570303072141452, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.0884511470794678, + "step": 6500, + "unclipped_grad_norm": 0.9458774328231812 + }, + { + "bits_per_byte": 1.5428137091928422, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 1.06939697265625, + "step": 6510, + "unclipped_grad_norm": 0.9376916885375977 + }, + { + "bits_per_byte": 1.5294702632430004, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 1.060148000717163, + "step": 6520, + "unclipped_grad_norm": 0.9308140873908997 + }, + { + "bits_per_byte": 1.4947786108415582, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.0361015796661377, + "step": 6530, + "unclipped_grad_norm": 0.887566328048706 + }, + { + "bits_per_byte": 1.474609517444273, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.0221214294433594, + "step": 6540, + "unclipped_grad_norm": 0.9056259989738464 + }, + { + "bits_per_byte": 1.510869995586886, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 1.047255277633667, + "step": 6550, + "unclipped_grad_norm": 0.8734410405158997 + }, + { + "bits_per_byte": 1.541226997255982, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.0682971477508545, + "step": 6560, + "unclipped_grad_norm": 0.880778431892395 + }, + { + "bits_per_byte": 1.5071823435870857, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 1.0446991920471191, + "step": 6570, + "unclipped_grad_norm": 0.8809495568275452 + }, + { + "bits_per_byte": 1.489114878183344, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.0321757793426514, + "step": 6580, + "unclipped_grad_norm": 0.8892857432365417 + }, + { + "bits_per_byte": 1.5137823497965968, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.04927396774292, + "step": 6590, + "unclipped_grad_norm": 0.9584245681762695 + }, + { + "bits_per_byte": 1.4111243576527286, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9781168699264526, + "step": 6600, + "unclipped_grad_norm": 0.8921281099319458 + }, + { + "bits_per_byte": 1.4590683051990554, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.011349081993103, + "step": 6610, + "unclipped_grad_norm": 0.8777216672897339 + }, + { + "bits_per_byte": 1.5778854432526914, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 1.0937068462371826, + "step": 6620, + "unclipped_grad_norm": 0.9462138414382935 + }, + { + "bits_per_byte": 1.5193213950329352, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.053113341331482, + "step": 6630, + "unclipped_grad_norm": 0.8844531774520874 + }, + { + "bits_per_byte": 1.46968943776839, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.0187110900878906, + "step": 6640, + "unclipped_grad_norm": 0.8731166124343872 + }, + { + "bits_per_byte": 1.4801686846507625, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.0259747505187988, + "step": 6650, + "unclipped_grad_norm": 0.9090813994407654 + }, + { + "bits_per_byte": 1.5121753439069348, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.0481600761413574, + "step": 6660, + "unclipped_grad_norm": 0.9291538000106812 + }, + { + "bits_per_byte": 1.4834693756861215, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.0282626152038574, + "step": 6670, + "unclipped_grad_norm": 0.9064475297927856 + }, + { + "bits_per_byte": 1.504597100379316, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.0429072380065918, + "step": 6680, + "unclipped_grad_norm": 0.9069545865058899 + }, + { + "bits_per_byte": 1.4923935554393928, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.0344483852386475, + "step": 6690, + "unclipped_grad_norm": 0.9786862730979919 + }, + { + "bits_per_byte": 1.4981369160649562, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.0384293794631958, + "step": 6700, + "unclipped_grad_norm": 0.8994670510292053 + }, + { + "bits_per_byte": 1.5189211914043774, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.0528359413146973, + "step": 6710, + "unclipped_grad_norm": 0.9135090112686157 + }, + { + "bits_per_byte": 1.5860295097016488, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.0993518829345703, + "step": 6720, + "unclipped_grad_norm": 1.009243369102478 + }, + { + "bits_per_byte": 1.4490320856253054, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 1.0043925046920776, + "step": 6730, + "unclipped_grad_norm": 0.861769437789917 + }, + { + "bits_per_byte": 1.4784763753662724, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 1.0248017311096191, + "step": 6740, + "unclipped_grad_norm": 0.9036325216293335 + }, + { + "bits_per_byte": 1.4557223827265195, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 1.0090298652648926, + "step": 6750, + "unclipped_grad_norm": 0.9023477435112 + }, + { + "bits_per_byte": 1.446804910296632, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 1.002848744392395, + "step": 6760, + "unclipped_grad_norm": 0.8948139548301697 + }, + { + "bits_per_byte": 1.4662802256803364, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 1.0163480043411255, + "step": 6770, + "unclipped_grad_norm": 0.9269553422927856 + }, + { + "bits_per_byte": 1.459933893880848, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 1.011949062347412, + "step": 6780, + "unclipped_grad_norm": 0.8839923739433289 + }, + { + "bits_per_byte": 1.4856083239149023, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.0297452211380005, + "step": 6790, + "unclipped_grad_norm": 0.907511830329895 + }, + { + "bits_per_byte": 1.474204670284142, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0218408107757568, + "step": 6800, + "unclipped_grad_norm": 0.9107438325881958 + }, + { + "bits_per_byte": 1.47431387926744, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0219165086746216, + "step": 6810, + "unclipped_grad_norm": 0.8983246684074402 + }, + { + "bits_per_byte": 1.4866577620504673, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 1.0304726362228394, + "step": 6820, + "unclipped_grad_norm": 0.8864526748657227 + }, + { + "bits_per_byte": 1.482665528775989, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.027705430984497, + "step": 6830, + "unclipped_grad_norm": 0.924880862236023 + }, + { + "bits_per_byte": 1.4451179324743186, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 1.0016794204711914, + "step": 6840, + "unclipped_grad_norm": 0.9111179113388062 + }, + { + "bits_per_byte": 1.5433664614327154, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.0697801113128662, + "step": 6850, + "unclipped_grad_norm": 0.9932543635368347 + }, + { + "bits_per_byte": 1.4952388364152671, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.0364205837249756, + "step": 6860, + "unclipped_grad_norm": 0.9450015425682068 + }, + { + "bits_per_byte": 1.4372689882542418, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 0.9962389469146729, + "step": 6870, + "unclipped_grad_norm": 0.925460159778595 + }, + { + "bits_per_byte": 1.468765374985304, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 1.0180705785751343, + "step": 6880, + "unclipped_grad_norm": 0.9559142589569092 + }, + { + "bits_per_byte": 1.4594270610087552, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 1.011597752571106, + "step": 6890, + "unclipped_grad_norm": 0.916236162185669 + }, + { + "bits_per_byte": 1.5092156944682367, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 1.046108603477478, + "step": 6900, + "unclipped_grad_norm": 0.9155739545822144 + }, + { + "bits_per_byte": 1.5083105497767457, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.0454812049865723, + "step": 6910, + "unclipped_grad_norm": 0.9489511251449585 + }, + { + "bits_per_byte": 1.4408204299945588, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 0.9987006187438965, + "step": 6920, + "unclipped_grad_norm": 0.9210790395736694 + }, + { + "bits_per_byte": 1.4266341830641638, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.988867461681366, + "step": 6930, + "unclipped_grad_norm": 0.9696947336196899 + }, + { + "bits_per_byte": 1.505271272370698, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 1.0433745384216309, + "step": 6940, + "unclipped_grad_norm": 0.9561827778816223 + }, + { + "bits_per_byte": 1.4202685892077787, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 0.9844551682472229, + "step": 6950, + "unclipped_grad_norm": 0.9520508646965027 + }, + { + "bits_per_byte": 1.436690352625414, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 0.9958378672599792, + "step": 6960, + "unclipped_grad_norm": 0.9381454586982727 + }, + { + "bits_per_byte": 1.4058202407174687, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 0.974440336227417, + "step": 6970, + "unclipped_grad_norm": 0.9262760281562805 + }, + { + "bits_per_byte": 1.5237917400768122, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.0562119483947754, + "step": 6980, + "unclipped_grad_norm": 0.9989718794822693 + }, + { + "bits_per_byte": 1.4869202075756838, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.0306545495986938, + "step": 6990, + "unclipped_grad_norm": 0.9502958059310913 + }, + { + "bits_per_byte": 1.495212351087034, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 1.0364022254943848, + "step": 7000, + "unclipped_grad_norm": 0.9803596138954163 + }, + { + "bits_per_byte": 1.4144489542765575, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 0.9804213047027588, + "step": 7010, + "unclipped_grad_norm": 0.8976719975471497 + }, + { + "bits_per_byte": 1.4978571002920025, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 1.0382354259490967, + "step": 7020, + "unclipped_grad_norm": 0.9868860840797424 + }, + { + "bits_per_byte": 1.466947690348335, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 1.016810655593872, + "step": 7030, + "unclipped_grad_norm": 0.928314745426178 + }, + { + "bits_per_byte": 1.4566111890661784, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.009645938873291, + "step": 7040, + "unclipped_grad_norm": 0.9110724329948425 + }, + { + "bits_per_byte": 1.4428370985585761, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.000098466873169, + "step": 7050, + "unclipped_grad_norm": 0.9567721486091614 + }, + { + "bits_per_byte": 1.4311496735538922, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 0.9919973611831665, + "step": 7060, + "unclipped_grad_norm": 0.8998714685440063 + }, + { + "bits_per_byte": 1.4815839298847127, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 1.0269557237625122, + "step": 7070, + "unclipped_grad_norm": 0.960312008857727 + }, + { + "bits_per_byte": 1.4744239481639922, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 1.021992802619934, + "step": 7080, + "unclipped_grad_norm": 0.9459956884384155 + }, + { + "bits_per_byte": 1.4538379688210157, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 1.0077236890792847, + "step": 7090, + "unclipped_grad_norm": 0.9595316648483276 + }, + { + "bits_per_byte": 1.4346798754368277, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 0.9944443106651306, + "step": 7100, + "unclipped_grad_norm": 0.9327973127365112 + }, + { + "bits_per_byte": 1.3995189683725062, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 0.9700726270675659, + "step": 7110, + "unclipped_grad_norm": 0.8835809230804443 + }, + { + "bits_per_byte": 1.4691060726166632, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.0183067321777344, + "step": 7120, + "unclipped_grad_norm": 1.0125150680541992 + }, + { + "bits_per_byte": 1.5189961758401538, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 1.0528879165649414, + "step": 7130, + "unclipped_grad_norm": 0.9634286165237427 + }, + { + "bits_per_byte": 1.4641261429782797, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.014854907989502, + "step": 7140, + "unclipped_grad_norm": 0.9657710790634155 + }, + { + "bits_per_byte": 1.5009930319478348, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 1.0404090881347656, + "step": 7150, + "unclipped_grad_norm": 0.966037392616272 + }, + { + "bits_per_byte": 1.4960677927924257, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0369951725006104, + "step": 7160, + "unclipped_grad_norm": 0.9693772792816162 + }, + { + "bits_per_byte": 1.4959056131526622, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.036882758140564, + "step": 7170, + "unclipped_grad_norm": 1.01362144947052 + }, + { + "bits_per_byte": 1.4497315390663643, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 1.0048773288726807, + "step": 7180, + "unclipped_grad_norm": 0.9362047910690308 + }, + { + "bits_per_byte": 1.4999669834527873, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 1.0396978855133057, + "step": 7190, + "unclipped_grad_norm": 0.9817818403244019 + }, + { + "bits_per_byte": 1.4417734718543151, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 0.9993612170219421, + "step": 7200, + "unclipped_grad_norm": 0.9063348770141602 + }, + { + "bits_per_byte": 1.4395646126779587, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 0.9978301525115967, + "step": 7210, + "unclipped_grad_norm": 0.9212372899055481 + }, + { + "bits_per_byte": 1.4435238252834712, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 1.0005744695663452, + "step": 7220, + "unclipped_grad_norm": 0.9449443221092224 + }, + { + "bits_per_byte": 1.4777394297073254, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 1.024290919303894, + "step": 7230, + "unclipped_grad_norm": 0.9872192144393921 + }, + { + "bits_per_byte": 1.5044591702933239, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 1.042811632156372, + "step": 7240, + "unclipped_grad_norm": 0.9781456589698792 + }, + { + "bits_per_byte": 1.4730518705754088, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 1.021041750907898, + "step": 7250, + "unclipped_grad_norm": 0.9568102359771729 + }, + { + "bits_per_byte": 1.454242300033194, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 1.0080039501190186, + "step": 7260, + "unclipped_grad_norm": 0.9279366731643677 + }, + { + "bits_per_byte": 1.4676777567012476, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 1.0173166990280151, + "step": 7270, + "unclipped_grad_norm": 0.9813672304153442 + }, + { + "bits_per_byte": 1.4655941868860447, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.0158724784851074, + "step": 7280, + "unclipped_grad_norm": 0.9574845433235168 + }, + { + "bits_per_byte": 1.4683584640333627, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 1.0177885293960571, + "step": 7290, + "unclipped_grad_norm": 0.9963250160217285 + }, + { + "bits_per_byte": 1.4530573395687494, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 1.0071825981140137, + "step": 7300, + "unclipped_grad_norm": 0.9040587544441223 + }, + { + "bits_per_byte": 1.4288987646193998, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 0.9904371500015259, + "step": 7310, + "unclipped_grad_norm": 0.9713231921195984 + }, + { + "bits_per_byte": 1.4258029049212184, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 0.9882912635803223, + "step": 7320, + "unclipped_grad_norm": 0.9730747938156128 + }, + { + "bits_per_byte": 1.4716822007439372, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 1.0200923681259155, + "step": 7330, + "unclipped_grad_norm": 1.0055556297302246 + }, + { + "bits_per_byte": 1.4343633413679147, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.9942249059677124, + "step": 7340, + "unclipped_grad_norm": 0.9297077655792236 + }, + { + "bits_per_byte": 1.441064645358784, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 0.9988698959350586, + "step": 7350, + "unclipped_grad_norm": 1.007290005683899 + }, + { + "bits_per_byte": 1.443886536694141, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 1.0008258819580078, + "step": 7360, + "unclipped_grad_norm": 0.9124701619148254 + }, + { + "bits_per_byte": 1.4135542145354438, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 0.9798011183738708, + "step": 7370, + "unclipped_grad_norm": 0.9996932744979858 + }, + { + "bits_per_byte": 1.4007590492765525, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 0.9709321856498718, + "step": 7380, + "unclipped_grad_norm": 0.9084812998771667 + }, + { + "bits_per_byte": 1.416424003039066, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 0.98179030418396, + "step": 7390, + "unclipped_grad_norm": 0.9706323146820068 + }, + { + "bits_per_byte": 1.434177428122332, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.994096040725708, + "step": 7400, + "unclipped_grad_norm": 0.934651255607605 + }, + { + "bits_per_byte": 1.4815827260061565, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 1.0269548892974854, + "step": 7410, + "unclipped_grad_norm": 1.014525294303894 + }, + { + "bits_per_byte": 1.4582345333076725, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0107711553573608, + "step": 7420, + "unclipped_grad_norm": 0.936613142490387 + }, + { + "bits_per_byte": 1.4059580848121351, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 0.9745358824729919, + "step": 7430, + "unclipped_grad_norm": 0.9683210849761963 + }, + { + "bits_per_byte": 1.3910142543046895, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 0.9641776084899902, + "step": 7440, + "unclipped_grad_norm": 0.9340444207191467 + }, + { + "bits_per_byte": 1.4945049864440354, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 1.0359119176864624, + "step": 7450, + "unclipped_grad_norm": 0.9410877227783203 + }, + { + "bits_per_byte": 1.4247725568598992, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 0.9875770807266235, + "step": 7460, + "unclipped_grad_norm": 0.9573809504508972 + }, + { + "bits_per_byte": 1.412823546243253, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 0.9792946577072144, + "step": 7470, + "unclipped_grad_norm": 0.9741236567497253 + }, + { + "bits_per_byte": 1.4962365077729223, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 1.0371121168136597, + "step": 7480, + "unclipped_grad_norm": 0.9679830074310303 + }, + { + "bits_per_byte": 1.4530341219108829, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 1.0071665048599243, + "step": 7490, + "unclipped_grad_norm": 0.9712881445884705 + }, + { + "bits_per_byte": 1.4784044866182118, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.024751901626587, + "step": 7500, + "unclipped_grad_norm": 0.9908523559570312 + }, + { + "bits_per_byte": 1.463076360877413, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 1.014127254486084, + "step": 7510, + "unclipped_grad_norm": 0.9636465907096863 + }, + { + "bits_per_byte": 1.471756669231761, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 1.020143985748291, + "step": 7520, + "unclipped_grad_norm": 0.972665548324585 + }, + { + "bits_per_byte": 1.4404458517809797, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 0.9984409809112549, + "step": 7530, + "unclipped_grad_norm": 0.9699863195419312 + }, + { + "bits_per_byte": 1.4711538700404867, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.0197261571884155, + "step": 7540, + "unclipped_grad_norm": 0.980201780796051 + }, + { + "bits_per_byte": 1.4462072705848836, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 1.002434492111206, + "step": 7550, + "unclipped_grad_norm": 0.9429918527603149 + }, + { + "bits_per_byte": 1.380956020951629, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 0.9572057723999023, + "step": 7560, + "unclipped_grad_norm": 0.9305909872055054 + }, + { + "bits_per_byte": 1.4041671434773753, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 0.9732944965362549, + "step": 7570, + "unclipped_grad_norm": 0.9315748810768127 + }, + { + "bits_per_byte": 1.3960146498785255, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 0.9676436185836792, + "step": 7580, + "unclipped_grad_norm": 0.9846163392066956 + }, + { + "bits_per_byte": 1.4686554780714027, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 1.0179944038391113, + "step": 7590, + "unclipped_grad_norm": 0.9471601843833923 + }, + { + "bits_per_byte": 1.5076052489255574, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 1.0449923276901245, + "step": 7600, + "unclipped_grad_norm": 1.0260101556777954 + }, + { + "bits_per_byte": 1.4645736418358248, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 1.015165090560913, + "step": 7610, + "unclipped_grad_norm": 1.019962191581726 + }, + { + "bits_per_byte": 1.4685780858785145, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 1.0179407596588135, + "step": 7620, + "unclipped_grad_norm": 0.9725398421287537 + }, + { + "bits_per_byte": 1.4427345968986618, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 1.0000274181365967, + "step": 7630, + "unclipped_grad_norm": 0.9622707366943359 + }, + { + "bits_per_byte": 1.4240602907113509, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.987083375453949, + "step": 7640, + "unclipped_grad_norm": 1.0363737344741821 + }, + { + "bits_per_byte": 1.3863296188778373, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 0.9609304666519165, + "step": 7650, + "unclipped_grad_norm": 0.9393255114555359 + }, + { + "bits_per_byte": 1.4005732220222953, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 0.9708033800125122, + "step": 7660, + "unclipped_grad_norm": 1.0069085359573364 + }, + { + "bits_per_byte": 1.4675523813487685, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 1.0172297954559326, + "step": 7670, + "unclipped_grad_norm": 1.0074222087860107 + }, + { + "bits_per_byte": 1.4551704903999008, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 1.0086473226547241, + "step": 7680, + "unclipped_grad_norm": 0.9800598621368408 + }, + { + "bits_per_byte": 1.4356385927240625, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 0.9951088428497314, + "step": 7690, + "unclipped_grad_norm": 0.9748636484146118 + }, + { + "bits_per_byte": 1.4847313843781529, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 1.029137372970581, + "step": 7700, + "unclipped_grad_norm": 0.9703730344772339 + }, + { + "bits_per_byte": 1.408783759765811, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 0.9764944911003113, + "step": 7710, + "unclipped_grad_norm": 0.9494746327400208 + }, + { + "bits_per_byte": 1.4232927321405502, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 0.9865513443946838, + "step": 7720, + "unclipped_grad_norm": 0.9357646107673645 + }, + { + "bits_per_byte": 1.412793793244654, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 0.9792740345001221, + "step": 7730, + "unclipped_grad_norm": 0.943286120891571 + }, + { + "bits_per_byte": 1.4558001188847094, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 1.0090837478637695, + "step": 7740, + "unclipped_grad_norm": 1.0220750570297241 + }, + { + "bits_per_byte": 1.4080465561328877, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9759835004806519, + "step": 7750, + "unclipped_grad_norm": 0.957589864730835 + }, + { + "bits_per_byte": 1.4107376546622634, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 0.9778488278388977, + "step": 7760, + "unclipped_grad_norm": 0.9725729823112488 + }, + { + "bits_per_byte": 1.471751681734886, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 1.020140528678894, + "step": 7770, + "unclipped_grad_norm": 0.9984660744667053 + }, + { + "bits_per_byte": 1.444443072552333, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 1.0012116432189941, + "step": 7780, + "unclipped_grad_norm": 1.0003443956375122 + }, + { + "bits_per_byte": 1.457994617509719, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 1.0106048583984375, + "step": 7790, + "unclipped_grad_norm": 0.9923803210258484 + }, + { + "bits_per_byte": 1.4406072574988145, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 0.9985528588294983, + "step": 7800, + "unclipped_grad_norm": 0.9890941977500916 + }, + { + "bits_per_byte": 1.435981526129883, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9953465461730957, + "step": 7810, + "unclipped_grad_norm": 1.0434261560440063 + }, + { + "bits_per_byte": 1.452067407430383, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 1.0064964294433594, + "step": 7820, + "unclipped_grad_norm": 1.055299997329712 + }, + { + "bits_per_byte": 1.4621292524191116, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 1.0134707689285278, + "step": 7830, + "unclipped_grad_norm": 1.0030983686447144 + }, + { + "bits_per_byte": 1.425639349420248, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.9881778955459595, + "step": 7840, + "unclipped_grad_norm": 0.9846630692481995 + }, + { + "bits_per_byte": 1.4229758541063355, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.9863317012786865, + "step": 7850, + "unclipped_grad_norm": 0.9627443552017212 + }, + { + "bits_per_byte": 1.4582936953395693, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 1.0108121633529663, + "step": 7860, + "unclipped_grad_norm": 1.0141046047210693 + }, + { + "bits_per_byte": 1.4544431757694019, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0081431865692139, + "step": 7870, + "unclipped_grad_norm": 1.0117058753967285 + }, + { + "bits_per_byte": 1.416590052288474, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9819054007530212, + "step": 7880, + "unclipped_grad_norm": 0.9525494575500488 + }, + { + "bits_per_byte": 1.4524326985808156, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 1.0067496299743652, + "step": 7890, + "unclipped_grad_norm": 0.9639500379562378 + }, + { + "bits_per_byte": 1.3912243311127184, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 0.9643232226371765, + "step": 7900, + "unclipped_grad_norm": 0.9695078134536743 + }, + { + "bits_per_byte": 1.4483622132001948, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.0039281845092773, + "step": 7910, + "unclipped_grad_norm": 1.0733834505081177 + }, + { + "bits_per_byte": 1.4693131397283021, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 1.0184502601623535, + "step": 7920, + "unclipped_grad_norm": 0.994348406791687 + }, + { + "bits_per_byte": 1.4304835847471007, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.9915356636047363, + "step": 7930, + "unclipped_grad_norm": 0.9915876388549805 + }, + { + "bits_per_byte": 1.4602771712519702, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 1.0121870040893555, + "step": 7940, + "unclipped_grad_norm": 1.0345884561538696 + }, + { + "bits_per_byte": 1.3997759104528953, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 0.9702507257461548, + "step": 7950, + "unclipped_grad_norm": 0.9855912923812866 + }, + { + "bits_per_byte": 1.4742569530100045, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 1.0218770503997803, + "step": 7960, + "unclipped_grad_norm": 1.0853742361068726 + }, + { + "bits_per_byte": 1.438376298551822, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 0.9970064759254456, + "step": 7970, + "unclipped_grad_norm": 0.9717780947685242 + }, + { + "bits_per_byte": 1.4560212885737192, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 1.0092370510101318, + "step": 7980, + "unclipped_grad_norm": 1.0712512731552124 + }, + { + "bits_per_byte": 1.4229655351472839, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 0.9863245487213135, + "step": 7990, + "unclipped_grad_norm": 1.0590827465057373 + }, + { + "bits_per_byte": 1.446567918203743, + "learning_rate": 3e-05, + "loss_nats": 1.002684473991394, + "step": 8000, + "unclipped_grad_norm": 0.9954193234443665 + } + ], + "variant": "uniform_group_7_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073003.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073003.json new file mode 100644 index 0000000..1c7abb9 --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073003.json @@ -0,0 +1,14045 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "1db04837746b577f6bca4a5e4f98eea1cfce9dab38b1c7eecb7d014d98f330c1", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.00027819047681987286, + 0.0002560806169640273, + 0.0002331122086616233, + 0.00022053472639527172, + 0.00020666928321588784, + 0.00019684624567162246, + 0.0001890837011160329, + 0.00018263237143401057, + 0.00017469045997131616, + 0.0001688334159553051, + 0.0001649101759539917, + 0.00016157960635609925, + 0.00015717606584075838, + 0.00015400603297166526, + 0.00015046694898046553, + 0.00014767753600608557, + 0.00014588078192900866, + 0.00014366162940859795, + 0.00014116008242126554, + 0.00013849612150806934, + 0.0001361419854220003, + 0.00013430001854430884, + 0.00013224364374764264, + 0.00013045054220128804, + 0.00012888714263681322, + 0.00012767451698891819, + 0.0001263424928765744, + 0.00012512138346210122, + 0.00012419422273524106, + 0.00012343283742666245, + 0.0001227961911354214, + 0.00012195061572128907 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00022039370378479362, + "first_to_last_ratio": 1.7624457042066763, + "imbalance_abs_log_ratio": 0.5667024491058967, + "last_quartile_mean": 0.00012504992537287762, + "mean": 0.0001607882525149762, + "normalized": [ + 1.7301666786506156, + 1.592657504255016, + 1.4498087081325215, + 1.3715848200709229, + 1.2853506396348091, + 1.224257634451794, + 1.175979576607571, + 1.1358564358860717, + 1.0864628307036615, + 1.050035766385231, + 1.0256357250890054, + 1.0049217142965674, + 0.9775345112735686, + 0.9578189361646354, + 0.9358080993289649, + 0.9184597363064851, + 0.9072850761620224, + 0.8934833681037547, + 0.8779253472396409, + 0.8613572157279928, + 0.8467159963027754, + 0.8352601414820391, + 0.8224708066612338, + 0.81131886291965, + 0.8015955184587156, + 0.7940537632065271, + 0.7857694259399116, + 0.7781749070905981, + 0.7724085609032495, + 0.7676732316943716, + 0.7637136993200662, + 0.75845476155001 + ], + "population_cv": 0.2498932502086342 + }, + "activation_output_rms_by_block": [ + 0.0034565660171210766, + 0.004928664769977331, + 0.0060537648387253284, + 0.006960673723369837, + 0.0036145171616226435, + 0.0051584807224571705, + 0.006370538379997015, + 0.007546419743448496, + 0.00395671371370554, + 0.005665866658091545, + 0.006755253300070763, + 0.007946331985294819, + 0.004383862018585205, + 0.006652849726378918, + 0.007932198233902454, + 0.009347881190478802, + 0.004880861379206181, + 0.006906712893396616, + 0.008132148534059525, + 0.009816600009799004, + 0.005453306715935469, + 0.007622495759278536, + 0.009448722936213017, + 0.010670131072402, + 0.0055701653473079205, + 0.007738742511719465, + 0.009509840048849583, + 0.010865328833460808, + 0.005987230688333511, + 0.00815181527286768, + 0.009874933399260044, + 0.011666664853692055 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.005511203169589862, + "first_to_last_ratio": 0.6356203088456116, + "imbalance_abs_log_ratio": 0.45315389258709987, + "last_quartile_mean": 0.008670590119436383, + "mean": 0.007157071326219011, + "normalized": [ + 0.4829581625738996, + 0.6886426787339398, + 0.8458438602600115, + 0.9725589429115095, + 0.5050274053272774, + 0.7207530129760953, + 0.8901040788372985, + 1.0544005221525679, + 0.5528397766850007, + 0.7916459680002588, + 0.9438571997073384, + 1.1102770425361632, + 0.6125217730505899, + 0.9295491721602742, + 1.1083022471557418, + 1.3061042379491232, + 0.6819634955048963, + 0.9650194302374437, + 1.1362396940587196, + 1.3715945478756304, + 0.7619466772614072, + 1.0650300118366156, + 1.3201940438400879, + 1.490851576861243, + 0.7782743937317399, + 1.0812722353862223, + 1.3287334463206348, + 1.5181249897087195, + 0.8365475786723071, + 1.138987569259587, + 1.3797450031110483, + 1.6300892253166077 + ], + "population_cv": 0.3025375276561286 + }, + "bits_per_byte": 8.06433666787865, + "branch_output_rms_by_sublayer": [ + 0.0021227982360869646, + 0.0027448106557130814, + 0.00224129855632782, + 0.0027243648655712605, + 0.0024903982412070036, + 0.0027103275060653687, + 0.0023722455371171236, + 0.0027234479784965515, + 0.0023577234242111444, + 0.0027595730498433113, + 0.0023993616923689842, + 0.0027272943407297134, + 0.002688874490559101, + 0.002728298306465149, + 0.0029664887115359306, + 0.0027007071767002344, + 0.0029547461308538914, + 0.002684502862393856, + 0.003052416956052184, + 0.002683063969016075, + 0.0028214550111442804, + 0.0027139110025018454, + 0.0032544767018407583, + 0.002666700165718794, + 0.00348988501355052, + 0.002679243916645646, + 0.003914494998753071, + 0.0026584581937640905, + 0.0035071265883743763, + 0.002684473292902112, + 0.004211232997477055, + 0.0026407993864268064, + 0.004072706215083599, + 0.002663099905475974, + 0.004050548188388348, + 0.0026283918414264917, + 0.003929397091269493, + 0.002597011858597398, + 0.004410331137478352, + 0.0026434636674821377, + 0.004824656527489424, + 0.0025708195753395557, + 0.00433384021744132, + 0.002691218862310052, + 0.004475373309105635, + 0.0026598076801747084, + 0.004755842965096235, + 0.0026596153620630503, + 0.004843301139771938, + 0.0026765079237520695, + 0.0050001065246760845, + 0.002582729561254382, + 0.005346097983419895, + 0.002614464145153761, + 0.004958339501172304, + 0.0025760054122656584, + 0.0054801348596811295, + 0.002634295728057623, + 0.005455824080854654, + 0.00259007909335196, + 0.005286100786179304, + 0.00257018837146461, + 0.005402065813541412, + 0.0025643527042120695 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.006159982634052032, + 0.005514663551817784, + 0.005191681274696802, + 0.00489831575692433, + 0.004666415075367917, + 0.004399813239791133, + 0.004483878567781538, + 0.004868281936291656, + 0.00443828064080646, + 0.004266304440803418, + 0.0039471188591477255, + 0.004494388758593749, + 0.0042334459038140095, + 0.00446434979681851, + 0.004322339302329385, + 0.004638471384339534, + 0.004165330760400444, + 0.004499937764754925, + 0.003981478974932938, + 0.004372386299982334, + 0.00458495381977442, + 0.00429501466057645, + 0.003963900484885007, + 0.004271473918508079, + 0.004041799650627023, + 0.004206875397956982, + 0.003955591472942362, + 0.003916464042690177, + 0.003989587235588314, + 0.003922309599766117, + 0.003679536678174494, + 0.003763244111925236 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.005022879004590398, + "first_to_last_ratio": 1.2766484804448084, + "imbalance_abs_log_ratio": 0.24423826934058726, + "last_quartile_mean": 0.003934426023708838, + "mean": 0.0043936754999019155, + "normalized": [ + 1.4020112851278042, + 1.255136741878387, + 1.1816260155791434, + 1.1148560600421402, + 1.0620754936207943, + 1.0013969488391563, + 1.0205302070855338, + 1.108020366183241, + 1.0101521245493756, + 0.9710103627130996, + 0.8983637638318627, + 1.0229223252136126, + 0.9635317637610054, + 1.0160854612313022, + 0.9837638902613262, + 1.0557155130011497, + 0.9480287655502622, + 1.0241852783291305, + 0.9061841219320409, + 0.995154580733134, + 1.0435349219296635, + 0.9775448051801577, + 0.9021832597727114, + 0.9721869351988865, + 0.9199131002545027, + 0.9574843199164109, + 0.9002921296829196, + 0.891386731400079, + 0.9080295610536958, + 0.892717179467095, + 0.8374620925593245, + 0.8565138941210488 + ], + "population_cv": 0.11556590700625036 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028097106143832207, + 0.014079716056585312, + 0.014136734418570995, + 0.014167090877890587, + 0.014224153943359852, + 0.014255538582801819, + 0.014337832108139992, + 0.014355218037962914, + 0.014408928342163563, + 0.009655896574258804, + 0.009704872034490108, + 0.009734091348946095, + 0.009773381985723972, + 0.009828771464526653, + 0.00987160298973322, + 0.009915035218000412, + 0.00994801800698042, + 0.007479202933609486, + 0.007515520788729191, + 0.007559895049780607, + 0.007602147292345762, + 0.0076358383521437645, + 0.007658426649868488, + 0.007706487085670233, + 0.007734050042927265, + 0.0062490664422512054, + 0.0062688966281712055, + 0.006315518636256456, + 0.006336068734526634, + 0.0063744354993104935, + 0.006399397272616625, + 0.006439714692533016, + 0.006462901830673218, + 0.0054116579703986645, + 0.005432929843664169, + 0.005481995176523924, + 0.005504137370735407, + 0.005541813559830189, + 0.005560056772083044, + 0.005614723544567823, + 0.005631812382489443, + 0.004862982779741287, + 0.004873365629464388, + 0.004923288244754076, + 0.004937449004501104, + 0.0049925679340958595, + 0.005005750339478254, + 0.005004179198294878, + 0.005022916477173567, + 0.004448203835636377, + 0.0044684382155537605, + 0.004490054678171873, + 0.004500201437622309, + 0.00455933902412653, + 0.004573327023535967, + 0.004625064320862293, + 0.00463611725717783, + 0.004160413984209299, + 0.004166214261204004, + 0.004242690280079842, + 0.0042496914975345135, + 0.004270957782864571, + 0.004283483140170574, + 0.004303553607314825 + ], + "loss_nats": 5.5897722244262695, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0021227982360869646, + 0.0034565660171210766, + 0.00413061399012804, + 0.004928664769977331, + 0.005416445434093475, + 0.0060537648387253284, + 0.006388718727976084, + 0.006960673723369837, + 0.0023577234242111444, + 0.0036145171616226435, + 0.004388594534248114, + 0.0051584807224571705, + 0.00574506726115942, + 0.006370538379997015, + 0.007066869642585516, + 0.007546419743448496, + 0.0029547461308538914, + 0.00395671371370554, + 0.004991721361875534, + 0.005665866658091545, + 0.006190058775246143, + 0.006755253300070763, + 0.0074937604367733, + 0.007946331985294819, + 0.00348988501355052, + 0.004383862018585205, + 0.006057556718587875, + 0.006652849726378918, + 0.0074848332442343235, + 0.007932198233902454, + 0.00897275935858488, + 0.009347881190478802, + 0.004072706215083599, + 0.004880861379206181, + 0.0063449000008404255, + 0.006906712893396616, + 0.007735534105449915, + 0.008132148534059525, + 0.00946879480034113, + 0.009816600009799004, + 0.004824656527489424, + 0.005453306715935469, + 0.007074959110468626, + 0.007622495759278536, + 0.009095527231693268, + 0.009448722936213017, + 0.010346688330173492, + 0.010670131072402, + 0.004843301139771938, + 0.0055701653473079205, + 0.007334613241255283, + 0.007738742511719465, + 0.009146629832684994, + 0.009509840048849583, + 0.010587968863546848, + 0.010865328833460808, + 0.0054801348596811295, + 0.005987230688333511, + 0.0077637480571866035, + 0.00815181527286768, + 0.009536821395158768, + 0.009874933399260044, + 0.011367839761078358, + 0.011666664853692055 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.856055551907048e-05, + 6.497735012089834e-05, + 6.158350879559293e-05, + 5.873005648027174e-05, + 5.403308750828728e-05, + 5.094989319331944e-05, + 4.805844582733698e-05, + 4.539640576695092e-05, + 4.127224747207947e-05, + 3.848878259304911e-05, + 3.6076537071494386e-05, + 3.3792941394494846e-05, + 3.130429104203358e-05, + 2.9398181141004898e-05, + 2.774226049950812e-05, + 2.6118363166460767e-05, + 2.4807424779282883e-05, + 2.351236980757676e-05, + 2.2460511900135316e-05, + 2.1600159016088583e-05, + 2.116589530487545e-05, + 2.0522125851130113e-05, + 2.0003966710646637e-05, + 1.9538212654879317e-05, + 1.91778290172806e-05, + 1.87541500054067e-05, + 1.8374463252257556e-05, + 1.8080017980537377e-05, + 1.7766866221791133e-05, + 1.7555596059537493e-05, + 1.7353393559460528e-05, + 1.720745967759285e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.6536162901466014e-05, + "first_to_last_ratio": 3.1350246493809575, + "imbalance_abs_log_ratio": 1.1426370366954273, + "last_quartile_mean": 1.803372197173303e-05, + "mean": 3.232385466844789e-05, + "normalized": [ + 2.121051348061966, + 2.010198065403515, + 1.905203120954077, + 1.8169261396166216, + 1.6716164598101073, + 1.5762319722051243, + 1.486779541619707, + 1.404424262903999, + 1.2768355722241977, + 1.1907237855087547, + 1.1160963765472431, + 1.0454489955209756, + 0.9684578576140694, + 0.9094887179313172, + 0.8582596594393187, + 0.8080213029776904, + 0.7674649274889241, + 0.7273999357053094, + 0.6948587082362913, + 0.668242053357981, + 0.6548072784628622, + 0.634891044450904, + 0.6188608046853087, + 0.6044518160128669, + 0.5933026618882973, + 0.5801953448241767, + 0.5684490120602268, + 0.5593397868536306, + 0.549651840847243, + 0.5431157960462537, + 0.5368602766426741, + 0.5323455340983659 + ], + "population_cv": 0.4871758972207208 + }, + "activation_output_rms_by_block": [ + 0.005520354490727186, + 0.009551535360515118, + 0.013990547508001328, + 0.01876879669725895, + 0.006623269058763981, + 0.011789736337959766, + 0.017194580286741257, + 0.02232358604669571, + 0.0074646128341555595, + 0.013205114752054214, + 0.019117774441838264, + 0.025769757106900215, + 0.008969931863248348, + 0.016233347356319427, + 0.023725751787424088, + 0.032117463648319244, + 0.009894398041069508, + 0.01924053207039833, + 0.028615590184926987, + 0.038781892508268356, + 0.012692742049694061, + 0.022670142352581024, + 0.03462707996368408, + 0.045925430953502655, + 0.014747132547199726, + 0.029425622895359993, + 0.04612904414534569, + 0.062327586114406586, + 0.019631674513220787, + 0.037373341619968414, + 0.05553118884563446, + 0.07465041428804398 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013220300723332912, + "first_to_last_ratio": 0.311234327518669, + "imbalance_abs_log_ratio": 1.1672091858246663, + "last_quartile_mean": 0.042477000621147454, + "mean": 0.025144686645944603, + "normalized": [ + 0.2195435786676436, + 0.3798629704560511, + 0.5564017442332199, + 0.7464319186608765, + 0.2634063073453412, + 0.4688758505511638, + 0.6838255942041911, + 0.8878052990290938, + 0.2968664091647955, + 0.5251652143449546, + 0.7603107054396968, + 1.0248589481251866, + 0.3567326962372403, + 0.6455975206570165, + 0.9435691970037191, + 1.2773061800513321, + 0.39349856215707907, + 0.7651927558819466, + 1.1380372556578342, + 1.5423494058345402, + 0.5047882373090212, + 0.9015877855943503, + 1.3771132029306408, + 1.8264467400276638, + 0.5864910052310467, + 1.1702521216388206, + 1.834544402754985, + 2.4787577189576524, + 0.7807484256872628, + 1.4863315716044558, + 2.208466131535215, + 2.968834543025963 + ], + "population_cv": 0.6640351054242409 + }, + "bits_per_byte": 5.183306690265396, + "branch_output_rms_by_sublayer": [ + 0.0027192807756364346, + 0.004830502439290285, + 0.0025354167446494102, + 0.004947597160935402, + 0.0027586505748331547, + 0.00550014479085803, + 0.0026530595496296883, + 0.005900143180042505, + 0.002794948872178793, + 0.006002394482493401, + 0.0025344090536236763, + 0.006053258664906025, + 0.0034126618411391973, + 0.006182783283293247, + 0.003361097304150462, + 0.005940655246376991, + 0.0035717571154236794, + 0.006378115154802799, + 0.0038430916611105204, + 0.006210959982126951, + 0.0033996535930782557, + 0.006491359323263168, + 0.00462525337934494, + 0.006910249590873718, + 0.004883658606559038, + 0.007106319069862366, + 0.005147983785718679, + 0.006858345586806536, + 0.0053953747265040874, + 0.006823142524808645, + 0.006122484337538481, + 0.007109105121344328, + 0.005602571647614241, + 0.007117910776287317, + 0.006383119150996208, + 0.007638407871127129, + 0.007230950053781271, + 0.007455360610038042, + 0.00872944574803114, + 0.007448327261954546, + 0.00841959286481142, + 0.007472616620361805, + 0.007553893607109785, + 0.007771024480462074, + 0.009505834430456161, + 0.007440135348588228, + 0.008882961235940456, + 0.007348866201937199, + 0.009543929249048233, + 0.008383802138268948, + 0.011848044581711292, + 0.00832671020179987, + 0.014113695360720158, + 0.008139979094266891, + 0.013638543896377087, + 0.007711624726653099, + 0.014510786160826683, + 0.007937537506222725, + 0.01456385012716055, + 0.007786803413182497, + 0.013712815940380096, + 0.008720467798411846, + 0.015197417698800564, + 0.008378061465919018 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00023799521275294372, + 0.0002059742287546603, + 0.00019808145059403495, + 0.0001821546631258852, + 0.00017103805826090878, + 0.00016024913986910393, + 0.00017452052305308162, + 0.00016100726640073227, + 0.00016519216481828496, + 0.0001623884351250268, + 0.00014788080551127197, + 0.0001707281811685715, + 0.00016353530249407712, + 0.00016673768929249388, + 0.00015432336283583494, + 0.0001606992521370764, + 0.00014143617120038071, + 0.0001546893693026788, + 0.0001458281741227443, + 0.0001641505213035429, + 0.00015497260242045924, + 0.000136327103853323, + 0.00015227871410822036, + 0.00014425676537967722, + 0.00014468544478740003, + 0.00015395895400407338, + 0.00015694479737136447, + 0.00014458118440368388, + 0.00014464904726422444, + 0.0001414258696651933, + 0.0001362863701099936, + 0.00013493478268524059 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00018637756785141885, + "first_to_last_ratio": 1.288176035198487, + "imbalance_abs_log_ratio": 0.25322729163396934, + "last_quartile_mean": 0.00014468330628639671, + "mean": 0.0001604347377555059, + "normalized": [ + 1.4834394102082549, + 1.2838505652594652, + 1.2346543732685913, + 1.135381686499088, + 1.0660911760990428, + 0.9988431564822027, + 1.087797602281382, + 1.0035686077294488, + 1.0296533477059633, + 1.0121775208839012, + 0.9217505359508521, + 1.064159692327682, + 1.019326018678675, + 1.039286700780434, + 0.9619074085502332, + 1.0016487350886165, + 0.8815807173626076, + 0.9641887503092832, + 0.9089563529874607, + 1.023160716937124, + 0.965954160480066, + 0.8497355732340107, + 0.9491629820237878, + 0.8991616616067085, + 0.901833647821914, + 0.9596360249530169, + 0.9782469779739353, + 0.9011837861699129, + 0.9016067797278541, + 0.881516507233732, + 0.8494816772018966, + 0.8410571461828553 + ], + "population_cv": 0.13324883769890492 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6920492649078369, + "mean_weights": [ + 0.5230485200881958, + 0.476951539516449 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6922354102134705, + "mean_weights": [ + 0.520708441734314, + 0.47929155826568604 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6930611729621887, + "mean_weights": [ + 0.502327024936676, + 0.49767300486564636 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6924432516098022, + "mean_weights": [ + 0.5176426768302917, + 0.48235729336738586 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929880380630493, + "mean_weights": [ + 0.5013898611068726, + 0.49861010909080505 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929055452346802, + "mean_weights": [ + 0.5098497867584229, + 0.49015021324157715 + ], + "sources": 2 + }, + { + "entropy_mean": 0.692742109298706, + "mean_weights": [ + 0.5127978324890137, + 0.48720216751098633 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928870677947998, + "mean_weights": [ + 0.5100737810134888, + 0.48992621898651123 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0981076955795288, + "mean_weights": [ + 0.33567601442337036, + 0.3208814263343811, + 0.3434425890445709 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985171794891357, + "mean_weights": [ + 0.3379390239715576, + 0.33097290992736816, + 0.33108803629875183 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984209775924683, + "mean_weights": [ + 0.3407840430736542, + 0.3312426805496216, + 0.32797327637672424 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985374450683594, + "mean_weights": [ + 0.3372366726398468, + 0.3311164975166321, + 0.33164680004119873 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0983848571777344, + "mean_weights": [ + 0.34108036756515503, + 0.33306455612182617, + 0.325855016708374 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984746217727661, + "mean_weights": [ + 0.33782216906547546, + 0.33313143253326416, + 0.32904642820358276 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982099771499634, + "mean_weights": [ + 0.3427715599536896, + 0.33253389596939087, + 0.32469451427459717 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985102653503418, + "mean_weights": [ + 0.33662939071655273, + 0.33529555797576904, + 0.32807502150535583 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3860622644424438, + "mean_weights": [ + 0.25567787885665894, + 0.25280827283859253, + 0.24703410267829895, + 0.24447977542877197 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861664533615112, + "mean_weights": [ + 0.25407174229621887, + 0.2527073621749878, + 0.24657712876796722, + 0.24664375185966492 + ], + "sources": 4 + }, + { + "entropy_mean": 1.385995864868164, + "mean_weights": [ + 0.25519782304763794, + 0.2541751265525818, + 0.24830767512321472, + 0.24231936037540436 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862323760986328, + "mean_weights": [ + 0.2518538534641266, + 0.25246989727020264, + 0.24845747649669647, + 0.2472187578678131 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859524726867676, + "mean_weights": [ + 0.2546083331108093, + 0.2549782693386078, + 0.24969200789928436, + 0.24072137475013733 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861347436904907, + "mean_weights": [ + 0.25202351808547974, + 0.25434941053390503, + 0.24928084015846252, + 0.24434620141983032 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859026432037354, + "mean_weights": [ + 0.254156231880188, + 0.25646376609802246, + 0.24989977478981018, + 0.23948019742965698 + ], + "sources": 4 + }, + { + "entropy_mean": 1.386173129081726, + "mean_weights": [ + 0.2492857575416565, + 0.2551034390926361, + 0.250427782535553, + 0.24518299102783203 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6091642379760742, + "mean_weights": [ + 0.2029721438884735, + 0.20569172501564026, + 0.20075879991054535, + 0.19489683210849762, + 0.19568051397800446 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093814373016357, + "mean_weights": [ + 0.19996729493141174, + 0.20290382206439972, + 0.1990462839603424, + 0.1980418562889099, + 0.20004072785377502 + ], + "sources": 5 + }, + { + "entropy_mean": 1.608886957168579, + "mean_weights": [ + 0.2043994665145874, + 0.20701739192008972, + 0.20311541855335236, + 0.19478896260261536, + 0.19067879021167755 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093982458114624, + "mean_weights": [ + 0.2001650035381317, + 0.2025347352027893, + 0.20030495524406433, + 0.19929912686347961, + 0.19769617915153503 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085150241851807, + "mean_weights": [ + 0.20291879773139954, + 0.21057289838790894, + 0.20388361811637878, + 0.1945766806602478, + 0.18804802000522614 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609283208847046, + "mean_weights": [ + 0.20027728378772736, + 0.20463380217552185, + 0.20175579190254211, + 0.1985601782798767, + 0.19477295875549316 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085457801818848, + "mean_weights": [ + 0.20302599668502808, + 0.21113203465938568, + 0.20331484079360962, + 0.19478978216648102, + 0.187737375497818 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093010902404785, + "mean_weights": [ + 0.2015986442565918, + 0.2040453553199768, + 0.20027345418930054, + 0.19875933229923248, + 0.19532325863838196 + ], + "sources": 5 + }, + { + "entropy_mean": 1.790800929069519, + "mean_weights": [ + 0.1695481389760971, + 0.17701643705368042, + 0.1705975979566574, + 0.16259069740772247, + 0.15733328461647034, + 0.16291385889053345 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791712760925293, + "mean_weights": [ + 0.1673981100320816, + 0.1689898520708084, + 0.16775551438331604, + 0.16575247049331665, + 0.16473346948623657, + 0.16537058353424072 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7902085781097412, + "mean_weights": [ + 0.17062905430793762, + 0.1804160624742508, + 0.17242339253425598, + 0.16423535346984863, + 0.15749101340770721, + 0.15480509400367737 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917354106903076, + "mean_weights": [ + 0.16661706566810608, + 0.1685091108083725, + 0.16663718223571777, + 0.16635015606880188, + 0.1662556231021881, + 0.16563087701797485 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7898712158203125, + "mean_weights": [ + 0.17190133035182953, + 0.18024806678295135, + 0.17467562854290009, + 0.16425392031669617, + 0.15632842481136322, + 0.15259261429309845 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791717529296875, + "mean_weights": [ + 0.1659175157546997, + 0.16852334141731262, + 0.16692936420440674, + 0.16678501665592194, + 0.16574949026107788, + 0.16609525680541992 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7883853912353516, + "mean_weights": [ + 0.17291955649852753, + 0.18508964776992798, + 0.17579255998134613, + 0.1639368087053299, + 0.15379686653614044, + 0.14846453070640564 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917416095733643, + "mean_weights": [ + 0.16680282354354858, + 0.16815027594566345, + 0.1662275195121765, + 0.16708579659461975, + 0.16612832248210907, + 0.16560523211956024 + ], + "sources": 6 + }, + { + "entropy_mean": 1.942642092704773, + "mean_weights": [ + 0.14893436431884766, + 0.15992742776870728, + 0.1534634679555893, + 0.14322210848331451, + 0.1347082406282425, + 0.12977547943592072, + 0.12996888160705566 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458978176116943, + "mean_weights": [ + 0.1420164704322815, + 0.1434592306613922, + 0.1428285390138626, + 0.14312198758125305, + 0.1427583545446396, + 0.14280298352241516, + 0.1430124193429947 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9415806531906128, + "mean_weights": [ + 0.1496899276971817, + 0.16303203999996185, + 0.15404042601585388, + 0.14395901560783386, + 0.1344446837902069, + 0.12874644994735718, + 0.126087486743927 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458822011947632, + "mean_weights": [ + 0.14301417768001556, + 0.14306408166885376, + 0.14219683408737183, + 0.1414903700351715, + 0.14211279153823853, + 0.14377698302268982, + 0.1443447768688202 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9424853324890137, + "mean_weights": [ + 0.14929699897766113, + 0.16034753620624542, + 0.15305374562740326, + 0.14401423931121826, + 0.1357293725013733, + 0.13007207214832306, + 0.12748602032661438 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458880424499512, + "mean_weights": [ + 0.1432173103094101, + 0.1439918577671051, + 0.14288677275180817, + 0.1420578956604004, + 0.14131489396095276, + 0.14336320757865906, + 0.14316806197166443 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9396699666976929, + "mean_weights": [ + 0.15016242861747742, + 0.16698579490184784, + 0.15582789480686188, + 0.14568084478378296, + 0.1339108645915985, + 0.12636634707450867, + 0.12106583267450333 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0792927742004395, + "mean_weights": [ + 0.12389281392097473, + 0.12217633426189423, + 0.12296149879693985, + 0.12415584921836853, + 0.12564301490783691, + 0.12552481889724731, + 0.1266217827796936, + 0.12902389466762543 + ], + "sources": 8 + }, + { + "entropy_mean": 2.189755439758301, + "mean_weights": [ + 0.12039311230182648, + 0.1345410943031311, + 0.12449826300144196, + 0.11651842296123505, + 0.1091947853565216, + 0.10507211089134216, + 0.09920457750558853, + 0.09485669434070587, + 0.09572093933820724 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197061061859131, + "mean_weights": [ + 0.109954334795475, + 0.10848210752010345, + 0.10946077108383179, + 0.10999251902103424, + 0.11067993938922882, + 0.1111636757850647, + 0.11218364536762238, + 0.11501897871494293, + 0.11306402087211609 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1907906532287598, + "mean_weights": [ + 0.11952801793813705, + 0.13219305872917175, + 0.12340351939201355, + 0.11666150391101837, + 0.11039217561483383, + 0.1056135818362236, + 0.10048577934503555, + 0.09683912992477417, + 0.09488324075937271 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1971700191497803, + "mean_weights": [ + 0.11098132282495499, + 0.11007764935493469, + 0.10996879637241364, + 0.10998346656560898, + 0.10995160788297653, + 0.11261789500713348, + 0.11266392469406128, + 0.11191104352474213, + 0.11184429377317429 + ], + "sources": 9 + }, + { + "entropy_mean": 2.190371513366699, + "mean_weights": [ + 0.11987325549125671, + 0.13343080878257751, + 0.1237730160355568, + 0.11612145602703094, + 0.10980556905269623, + 0.10602204501628876, + 0.10080036520957947, + 0.09580431878566742, + 0.09436915814876556 + ], + "sources": 9 + }, + { + "entropy_mean": 2.196929454803467, + "mean_weights": [ + 0.10958486795425415, + 0.10723607242107391, + 0.10892888903617859, + 0.10961727052927017, + 0.11029283702373505, + 0.11106503009796143, + 0.11383815109729767, + 0.11435244977474213, + 0.1150844395160675 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1888015270233154, + "mean_weights": [ + 0.12048850953578949, + 0.13505060970783234, + 0.12494432926177979, + 0.11720074713230133, + 0.110526904463768, + 0.10586223006248474, + 0.09970010817050934, + 0.09478772431612015, + 0.09143882989883423 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028617706149816513, + 0.0149559136480093, + 0.015275427140295506, + 0.014781143516302109, + 0.01579420082271099, + 0.015445198863744736, + 0.01659693568944931, + 0.016700318083167076, + 0.017836440354585648, + 0.011791056022047997, + 0.01289808563888073, + 0.012997820973396301, + 0.014001362025737762, + 0.01415447611361742, + 0.01528822910040617, + 0.01543514709919691, + 0.016476599499583244, + 0.012522879056632519, + 0.013484617695212364, + 0.013611003756523132, + 0.014504051767289639, + 0.014686089009046555, + 0.015652155503630638, + 0.015790550038218498, + 0.01694054901599884, + 0.013788674026727676, + 0.014674022793769836, + 0.014920410700142384, + 0.01580130122601986, + 0.016082806512713432, + 0.01699301227927208, + 0.0172477588057518, + 0.018260842189192772, + 0.015473411418497562, + 0.016316166147589684, + 0.016675656661391258, + 0.01762448064982891, + 0.01778457872569561, + 0.018877588212490082, + 0.018956352025270462, + 0.020222412422299385, + 0.017603235319256783, + 0.018591728061437607, + 0.01862557791173458, + 0.019801458343863487, + 0.019871672615408897, + 0.021144596859812737, + 0.020878318697214127, + 0.022527050226926804, + 0.02028649114072323, + 0.021051261574029922, + 0.02177361585199833, + 0.02254091389477253, + 0.023530568927526474, + 0.02428813837468624, + 0.025259727612137794, + 0.026222435757517815, + 0.02289707586169243, + 0.024971475824713707, + 0.024427343159914017, + 0.02649919129908085, + 0.025686809793114662, + 0.028463052585721016, + 0.026970377191901207 + ], + "loss_nats": 3.592794418334961, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.192328929901123, + "mean_weights": [ + 0.09992179274559021, + 0.09311120957136154, + 0.10042804479598999, + 0.10806103050708771, + 0.11519140005111694, + 0.11805247515439987, + 0.1209874153137207, + 0.12242934107780457, + 0.12181729078292847 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.0027192807756364346, + 0.005520354490727186, + 0.006244984921067953, + 0.009551535360515118, + 0.00999026745557785, + 0.013990547508001328, + 0.014324866235256195, + 0.01876879669725895, + 0.002794948872178793, + 0.006623269058763981, + 0.007186788599938154, + 0.011789736337959766, + 0.012359116226434708, + 0.017194580286741257, + 0.01774122752249241, + 0.02232358604669571, + 0.0035717571154236794, + 0.0074646128341555595, + 0.008665312081575394, + 0.013205114752054214, + 0.014134838245809078, + 0.019117774441838264, + 0.02030739188194275, + 0.025769757106900215, + 0.004883658606559038, + 0.008969931863248348, + 0.011208492331206799, + 0.016233347356319427, + 0.01854560896754265, + 0.023725751787424088, + 0.026544949039816856, + 0.032117463648319244, + 0.005602571647614241, + 0.009894398041069508, + 0.013632542453706264, + 0.01924053207039833, + 0.022953463718295097, + 0.028615590184926987, + 0.03298711031675339, + 0.038781892508268356, + 0.00841959286481142, + 0.012692742049694061, + 0.017108004540205002, + 0.022670142352581024, + 0.0290161594748497, + 0.03462707996368408, + 0.0402880497276783, + 0.045925430953502655, + 0.009543929249048233, + 0.014747132547199726, + 0.02326968125998974, + 0.029425622895359993, + 0.039871443063020706, + 0.04612904414534569, + 0.056299544870853424, + 0.062327586114406586, + 0.014510786160826683, + 0.019631674513220787, + 0.03176560997962952, + 0.037373341619968414, + 0.048608578741550446, + 0.05553118884563446, + 0.06796237081289291, + 0.07465041428804398 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00015037160483188927, + 0.00015531893586739898, + 0.00014888793521095067, + 0.00014173146337270737, + 0.00014074062346480787, + 0.0001267959305550903, + 0.00011449279554653913, + 0.0001056772016454488, + 0.0001005215963232331, + 9.220124775310978e-05, + 8.474739297525957e-05, + 7.880839984863997e-05, + 7.79260735725984e-05, + 7.202131382655352e-05, + 6.493543332908303e-05, + 6.0076497902628034e-05, + 5.988969132886268e-05, + 5.5038544815033674e-05, + 4.9646685511106625e-05, + 4.5450724428519607e-05, + 4.351665484136902e-05, + 3.985156217822805e-05, + 3.7343925214372575e-05, + 3.5149059840478e-05, + 3.101725087617524e-05, + 2.8898630262119696e-05, + 2.7442893042461947e-05, + 2.673451126611326e-05, + 2.752919499471318e-05, + 2.6972027626470663e-05, + 2.6814161174115725e-05, + 2.6812700525624678e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00013550206131185405, + "first_to_last_ratio": 4.878092919810336, + "imbalance_abs_log_ratio": 1.5847543483639097, + "last_quartile_mean": 2.7777671220974298e-05, + "mean": 7.19800832484907e-05, + "normalized": [ + 2.089072393995075, + 2.1578043377805587, + 2.0684601696966305, + 1.9690372249698562, + 1.9552717517558436, + 1.7615418714835815, + 1.590617715059948, + 1.4681450322949532, + 1.3965195896788694, + 1.280927217530595, + 1.1773728113469024, + 1.0948639719764766, + 1.0826060495593046, + 1.000572805367847, + 0.9021305672141552, + 0.8346266799279888, + 0.8320314262781637, + 0.7646357482670422, + 0.6897280924185043, + 0.6314347299601467, + 0.6045652196752841, + 0.5536470698519742, + 0.5188091417656927, + 0.48831646553027597, + 0.4309143512532075, + 0.40148092302637967, + 0.38125675609075343, + 0.3714153979764095, + 0.38245572597737193, + 0.3747151490969722, + 0.3725219527955738, + 0.37250166039766136 + ], + "population_cv": 0.5908844388151021 + }, + "activation_output_rms_by_block": [ + 0.021565889939665794, + 0.036469344049692154, + 0.04872849956154823, + 0.05858244374394417, + 0.017551003023982048, + 0.025556011125445366, + 0.033311840146780014, + 0.03929486125707626, + 0.022112373262643814, + 0.028847798705101013, + 0.03594932332634926, + 0.0407695509493351, + 0.021146347746253014, + 0.02899041399359703, + 0.039105549454689026, + 0.04533020406961441, + 0.021752145141363144, + 0.039439886808395386, + 0.04092313349246979, + 0.05069950595498085, + 0.027180669829249382, + 0.04007871076464653, + 0.05619155615568161, + 0.07526125758886337, + 0.023234715685248375, + 0.04547948017716408, + 0.06462902575731277, + 0.08550379425287247, + 0.02780340425670147, + 0.05829066410660744, + 0.09081556648015976, + 0.12836997210979462 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.035132486606016755, + "first_to_last_ratio": 0.5362442597034707, + "imbalance_abs_log_ratio": 0.6231655133128765, + "last_quartile_mean": 0.06551582785323262, + "mean": 0.04434265446616337, + "normalized": [ + 0.48634639038405153, + 0.8224438633352653, + 1.098908041211912, + 1.3211307362901963, + 0.3958040680080333, + 0.5763302047004525, + 0.7512369421230599, + 0.8861639369618945, + 0.49867049072394043, + 0.6505654443198469, + 0.8107165382663588, + 0.9194206219757366, + 0.47688502190118537, + 0.6537816543147817, + 0.8818946435543089, + 1.0222708724892555, + 0.4905467524043152, + 0.8894345023591417, + 0.9228841616529088, + 1.143357486495722, + 0.6129689382937187, + 0.903841036292256, + 1.2672122774823913, + 1.6972654996622523, + 0.5239811636215449, + 1.0256372949406578, + 1.457491134334986, + 1.928251596171763, + 0.6270126268132519, + 1.3145506241870883, + 2.0480408214952166, + 2.894954613232506 + ], + "population_cv": 0.5330271496860788 + }, + "bits_per_byte": 3.0765498301078438, + "branch_output_rms_by_sublayer": [ + 0.010663045570254326, + 0.017228109762072563, + 0.013754382729530334, + 0.012785768136382103, + 0.013033404015004635, + 0.013332822360098362, + 0.00970276165753603, + 0.012643634341657162, + 0.014693930745124817, + 0.008590146899223328, + 0.01348587591201067, + 0.008184721693396568, + 0.013327986001968384, + 0.007770082913339138, + 0.012535307556390762, + 0.00807273294776678, + 0.019638333469629288, + 0.008253859356045723, + 0.023146430030465126, + 0.008218511007726192, + 0.022952934727072716, + 0.007821362465620041, + 0.02555258758366108, + 0.008400500752031803, + 0.018209965899586678, + 0.009391991421580315, + 0.015834715217351913, + 0.00904791709035635, + 0.01481566857546568, + 0.009485187940299511, + 0.013663850724697113, + 0.009231415577232838, + 0.018229452893137932, + 0.010688192211091518, + 0.0216380525380373, + 0.010875235311686993, + 0.024837728589773178, + 0.012454677373170853, + 0.018861932680010796, + 0.014204870909452438, + 0.022439414635300636, + 0.014683135785162449, + 0.016518225893378258, + 0.017171991989016533, + 0.02124573290348053, + 0.018020784482359886, + 0.02404467575252056, + 0.022387230768799782, + 0.01352726761251688, + 0.017979588359594345, + 0.018088674172759056, + 0.019181283190846443, + 0.011791687458753586, + 0.02067566104233265, + 0.006281886249780655, + 0.023434488102793694, + 0.006251647602766752, + 0.025439132004976273, + 0.009531536139547825, + 0.027843881398439407, + 0.005653195548802614, + 0.03192751482129097, + 0.01025710441172123, + 0.03521641343832016 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00047066798455832134, + 0.0004001648516679536, + 0.0003967067689622585, + 0.00036485465053122515, + 0.0003303298047487693, + 0.0002735257484091121, + 0.00024634005442437544, + 0.0002879726781847622, + 0.0002804833850065481, + 0.0002626314219081706, + 0.00022876266715393713, + 0.00021079944113804996, + 0.00020413883022779155, + 0.00018426171309691338, + 0.00016773126738824047, + 0.00014734109770641942, + 0.00015413415329831883, + 0.00014308429188069618, + 0.00013179860681704594, + 0.00012162232045861975, + 0.00012548041696308594, + 0.00011568775264515247, + 0.00011700045543982305, + 0.00011911219021127675, + 9.889119725601648e-05, + 9.601002508252603e-05, + 8.844258169210919e-05, + 7.726016990326742e-05, + 8.587977495744223e-05, + 8.78708730367275e-05, + 8.741786084116789e-05, + 9.286267453972122e-05 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0003463203176858472, + "first_to_last_ratio": 3.8768909046114897, + "imbalance_abs_log_ratio": 1.355033519167651, + "last_quartile_mean": 8.932939466362224e-05, + "mean": 0.00019372711594174515, + "normalized": [ + 2.4295410700268407, + 2.0656109482798755, + 2.0477606711574157, + 1.8833432209920424, + 1.705129419508322, + 1.411912561024048, + 1.2715827272133204, + 1.4864861678494055, + 1.4478271853842652, + 1.3556771370464493, + 1.1808500118420562, + 1.0881256354502205, + 1.0537442279849831, + 0.9511405370444989, + 0.8658120293221018, + 0.7605600124183222, + 0.7956250860858728, + 0.7385868064216805, + 0.6803312286787903, + 0.6278022561136577, + 0.6477173644644492, + 0.5971686105105721, + 0.6039446510678751, + 0.6148452147870433, + 0.5104664712282901, + 0.49559414858267337, + 0.45653176253707495, + 0.398809271111536, + 0.4433028104504832, + 0.4535806596282101, + 0.45124225597543577, + 0.47934783981219004 + ], + "population_cv": 0.5521724357324023 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6400090456008911, + "mean_weights": [ + 0.36098042130470276, + 0.6390195488929749 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6791766285896301, + "mean_weights": [ + 0.4528862237930298, + 0.5471137762069702 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6883224248886108, + "mean_weights": [ + 0.47658076882362366, + 0.523419201374054 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6801540851593018, + "mean_weights": [ + 0.4740057587623596, + 0.5259942412376404 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6770029067993164, + "mean_weights": [ + 0.4345158338546753, + 0.5654841661453247 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6838035583496094, + "mean_weights": [ + 0.4726506471633911, + 0.5273493528366089 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6871338486671448, + "mean_weights": [ + 0.5084361433982849, + 0.4915638566017151 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6837936639785767, + "mean_weights": [ + 0.46237531304359436, + 0.5376246571540833 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0874912738800049, + "mean_weights": [ + 0.2803909182548523, + 0.3660100996494293, + 0.3535989820957184 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0815104246139526, + "mean_weights": [ + 0.29584482312202454, + 0.32035398483276367, + 0.3838011622428894 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0888757705688477, + "mean_weights": [ + 0.28722521662712097, + 0.3439940810203552, + 0.3687807023525238 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0712084770202637, + "mean_weights": [ + 0.2902437746524811, + 0.3058721423149109, + 0.40388408303260803 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0888686180114746, + "mean_weights": [ + 0.2993178367614746, + 0.3209303915500641, + 0.3797518014907837 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0681074857711792, + "mean_weights": [ + 0.29420867562294006, + 0.30017125606536865, + 0.40562009811401367 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0807197093963623, + "mean_weights": [ + 0.27623888850212097, + 0.33169907331466675, + 0.3920620083808899 + ], + "sources": 3 + }, + { + "entropy_mean": 1.049599289894104, + "mean_weights": [ + 0.27415981888771057, + 0.29408201575279236, + 0.43175816535949707 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3794260025024414, + "mean_weights": [ + 0.22683823108673096, + 0.24667194485664368, + 0.2656657099723816, + 0.2608241140842438 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3584692478179932, + "mean_weights": [ + 0.23203331232070923, + 0.21317778527736664, + 0.2513449192047119, + 0.3034440279006958 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3821847438812256, + "mean_weights": [ + 0.2378484606742859, + 0.24123439192771912, + 0.25613898038864136, + 0.264778196811676 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3464672565460205, + "mean_weights": [ + 0.23562151193618774, + 0.2193756252527237, + 0.22607606649398804, + 0.3189268112182617 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3816092014312744, + "mean_weights": [ + 0.24845418334007263, + 0.23746336996555328, + 0.24082297086715698, + 0.2732594907283783 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3438568115234375, + "mean_weights": [ + 0.2435237169265747, + 0.21819522976875305, + 0.2184712439775467, + 0.31980982422828674 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3809527158737183, + "mean_weights": [ + 0.24657869338989258, + 0.23760980367660522, + 0.2439434975385666, + 0.2718679904937744 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3471378087997437, + "mean_weights": [ + 0.25225478410720825, + 0.2180073857307434, + 0.21758286654949188, + 0.31215494871139526 + ], + "sources": 4 + }, + { + "entropy_mean": 1.606877326965332, + "mean_weights": [ + 0.19576358795166016, + 0.19534793496131897, + 0.1907539963722229, + 0.19958166778087616, + 0.21855281293392181 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5734713077545166, + "mean_weights": [ + 0.19881275296211243, + 0.17522111535072327, + 0.16872526705265045, + 0.20819878578186035, + 0.2490420937538147 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6040270328521729, + "mean_weights": [ + 0.21292802691459656, + 0.1855008900165558, + 0.1801277995109558, + 0.19993627071380615, + 0.2215070277452469 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5779120922088623, + "mean_weights": [ + 0.2017330527305603, + 0.17727899551391602, + 0.16909101605415344, + 0.20081563293933868, + 0.25108128786087036 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5998265743255615, + "mean_weights": [ + 0.2036537528038025, + 0.18378914892673492, + 0.1785627007484436, + 0.19737926125526428, + 0.2366151213645935 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5833067893981934, + "mean_weights": [ + 0.21128389239311218, + 0.1774863600730896, + 0.17054985463619232, + 0.19523648917675018, + 0.2454434037208557 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6030279397964478, + "mean_weights": [ + 0.20433253049850464, + 0.19323119521141052, + 0.1791742891073227, + 0.19461140036582947, + 0.22865056991577148 + ], + "sources": 5 + }, + { + "entropy_mean": 1.583907127380371, + "mean_weights": [ + 0.20781221985816956, + 0.18192699551582336, + 0.16940940916538239, + 0.1936372071504593, + 0.2472141683101654 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7854300737380981, + "mean_weights": [ + 0.16390785574913025, + 0.1649724543094635, + 0.15174438059329987, + 0.1579718440771103, + 0.17442190647125244, + 0.18698154389858246 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7611892223358154, + "mean_weights": [ + 0.1716495156288147, + 0.14782291650772095, + 0.13925832509994507, + 0.14533402025699615, + 0.1777762770652771, + 0.21815894544124603 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7871568202972412, + "mean_weights": [ + 0.16264018416404724, + 0.1673935055732727, + 0.15543673932552338, + 0.1586773842573166, + 0.16756463050842285, + 0.18828755617141724 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7737398147583008, + "mean_weights": [ + 0.169534832239151, + 0.1531105488538742, + 0.14653174579143524, + 0.15359164774417877, + 0.1676221787929535, + 0.20960906147956848 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7823994159698486, + "mean_weights": [ + 0.15739159286022186, + 0.17121702432632446, + 0.15522488951683044, + 0.1574881672859192, + 0.16679887473583221, + 0.19187945127487183 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7769070863723755, + "mean_weights": [ + 0.17015579342842102, + 0.16008082032203674, + 0.15160809457302094, + 0.15131033957004547, + 0.15850737690925598, + 0.20833760499954224 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7791379690170288, + "mean_weights": [ + 0.15263843536376953, + 0.1744789183139801, + 0.1579395830631256, + 0.1559261679649353, + 0.1628972589969635, + 0.19611963629722595 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7784135341644287, + "mean_weights": [ + 0.17716139554977417, + 0.15954381227493286, + 0.15142494440078735, + 0.14819344878196716, + 0.15881526470184326, + 0.2048611342906952 + ], + "sources": 6 + }, + { + "entropy_mean": 1.938215732574463, + "mean_weights": [ + 0.12483463436365128, + 0.15879923105239868, + 0.14282163977622986, + 0.1369495987892151, + 0.1367420256137848, + 0.14910337328910828, + 0.15074948966503143 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9434072971343994, + "mean_weights": [ + 0.1451970338821411, + 0.14693152904510498, + 0.13774025440216064, + 0.13556107878684998, + 0.14056631922721863, + 0.14916568994522095, + 0.1448380947113037 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9345760345458984, + "mean_weights": [ + 0.12217546254396439, + 0.16222885251045227, + 0.14348426461219788, + 0.13614335656166077, + 0.1331687867641449, + 0.1449238359928131, + 0.1578754484653473 + ], + "sources": 7 + }, + { + "entropy_mean": 1.942252278327942, + "mean_weights": [ + 0.14220868051052094, + 0.14608189463615417, + 0.140741765499115, + 0.13329893350601196, + 0.1350077986717224, + 0.14559787511825562, + 0.15706303715705872 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9367260932922363, + "mean_weights": [ + 0.1235148161649704, + 0.16515511274337769, + 0.1464528739452362, + 0.13824108242988586, + 0.13917279243469238, + 0.1463068723678589, + 0.14115647971630096 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9425405263900757, + "mean_weights": [ + 0.14511889219284058, + 0.15030623972415924, + 0.14017556607723236, + 0.13268452882766724, + 0.13446278870105743, + 0.14467889070510864, + 0.1525731086730957 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9243848323822021, + "mean_weights": [ + 0.11108033359050751, + 0.17919227480888367, + 0.1615527868270874, + 0.1391662061214447, + 0.1394817978143692, + 0.13981659710407257, + 0.12971001863479614 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.072169303894043, + "mean_weights": [ + 0.11806559562683105, + 0.1389603316783905, + 0.13016337156295776, + 0.12208377569913864, + 0.12250401079654694, + 0.12966252863407135, + 0.12484695017337799, + 0.11371345818042755 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1281957626342773, + "mean_weights": [ + 0.08533816784620285, + 0.17273354530334473, + 0.1569756716489792, + 0.12265726923942566, + 0.11592777073383331, + 0.10392434895038605, + 0.09507645666599274, + 0.07588343322277069, + 0.071483314037323 + ], + "sources": 9 + }, + { + "entropy_mean": 2.171718120574951, + "mean_weights": [ + 0.10228429734706879, + 0.13677777349948883, + 0.12091431021690369, + 0.11207222938537598, + 0.11428679525852203, + 0.12278461456298828, + 0.10960376262664795, + 0.08961573988199234, + 0.09166047722101212 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1293692588806152, + "mean_weights": [ + 0.08556902408599854, + 0.1737067550420761, + 0.15783612430095673, + 0.12235160171985626, + 0.11228232085704803, + 0.10041844844818115, + 0.09816212207078934, + 0.07761401683092117, + 0.07205958664417267 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1861257553100586, + "mean_weights": [ + 0.10352809727191925, + 0.1306774914264679, + 0.12128166854381561, + 0.11306338012218475, + 0.11160679161548615, + 0.11273257434368134, + 0.11206908524036407, + 0.09857702255249023, + 0.09646390378475189 + ], + "sources": 9 + }, + { + "entropy_mean": 2.118931531906128, + "mean_weights": [ + 0.08414841443300247, + 0.17577886581420898, + 0.1621742993593216, + 0.12455005943775177, + 0.11576595157384872, + 0.10289883613586426, + 0.0956137552857399, + 0.07483348250389099, + 0.0642363429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1879653930664062, + "mean_weights": [ + 0.1115165501832962, + 0.11579209566116333, + 0.10660408437252045, + 0.10799701511859894, + 0.11241275817155838, + 0.12142005562782288, + 0.11671245098114014, + 0.10222253948450089, + 0.1053224429488182 + ], + "sources": 9 + }, + { + "entropy_mean": 2.113354444503784, + "mean_weights": [ + 0.08345066010951996, + 0.17587825655937195, + 0.16778676211833954, + 0.12578019499778748, + 0.11801718175411224, + 0.10158660262823105, + 0.09391272068023682, + 0.07420934736728668, + 0.059378258883953094 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.04385647177696228, + 0.016716627404093742, + 0.023853398859500885, + 0.02631799690425396, + 0.029633011668920517, + 0.030924035236239433, + 0.034180350601673126, + 0.03414159640669823, + 0.03801807761192322, + 0.025688115507364273, + 0.02659021131694317, + 0.026976166293025017, + 0.028039155527949333, + 0.028344007208943367, + 0.03002496063709259, + 0.03041207231581211, + 0.03166870027780533, + 0.02362353727221489, + 0.024404184892773628, + 0.02398102730512619, + 0.025770720094442368, + 0.02497660554945469, + 0.02696264162659645, + 0.025439584627747536, + 0.027456484735012054, + 0.02067279815673828, + 0.022224770858883858, + 0.021025508642196655, + 0.02260744944214821, + 0.021831434220075607, + 0.02316143922507763, + 0.022327570244669914, + 0.02387402206659317, + 0.018859945237636566, + 0.019507037475705147, + 0.0193997360765934, + 0.020457888022065163, + 0.020163770765066147, + 0.020851148292422295, + 0.02107001654803753, + 0.022065866738557816, + 0.018605999648571014, + 0.01921229064464569, + 0.01954682357609272, + 0.020514458417892456, + 0.021012376993894577, + 0.021912841126322746, + 0.022172000259160995, + 0.023965176194906235, + 0.02102574147284031, + 0.02225077897310257, + 0.02232026495039463, + 0.02367965318262577, + 0.023714035749435425, + 0.025225302204489708, + 0.025281865149736404, + 0.026469727978110313, + 0.022601939737796783, + 0.025214210152626038, + 0.02405940368771553, + 0.02765660174190998, + 0.025228707119822502, + 0.03158720210194588, + 0.02647186629474163 + ], + "loss_nats": 2.1325018405914307, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1645100116729736, + "mean_weights": [ + 0.12699784338474274, + 0.0868702158331871, + 0.08687461167573929, + 0.09149467945098877, + 0.09256475418806076, + 0.10866943746805191, + 0.13657689094543457, + 0.13026893138885498, + 0.13968263566493988 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.010663045570254326, + 0.021565889939665794, + 0.028368694707751274, + 0.036469344049692154, + 0.04090480878949165, + 0.04872849956154823, + 0.050470490008592606, + 0.05858244374394417, + 0.014693930745124817, + 0.017551003023982048, + 0.02190772444009781, + 0.025556011125445366, + 0.02960398606956005, + 0.033311840146780014, + 0.03541981056332588, + 0.03929486125707626, + 0.019638333469629288, + 0.022112373262643814, + 0.02556230127811432, + 0.028847798705101013, + 0.03241894394159317, + 0.03594932332634926, + 0.03664650768041611, + 0.0407695509493351, + 0.018209965899586678, + 0.021146347746253014, + 0.024955688044428825, + 0.02899041399359703, + 0.034678488969802856, + 0.039105549454689026, + 0.04065592959523201, + 0.04533020406961441, + 0.018229452893137932, + 0.021752145141363144, + 0.034592531621456146, + 0.039439886808395386, + 0.033741310238838196, + 0.04092313349246979, + 0.04186084121465683, + 0.05069950595498085, + 0.022439414635300636, + 0.027180669829249382, + 0.029904387891292572, + 0.04007871076464653, + 0.04410696402192116, + 0.05619155615568161, + 0.057447679340839386, + 0.07526125758886337, + 0.01352726761251688, + 0.023234715685248375, + 0.03211096674203873, + 0.04547948017716408, + 0.048427797853946686, + 0.06462902575731277, + 0.06638108938932419, + 0.08550379425287247, + 0.006251647602766752, + 0.02780340425670147, + 0.03265053778886795, + 0.05829066410660744, + 0.06080053001642227, + 0.09081556648015976, + 0.09526639431715012, + 0.12836997210979462 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.785148045513779e-05, + 8.212970715248957e-05, + 7.901267963461578e-05, + 7.481906504835933e-05, + 8.32517325761728e-05, + 8.461637480650097e-05, + 7.809923408785835e-05, + 7.281435682671145e-05, + 7.344305777223781e-05, + 6.877864507259801e-05, + 6.46581975161098e-05, + 6.150812259875238e-05, + 5.450130993267521e-05, + 5.359771239454858e-05, + 5.45672737644054e-05, + 5.408914512372576e-05, + 7.829322566976771e-05, + 7.41191761335358e-05, + 6.719526572851464e-05, + 6.201949145179242e-05, + 8.774759771768004e-05, + 7.601781544508412e-05, + 6.271351594477892e-05, + 5.1511389756342396e-05, + 3.485357228782959e-05, + 3.2797412131913006e-05, + 3.119218672509305e-05, + 3.0088418498053215e-05, + 3.53662071574945e-05, + 3.247218410251662e-05, + 3.1223469704855233e-05, + 3.090241443715058e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 7.907432882348076e-05, + "first_to_last_ratio": 2.443432731063981, + "imbalance_abs_log_ratio": 0.8934039076024394, + "last_quartile_mean": 3.2361983130613226e-05, + "mean": 6.050785742672815e-05, + "normalized": [ + 1.2866342284456505, + 1.357339536471678, + 1.3058251109006132, + 1.236518168552924, + 1.375883002913899, + 1.398436143751528, + 1.2907287980314364, + 1.2033867983986348, + 1.2137772001127871, + 1.136689481293985, + 1.0685917542925312, + 1.016531161646824, + 0.9007311157674596, + 0.8857975587625558, + 0.9018212854501337, + 0.8939193589729215, + 1.2939348540737325, + 1.2249512589879794, + 1.1105213204728757, + 1.0249824417745212, + 1.4501851734535103, + 1.256329651684291, + 1.0364524313345866, + 0.8513173651656728, + 0.5760172937875951, + 0.5420355888758572, + 0.515506382999351, + 0.49726464921500013, + 0.5844894970925243, + 0.536660617041989, + 0.5160233898988279, + 0.5107173803761237 + ], + "population_cv": 0.3101401117414039 + }, + "activation_output_rms_by_block": [ + 0.08920358121395111, + 0.16476230323314667, + 0.20172803103923798, + 0.2172839194536209, + 0.10651351511478424, + 0.1277865320444107, + 0.1365494430065155, + 0.14577952027320862, + 0.041685812175273895, + 0.06442590802907944, + 0.09198766946792603, + 0.12648026645183563, + 0.09409327805042267, + 0.127781480550766, + 0.1514836698770523, + 0.17990845441818237, + 0.04308130219578743, + 0.06693905591964722, + 0.09104157984256744, + 0.1198044866323471, + 0.052153728902339935, + 0.07772442698478699, + 0.10783806443214417, + 0.14083683490753174, + 0.06691420078277588, + 0.10727331042289734, + 0.14309169352054596, + 0.19026093184947968, + 0.06030842289328575, + 0.12592560052871704, + 0.19766274094581604, + 0.2932122051715851 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.14870085567235947, + "first_to_last_ratio": 1.0041849854426779, + "imbalance_abs_log_ratio": 0.004176252746758971, + "last_quartile_mean": 0.14808113826438785, + "mean": 0.12348506157286465, + "normalized": [ + 0.7223835829025755, + 1.3342691102431496, + 1.6336229538194351, + 1.7595968021233768, + 0.8625619468305805, + 1.0348339338925454, + 1.1057972621728174, + 1.1805437711766351, + 0.3375777737348125, + 0.5217303794359266, + 0.7449295347651991, + 1.0242556050166733, + 0.7619810595057377, + 1.0347930261618419, + 1.2267368048212581, + 1.4569248468327811, + 0.34887865501339616, + 0.5420822168044074, + 0.7372679632900103, + 0.9701941684796768, + 0.4223484868375407, + 0.629423721337534, + 0.8732883399706799, + 1.140517185752289, + 0.5418809362887341, + 0.8687148798123953, + 1.1587773589610437, + 1.5407607157179306, + 0.48838638556858677, + 1.0197638396480233, + 1.6007016430014207, + 2.3744751100809856 + ], + "population_cv": 0.452413300189423 + }, + "bits_per_byte": 2.0788347888593717, + "branch_output_rms_by_sublayer": [ + 0.047890253365039825, + 0.0724048763513565, + 0.09392330050468445, + 0.019805217161774635, + 0.05155516415834427, + 0.009240997955203056, + 0.028290895745158195, + 0.029252246022224426, + 0.10515304654836655, + 0.009988512843847275, + 0.036857083439826965, + 0.007354213856160641, + 0.030265124514698982, + 0.009024721570312977, + 0.02873866818845272, + 0.020077969878911972, + 0.03568773716688156, + 0.019331395626068115, + 0.02450737915933132, + 0.02059691585600376, + 0.027077239006757736, + 0.02594342641532421, + 0.03273917734622955, + 0.029292359948158264, + 0.0856640487909317, + 0.025452638044953346, + 0.03311077505350113, + 0.02464992366731167, + 0.027853012084960938, + 0.025597043335437775, + 0.03691839426755905, + 0.024950893595814705, + 0.030026134103536606, + 0.02893473021686077, + 0.027840863913297653, + 0.028623709455132484, + 0.027078833431005478, + 0.034121230244636536, + 0.03150705620646477, + 0.04500562697649002, + 0.03698531538248062, + 0.034060779958963394, + 0.02739582024514675, + 0.03838665410876274, + 0.03699810802936554, + 0.04473178833723068, + 0.033644624054431915, + 0.054883986711502075, + 0.046741727739572525, + 0.04381329193711281, + 0.03934064880013466, + 0.046058837324380875, + 0.02425607293844223, + 0.05279749259352684, + 0.015374752692878246, + 0.07216396182775497, + 0.008622880093753338, + 0.059265173971652985, + 0.015607454814016819, + 0.07468733191490173, + 0.011398044414818287, + 0.08213626593351364, + 0.01687823049724102, + 0.11092311888933182 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0003197060052174766, + 0.00021473223069935944, + 0.00017000886488276594, + 0.00017587333645869598, + 0.00021844314727993092, + 0.00016279054132793154, + 0.00013368946738515455, + 0.00015507897052701879, + 0.00018937237754716193, + 0.00015807835440895333, + 0.00017637654240752466, + 0.00017942203797419035, + 0.00017938881894206114, + 0.00014125283909831378, + 0.00015643578933596588, + 0.0001753852603601794, + 0.0002336618336466446, + 0.0002565472818563129, + 0.00026974379220496994, + 0.0002930026868270223, + 0.0004028130704909499, + 0.00034545547323086546, + 0.00032351554793780203, + 0.0002907071322902779, + 0.0001595749338292262, + 0.00014685312709414154, + 0.00013462721955078127, + 0.00014024204184014355, + 0.00016889189127934948, + 0.00016902131816815604, + 0.00015850707773914266, + 0.00016744873340863954 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00019379032047229172, + "first_to_last_ratio": 1.2450726544340212, + "imbalance_abs_log_ratio": 0.2191938851890863, + "last_quartile_mean": 0.00015564579286369754, + "mean": 0.00020520774203897218, + "normalized": [ + 1.5579626871814583, + 1.0464138855861413, + 0.828471982555505, + 0.8570502005002075, + 1.0644975920959412, + 0.7932962943328671, + 0.6514835456830124, + 0.7557169577820648, + 0.9228325192097144, + 0.7703332868354049, + 0.8595023786871939, + 0.8743434150750281, + 0.8741815350611497, + 0.6883406917049341, + 0.7623288857505983, + 0.8546717517454627, + 1.1386599322469448, + 1.2501832499457575, + 1.314491302934084, + 1.4278344662618843, + 1.962952598613211, + 1.6834426898243344, + 1.5765270097673085, + 1.4166479753725276, + 0.7776262837048342, + 0.715631513874617, + 0.6560533155967061, + 0.6834149649846514, + 0.8230288467735991, + 0.8236595583029037, + 0.7724225029923075, + 0.8159961790176434 + ], + "population_cv": 0.33713769042478564 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.3882020115852356, + "mean_weights": [ + 0.8551924228668213, + 0.1448076069355011 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6341165900230408, + "mean_weights": [ + 0.5762301683425903, + 0.42376989126205444 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5635824203491211, + "mean_weights": [ + 0.7281181812286377, + 0.2718817889690399 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6844813823699951, + "mean_weights": [ + 0.5232460498809814, + 0.47675397992134094 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6596623659133911, + "mean_weights": [ + 0.4570625126361847, + 0.5429375171661377 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6911442279815674, + "mean_weights": [ + 0.5052971839904785, + 0.4947027564048767 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5367048978805542, + "mean_weights": [ + 0.7607460021972656, + 0.239253968000412 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6864078044891357, + "mean_weights": [ + 0.5320897698402405, + 0.4679102599620819 + ], + "sources": 2 + }, + { + "entropy_mean": 1.073125958442688, + "mean_weights": [ + 0.3128740191459656, + 0.2952309250831604, + 0.3918951153755188 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0924503803253174, + "mean_weights": [ + 0.32895076274871826, + 0.3053523600101471, + 0.36569687724113464 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0264496803283691, + "mean_weights": [ + 0.28269219398498535, + 0.23089176416397095, + 0.4864160418510437 + ], + "sources": 3 + }, + { + "entropy_mean": 1.082716703414917, + "mean_weights": [ + 0.3374946117401123, + 0.26259681582450867, + 0.3999086022377014 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0150713920593262, + "mean_weights": [ + 0.30969497561454773, + 0.19829383492469788, + 0.4920112192630768 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0759131908416748, + "mean_weights": [ + 0.33729270100593567, + 0.2485201358795166, + 0.41418716311454773 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0080678462982178, + "mean_weights": [ + 0.29085874557495117, + 0.1991720348596573, + 0.5099692344665527 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0891945362091064, + "mean_weights": [ + 0.3700854182243347, + 0.27556413412094116, + 0.3543504476547241 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3539717197418213, + "mean_weights": [ + 0.2609966993331909, + 0.1719343066215515, + 0.2525833249092102, + 0.31448569893836975 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3790147304534912, + "mean_weights": [ + 0.2805997431278229, + 0.21738627552986145, + 0.2319161295890808, + 0.2700978219509125 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3694558143615723, + "mean_weights": [ + 0.2795584201812744, + 0.19193390011787415, + 0.24360007047653198, + 0.28490763902664185 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3723174333572388, + "mean_weights": [ + 0.29385632276535034, + 0.2308381199836731, + 0.20058517158031464, + 0.27472037076950073 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3431222438812256, + "mean_weights": [ + 0.31385740637779236, + 0.18845698237419128, + 0.19518887996673584, + 0.3024967312812805 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3757907152175903, + "mean_weights": [ + 0.27200162410736084, + 0.20978213846683502, + 0.23660796880722046, + 0.2816082835197449 + ], + "sources": 4 + }, + { + "entropy_mean": 1.366483449935913, + "mean_weights": [ + 0.2711932957172394, + 0.20919060707092285, + 0.21517416834831238, + 0.3044419288635254 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3804211616516113, + "mean_weights": [ + 0.2684120535850525, + 0.249923974275589, + 0.23494675755500793, + 0.24671721458435059 + ], + "sources": 4 + }, + { + "entropy_mean": 1.596557378768921, + "mean_weights": [ + 0.18878842890262604, + 0.16890361905097961, + 0.18919318914413452, + 0.24000577628612518, + 0.21310898661613464 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5932904481887817, + "mean_weights": [ + 0.19562360644340515, + 0.18921726942062378, + 0.1648714691400528, + 0.18924641609191895, + 0.26104122400283813 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5985426902770996, + "mean_weights": [ + 0.22194094955921173, + 0.16600216925144196, + 0.18969787657260895, + 0.2181929051876068, + 0.20416611433029175 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5594522953033447, + "mean_weights": [ + 0.20154830813407898, + 0.1818983256816864, + 0.12484689056873322, + 0.18505612015724182, + 0.3066503703594208 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5977442264556885, + "mean_weights": [ + 0.19407649338245392, + 0.16917790472507477, + 0.1952357292175293, + 0.23032459616661072, + 0.2111852765083313 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5482151508331299, + "mean_weights": [ + 0.21108156442642212, + 0.21527524292469025, + 0.11766847968101501, + 0.14397022128105164, + 0.3120044767856598 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5955827236175537, + "mean_weights": [ + 0.18916109204292297, + 0.1775345653295517, + 0.18831372261047363, + 0.2127368450164795, + 0.23225374519824982 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5828441381454468, + "mean_weights": [ + 0.21598833799362183, + 0.20098453760147095, + 0.1344081461429596, + 0.18050014972686768, + 0.26811882853507996 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7731740474700928, + "mean_weights": [ + 0.13722717761993408, + 0.1383817344903946, + 0.1777014583349228, + 0.1899302452802658, + 0.15364009141921997, + 0.20311929285526276 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7353047132492065, + "mean_weights": [ + 0.15500149130821228, + 0.16006824374198914, + 0.09331689029932022, + 0.13140541315078735, + 0.19034865498542786, + 0.26985931396484375 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7435860633850098, + "mean_weights": [ + 0.12751621007919312, + 0.12942630052566528, + 0.1636285036802292, + 0.18134593963623047, + 0.13289809226989746, + 0.2651849389076233 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7337449789047241, + "mean_weights": [ + 0.1558353304862976, + 0.16734814643859863, + 0.09454137086868286, + 0.12724721431732178, + 0.18138957023620605, + 0.27363836765289307 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7050988674163818, + "mean_weights": [ + 0.12081533670425415, + 0.12675267457962036, + 0.15281721949577332, + 0.16128483414649963, + 0.12638553977012634, + 0.3119444251060486 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7524232864379883, + "mean_weights": [ + 0.1637900173664093, + 0.1779833436012268, + 0.11183260381221771, + 0.12575608491897583, + 0.17063459753990173, + 0.2500033974647522 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6986379623413086, + "mean_weights": [ + 0.1223093792796135, + 0.12897223234176636, + 0.14953243732452393, + 0.17079956829547882, + 0.12046270817518234, + 0.30792367458343506 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7519237995147705, + "mean_weights": [ + 0.17560675740242004, + 0.1863025426864624, + 0.0940234512090683, + 0.1278226673603058, + 0.20717735588550568, + 0.209067240357399 + ], + "sources": 6 + }, + { + "entropy_mean": 1.868872046470642, + "mean_weights": [ + 0.09987302124500275, + 0.11196130514144897, + 0.11685784161090851, + 0.13237999379634857, + 0.0951952189207077, + 0.2042895406484604, + 0.2394430786371231 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8924400806427002, + "mean_weights": [ + 0.1354261040687561, + 0.15077045559883118, + 0.09174783527851105, + 0.09931328147649765, + 0.14462494850158691, + 0.13262826204299927, + 0.24548910558223724 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7952430248260498, + "mean_weights": [ + 0.09434239566326141, + 0.10134626924991608, + 0.10413748770952225, + 0.11577381938695908, + 0.08764459192752838, + 0.173546701669693, + 0.323208749294281 + ], + "sources": 7 + }, + { + "entropy_mean": 1.89545738697052, + "mean_weights": [ + 0.14557233452796936, + 0.15975767374038696, + 0.09392085671424866, + 0.09782993793487549, + 0.1459488570690155, + 0.121831014752388, + 0.23513932526111603 + ], + "sources": 7 + }, + { + "entropy_mean": 1.81404447555542, + "mean_weights": [ + 0.10762816667556763, + 0.10907454788684845, + 0.1045805960893631, + 0.11277218908071518, + 0.09011976420879364, + 0.14942197501659393, + 0.3264027535915375 + ], + "sources": 7 + }, + { + "entropy_mean": 1.917024850845337, + "mean_weights": [ + 0.15232855081558228, + 0.16145695745944977, + 0.11022345721721649, + 0.10192710906267166, + 0.14877614378929138, + 0.12149082124233246, + 0.20379695296287537 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8169907331466675, + "mean_weights": [ + 0.09992079436779022, + 0.121503084897995, + 0.11485578119754791, + 0.11612003296613693, + 0.09115365147590637, + 0.14196760952472687, + 0.3144790530204773 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.046370506286621, + "mean_weights": [ + 0.10204233229160309, + 0.11707910895347595, + 0.12750816345214844, + 0.13114766776561737, + 0.10344943404197693, + 0.12833154201507568, + 0.16588172316551208, + 0.12456002831459045 + ], + "sources": 8 + }, + { + "entropy_mean": 2.124986171722412, + "mean_weights": [ + 0.08307430148124695, + 0.10967383533716202, + 0.10474039614200592, + 0.09965920448303223, + 0.08380291610956192, + 0.11342737078666687, + 0.17974808812141418, + 0.1147453784942627, + 0.11112850904464722 + ], + "sources": 9 + }, + { + "entropy_mean": 2.129880428314209, + "mean_weights": [ + 0.09322438389062881, + 0.11352132260799408, + 0.11415109038352966, + 0.11433498561382294, + 0.11000481247901917, + 0.11727333068847656, + 0.13621369004249573, + 0.10061542689800262, + 0.10066096484661102 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0958032608032227, + "mean_weights": [ + 0.08943143486976624, + 0.11672679334878922, + 0.09334117919206619, + 0.09560061991214752, + 0.08163025975227356, + 0.08318871259689331, + 0.2096577137708664, + 0.12659244239330292, + 0.10383082926273346 + ], + "sources": 9 + }, + { + "entropy_mean": 2.159498691558838, + "mean_weights": [ + 0.09486101567745209, + 0.1049872487783432, + 0.11879599839448929, + 0.1244531199336052, + 0.09490413218736649, + 0.11341483891010284, + 0.13785815238952637, + 0.11072608828544617, + 0.09999939054250717 + ], + "sources": 9 + }, + { + "entropy_mean": 2.099748134613037, + "mean_weights": [ + 0.08645306527614594, + 0.12121462821960449, + 0.10100937634706497, + 0.11141529679298401, + 0.08748406171798706, + 0.09598230570554733, + 0.18906186521053314, + 0.11053411662578583, + 0.09684529900550842 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1726603507995605, + "mean_weights": [ + 0.100678950548172, + 0.10167573392391205, + 0.1247546523809433, + 0.12257447093725204, + 0.10496487468481064, + 0.11530985683202744, + 0.11545974016189575, + 0.10518841445446014, + 0.10939331352710724 + ], + "sources": 9 + }, + { + "entropy_mean": 2.125883102416992, + "mean_weights": [ + 0.08610785007476807, + 0.12384747713804245, + 0.11156938970088959, + 0.12631548941135406, + 0.09369373321533203, + 0.10058163106441498, + 0.1809806525707245, + 0.1020854189991951, + 0.07481835037469864 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05586988851428032, + 0.04907773807644844, + 0.05666615068912506, + 0.0663827657699585, + 0.0902026817202568, + 0.1161629855632782, + 0.11026611924171448, + 0.07302531599998474, + 0.11260610818862915, + 0.08639314025640488, + 0.08827003091573715, + 0.08901327103376389, + 0.08757307380437851, + 0.08790616691112518, + 0.08773653954267502, + 0.0901152640581131, + 0.08613155782222748, + 0.05733782798051834, + 0.06344645470380783, + 0.059338539838790894, + 0.06300323456525803, + 0.05568402633070946, + 0.06184924393892288, + 0.05856350064277649, + 0.06753254681825638, + 0.04646647721529007, + 0.05085838586091995, + 0.04775134474039078, + 0.057208102196455, + 0.05061172693967819, + 0.06482923030853271, + 0.054800067096948624, + 0.06379656493663788, + 0.04343016818165779, + 0.04958910122513771, + 0.0419330857694149, + 0.05158477649092674, + 0.04409761354327202, + 0.052934035658836365, + 0.048405785113573074, + 0.058700788766145706, + 0.039685770869255066, + 0.044504519551992416, + 0.039809271693229675, + 0.04639745503664017, + 0.04401715472340584, + 0.04728193208575249, + 0.049276985228061676, + 0.04587510600686073, + 0.03968427702784538, + 0.040653668344020844, + 0.04054687172174454, + 0.042095985263586044, + 0.042120929807424545, + 0.04440675303339958, + 0.04440958797931671, + 0.050673503428697586, + 0.047666996717453, + 0.04782495275139809, + 0.053426820784807205, + 0.0514180064201355, + 0.05560879036784172, + 0.057846032083034515, + 0.05493539944291115 + ], + "loss_nats": 1.4409384727478027, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.976684808731079, + "mean_weights": [ + 0.11598542332649231, + 0.08477729558944702, + 0.07723928987979889, + 0.06115991622209549, + 0.07382568717002869, + 0.05216994509100914, + 0.1197531670331955, + 0.15105611085891724, + 0.26403316855430603 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.047890253365039825, + 0.08920358121395111, + 0.15575747191905975, + 0.16476230323314667, + 0.1985483169555664, + 0.20172803103923798, + 0.20539721846580505, + 0.2172839194536209, + 0.10515304654836655, + 0.10651351511478424, + 0.12655502557754517, + 0.1277865320444107, + 0.13537335395812988, + 0.1365494430065155, + 0.14247354865074158, + 0.14577952027320862, + 0.03568773716688156, + 0.041685812175273895, + 0.054488107562065125, + 0.06442590802907944, + 0.07887282967567444, + 0.09198766946792603, + 0.10933830589056015, + 0.12648026645183563, + 0.0856640487909317, + 0.09409327805042267, + 0.11810280382633209, + 0.127781480550766, + 0.14009103178977966, + 0.1514836698770523, + 0.16954603791236877, + 0.17990845441818237, + 0.030026134103536606, + 0.04308130219578743, + 0.052875518798828125, + 0.06693905591964722, + 0.07490614056587219, + 0.09104157984256744, + 0.09937967360019684, + 0.1198044866323471, + 0.03698531538248062, + 0.052153728902339935, + 0.060006655752658844, + 0.07772442698478699, + 0.08882702887058258, + 0.10783806443214417, + 0.11640279740095139, + 0.14083683490753174, + 0.046741727739572525, + 0.06691420078277588, + 0.08285626024007797, + 0.10727331042289734, + 0.11227487772703171, + 0.14309169352054596, + 0.1445719301700592, + 0.19026093184947968, + 0.008622880093753338, + 0.06030842289328575, + 0.06328795105218887, + 0.12592560052871704, + 0.12800289690494537, + 0.19766274094581604, + 0.19954679906368256, + 0.2932122051715851 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.782868877053261e-05, + 7.111075683496892e-05, + 6.862545706098899e-05, + 6.57451237202622e-05, + 7.662604912184179e-05, + 7.792699034325778e-05, + 7.147213182179257e-05, + 6.637706246692687e-05, + 6.73862814437598e-05, + 6.328667950583622e-05, + 5.89948394917883e-05, + 5.5974429415073246e-05, + 4.847306263400242e-05, + 4.747042112285271e-05, + 4.844431168748997e-05, + 4.801477552973665e-05, + 7.729515345999971e-05, + 7.227301830425858e-05, + 6.455788388848305e-05, + 5.8028235798701644e-05, + 9.002238948596641e-05, + 7.566103886347264e-05, + 6.105600186856464e-05, + 4.845014336751774e-05, + 3.273638503742404e-05, + 3.090824611717835e-05, + 2.944633706647437e-05, + 2.8458403903641738e-05, + 3.676292908494361e-05, + 3.2116196962306276e-05, + 2.993973612319678e-05, + 2.9532720873248763e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 7.071403251757147e-05, + "first_to_last_ratio": 2.263745889883956, + "imbalance_abs_log_ratio": 0.8170209145910685, + "last_quartile_mean": 3.123761939605174e-05, + "mean": 5.6281308786765294e-05, + "normalized": [ + 1.2051725560881539, + 1.2634879743893024, + 1.2193294459621107, + 1.1681520053016667, + 1.3614830709100463, + 1.3845980490343988, + 1.2699088447388673, + 1.179380222276132, + 1.1973119121850604, + 1.1244706434530936, + 1.04821371008453, + 0.9945474016452828, + 0.8612639556349655, + 0.8434491334006702, + 0.8607531120328493, + 0.8531211616213776, + 1.3733716419575495, + 1.2841389061878707, + 1.147057260751975, + 1.0310392037710954, + 1.599507748247948, + 1.3443368765664623, + 1.084836212673898, + 0.8608567286713653, + 0.5816564280950427, + 0.549174260219878, + 0.5231992237074409, + 0.5056457377610559, + 0.6531996124011337, + 0.5706369957384233, + 0.5319658829653441, + 0.52473408152501 + ], + "population_cv": 0.309223682826704 + }, + "activation_output_rms_by_block": [ + 0.12654753029346466, + 0.23922233283519745, + 0.27656757831573486, + 0.29357627034187317, + 0.14313384890556335, + 0.1585245579481125, + 0.16924446821212769, + 0.17900532484054565, + 0.056742530316114426, + 0.0834232047200203, + 0.11722027510404587, + 0.1701633483171463, + 0.10779952257871628, + 0.146687850356102, + 0.17411917448043823, + 0.2125592827796936, + 0.054119229316711426, + 0.08672213554382324, + 0.11636935919523239, + 0.15662381052970886, + 0.06710280478000641, + 0.10124816000461578, + 0.1392347514629364, + 0.18585234880447388, + 0.08930521458387375, + 0.1409701108932495, + 0.18493154644966125, + 0.2363770306110382, + 0.07579561322927475, + 0.15241676568984985, + 0.23702558875083923, + 0.3547191917896271 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.19822773896157742, + "first_to_last_ratio": 1.0776606597304768, + "imbalance_abs_log_ratio": 0.07479263603906072, + "last_quartile_mean": 0.1839426327496767, + "mean": 0.15729221131186932, + "normalized": [ + 0.8045378041164034, + 1.5208784391801964, + 1.7583043432924577, + 1.8664386996237734, + 0.9099868818155679, + 1.0078347594325554, + 1.075987595320662, + 1.1380431576845524, + 0.360745963470555, + 0.5303708557737413, + 0.7452389036074311, + 1.0818294618527353, + 0.6853455850078808, + 0.9325817796868425, + 1.1069789980586224, + 1.3513655953265489, + 0.3440680810914861, + 0.5513441181895264, + 0.7398291258332073, + 0.9957505792779835, + 0.42661238099678755, + 0.6436946824014519, + 0.8851980037769975, + 1.1815737553334873, + 0.5677662856859763, + 0.8962307142706681, + 1.1757196679178878, + 1.5027891631732762, + 0.4818777267934257, + 0.9690038967514244, + 1.5069124324336662, + 2.2551605628222218 + ], + "population_cv": 0.4469233614562388 + }, + "bits_per_byte": 1.9064495266108934, + "branch_output_rms_by_sublayer": [ + 0.07185374200344086, + 0.10062059015035629, + 0.14619378745555878, + 0.014686613343656063, + 0.054412614554166794, + 0.01578865759074688, + 0.02974247932434082, + 0.038228269666433334, + 0.1399001181125641, + 0.01269806083291769, + 0.03587723150849342, + 0.005516543053090572, + 0.03192402049899101, + 0.009458418004214764, + 0.03299693018198013, + 0.02393507771193981, + 0.04883766546845436, + 0.025688232854008675, + 0.03311177343130112, + 0.027711156755685806, + 0.03299272060394287, + 0.03556015342473984, + 0.04865460470318794, + 0.042843904346227646, + 0.09360788762569427, + 0.037112053483724594, + 0.03513377904891968, + 0.03404729813337326, + 0.031736452132463455, + 0.03422597050666809, + 0.04574615880846977, + 0.033918268978595734, + 0.03604442626237869, + 0.037922605872154236, + 0.03289794921875, + 0.039070092141628265, + 0.03499779477715492, + 0.047646332532167435, + 0.039723630994558334, + 0.06680411100387573, + 0.04557310417294502, + 0.045984093099832535, + 0.0335250198841095, + 0.0551915317773819, + 0.04442346468567848, + 0.06537644565105438, + 0.047766126692295074, + 0.0832158699631691, + 0.06149710342288017, + 0.05903136357665062, + 0.05062469467520714, + 0.06259659677743912, + 0.03846435621380806, + 0.06986796110868454, + 0.025445446372032166, + 0.09093746542930603, + 0.015523440204560757, + 0.0736069530248642, + 0.019767511636018753, + 0.09244059771299362, + 0.01620522513985634, + 0.10173976421356201, + 0.023127742111682892, + 0.14117838442325592 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00026742505640546066, + 0.000152196875185642, + 0.0001215702608528516, + 0.0001549011768384087, + 0.00015006516372508994, + 0.0001064406223066867, + 9.579717685290747e-05, + 0.00013823459192787723, + 0.00019795450767675445, + 0.0001521351012967147, + 0.0001674562735333082, + 0.0001930502198804382, + 0.0001558289665195338, + 0.0001378252078217333, + 0.00015263313927783625, + 0.00018511697376470562, + 0.00027178187419018516, + 0.0002887437226070985, + 0.00031544269730619204, + 0.0003486922351480502, + 0.0004757098186915364, + 0.0004311118164786684, + 0.0003950256595773314, + 0.0003553428553987363, + 0.00018611227564225498, + 0.0001667894599423212, + 0.00015946438285379105, + 0.00016166089439116388, + 0.00021187948126009321, + 0.00020237534615772477, + 0.0001839055987251851, + 0.00018366204900628893 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00014832886551186553, + "first_to_last_ratio": 0.8150780241316986, + "imbalance_abs_log_ratio": 0.204471435192671, + "last_quartile_mean": 0.0001819811859973529, + "mean": 0.0002145728587888303, + "normalized": [ + 1.246313526859631, + 0.7093016145878217, + 0.5665686776000581, + 0.7219048006013338, + 0.6993669403117523, + 0.49605818232323196, + 0.4464552385312873, + 0.6442314871887846, + 0.9225514764267989, + 0.7090137222174818, + 0.7804168452549191, + 0.8996954273253478, + 0.7262286917325891, + 0.6423235846308623, + 0.7113347892151104, + 0.8627231552471722, + 1.266618134857664, + 1.345667500712486, + 1.4700959808557696, + 1.625052847393025, + 2.2170083456813208, + 2.0091628499154344, + 1.8409861424556584, + 1.6560475421005756, + 0.8673616816813513, + 0.7773092127484088, + 0.7431712647810984, + 0.7534079347391311, + 0.9874477250107957, + 0.943154447864678, + 0.8570776367675369, + 0.8559425923808848 + ], + "population_cv": 0.44439740033615893 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.23363181948661804, + "mean_weights": [ + 0.9301608204841614, + 0.06983920931816101 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5693116188049316, + "mean_weights": [ + 0.6736154556274414, + 0.32638460397720337 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6035985946655273, + "mean_weights": [ + 0.6857619285583496, + 0.3142380118370056 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6870463490486145, + "mean_weights": [ + 0.5050181746482849, + 0.4949818253517151 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6245008707046509, + "mean_weights": [ + 0.36360669136047363, + 0.6363933086395264 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6884408593177795, + "mean_weights": [ + 0.519433856010437, + 0.480566143989563 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5101574063301086, + "mean_weights": [ + 0.7815959453582764, + 0.21840405464172363 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6856078505516052, + "mean_weights": [ + 0.5456287860870361, + 0.45437127351760864 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0440328121185303, + "mean_weights": [ + 0.2871592938899994, + 0.2525012493133545, + 0.46033942699432373 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0884711742401123, + "mean_weights": [ + 0.31435179710388184, + 0.29442429542541504, + 0.3912239074707031 + ], + "sources": 3 + }, + { + "entropy_mean": 0.8869907259941101, + "mean_weights": [ + 0.20285123586654663, + 0.16300642490386963, + 0.6341423392295837 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0732495784759521, + "mean_weights": [ + 0.34784066677093506, + 0.2432682067155838, + 0.40889108180999756 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9738384485244751, + "mean_weights": [ + 0.27517104148864746, + 0.17857572436332703, + 0.5462532043457031 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0714457035064697, + "mean_weights": [ + 0.3649907410144806, + 0.2359543740749359, + 0.3990548849105835 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0047929286956787, + "mean_weights": [ + 0.2926143407821655, + 0.1949499547481537, + 0.5124356746673584 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0821269750595093, + "mean_weights": [ + 0.39447736740112305, + 0.25779837369918823, + 0.3477242887020111 + ], + "sources": 3 + }, + { + "entropy_mean": 1.34441077709198, + "mean_weights": [ + 0.26774945855140686, + 0.15676936507225037, + 0.2610708475112915, + 0.31441032886505127 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3662428855895996, + "mean_weights": [ + 0.31312593817710876, + 0.19764897227287292, + 0.22056694328784943, + 0.2686581611633301 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3629447221755981, + "mean_weights": [ + 0.27842235565185547, + 0.18004360795021057, + 0.26532843708992004, + 0.27620556950569153 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3617820739746094, + "mean_weights": [ + 0.3079458475112915, + 0.20984455943107605, + 0.20363855361938477, + 0.2785710096359253 + ], + "sources": 4 + }, + { + "entropy_mean": 1.311295747756958, + "mean_weights": [ + 0.3308306038379669, + 0.16218867897987366, + 0.1887943595647812, + 0.31818637251853943 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3658685684204102, + "mean_weights": [ + 0.2829011380672455, + 0.18984708189964294, + 0.23798668384552002, + 0.28926509618759155 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3573216199874878, + "mean_weights": [ + 0.27302947640419006, + 0.20664626359939575, + 0.20834600925445557, + 0.3119782507419586 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3810694217681885, + "mean_weights": [ + 0.27037423849105835, + 0.2429540902376175, + 0.2390410304069519, + 0.24763062596321106 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5861388444900513, + "mean_weights": [ + 0.19025157392024994, + 0.15305551886558533, + 0.19576476514339447, + 0.255463182926178, + 0.20546500384807587 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5885292291641235, + "mean_weights": [ + 0.19492539763450623, + 0.17890498042106628, + 0.1693384349346161, + 0.18084484338760376, + 0.27598631381988525 + ], + "sources": 5 + }, + { + "entropy_mean": 1.591423511505127, + "mean_weights": [ + 0.23808220028877258, + 0.1544100046157837, + 0.18807706236839294, + 0.20902307331562042, + 0.21040767431259155 + ], + "sources": 5 + }, + { + "entropy_mean": 1.546224594116211, + "mean_weights": [ + 0.20235076546669006, + 0.19105949997901917, + 0.1149272620677948, + 0.16552551090717316, + 0.3261369466781616 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5894932746887207, + "mean_weights": [ + 0.2006237506866455, + 0.15459907054901123, + 0.19401878118515015, + 0.2331334352493286, + 0.2176249623298645 + ], + "sources": 5 + }, + { + "entropy_mean": 1.533093810081482, + "mean_weights": [ + 0.20905007421970367, + 0.22428852319717407, + 0.11060983687639236, + 0.12995758652687073, + 0.32609397172927856 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5892066955566406, + "mean_weights": [ + 0.19411224126815796, + 0.16202011704444885, + 0.1960372030735016, + 0.2054326832294464, + 0.2423977553844452 + ], + "sources": 5 + }, + { + "entropy_mean": 1.580901861190796, + "mean_weights": [ + 0.21790803968906403, + 0.2073095440864563, + 0.13282257318496704, + 0.17347125709056854, + 0.2684885859489441 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7687122821807861, + "mean_weights": [ + 0.13978473842144012, + 0.1306772530078888, + 0.17845207452774048, + 0.19443058967590332, + 0.1545381247997284, + 0.20211723446846008 + ], + "sources": 6 + }, + { + "entropy_mean": 1.712350606918335, + "mean_weights": [ + 0.15645939111709595, + 0.16727988421916962, + 0.08566154539585114, + 0.11432348191738129, + 0.17963746190071106, + 0.29663825035095215 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7189819812774658, + "mean_weights": [ + 0.12598168849945068, + 0.1200004369020462, + 0.15623429417610168, + 0.17591747641563416, + 0.12399087846279144, + 0.29787522554397583 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7178866863250732, + "mean_weights": [ + 0.15556901693344116, + 0.1700376570224762, + 0.09179392457008362, + 0.11781732738018036, + 0.16956104338169098, + 0.2952210307121277 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6833292245864868, + "mean_weights": [ + 0.12654924392700195, + 0.12019647657871246, + 0.13885153830051422, + 0.15450379252433777, + 0.12353726476430893, + 0.3363616466522217 + ], + "sources": 6 + }, + { + "entropy_mean": 1.750084638595581, + "mean_weights": [ + 0.1662483662366867, + 0.17961356043815613, + 0.10995318740606308, + 0.12445616722106934, + 0.16552826762199402, + 0.25420045852661133 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6802520751953125, + "mean_weights": [ + 0.1262667030096054, + 0.12312205135822296, + 0.14644092321395874, + 0.16485820710659027, + 0.1135457456111908, + 0.325766384601593 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7465004920959473, + "mean_weights": [ + 0.1759255826473236, + 0.2043798565864563, + 0.092103511095047, + 0.11891678720712662, + 0.198077991604805, + 0.21059629321098328 + ], + "sources": 6 + }, + { + "entropy_mean": 1.8257877826690674, + "mean_weights": [ + 0.09834491461515427, + 0.0991557389497757, + 0.1064278781414032, + 0.1238975077867508, + 0.08494329452514648, + 0.19843076169490814, + 0.28879988193511963 + ], + "sources": 7 + }, + { + "entropy_mean": 1.862605094909668, + "mean_weights": [ + 0.13117730617523193, + 0.14892375469207764, + 0.08699269592761993, + 0.09195820987224579, + 0.13556955754756927, + 0.12358150631189346, + 0.28179696202278137 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7413341999053955, + "mean_weights": [ + 0.09430693089962006, + 0.08963757753372192, + 0.09734517335891724, + 0.11076284945011139, + 0.0783233493566513, + 0.16400417685508728, + 0.365619957447052 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8728678226470947, + "mean_weights": [ + 0.14261263608932495, + 0.16108593344688416, + 0.086087666451931, + 0.0913991630077362, + 0.1429804563522339, + 0.11575779318809509, + 0.2600763440132141 + ], + "sources": 7 + }, + { + "entropy_mean": 1.773127555847168, + "mean_weights": [ + 0.11087753623723984, + 0.10067956149578094, + 0.10114112496376038, + 0.10957588255405426, + 0.08134684711694717, + 0.13775061070919037, + 0.35862845182418823 + ], + "sources": 7 + }, + { + "entropy_mean": 1.901389718055725, + "mean_weights": [ + 0.15126930177211761, + 0.16758683323860168, + 0.09895531833171844, + 0.09678121656179428, + 0.1500449925661087, + 0.1145026832818985, + 0.22085966169834137 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7860735654830933, + "mean_weights": [ + 0.10317397117614746, + 0.11538241058588028, + 0.10709191858768463, + 0.12084762752056122, + 0.08599457144737244, + 0.12617366015911102, + 0.34133583307266235 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.04544734954834, + "mean_weights": [ + 0.10310642421245575, + 0.12280414998531342, + 0.116572305560112, + 0.13426506519317627, + 0.11500787734985352, + 0.1287471354007721, + 0.15969684720039368, + 0.11980022490024567 + ], + "sources": 8 + }, + { + "entropy_mean": 2.075040578842163, + "mean_weights": [ + 0.08472354710102081, + 0.10710075497627258, + 0.09256899356842041, + 0.09574858099222183, + 0.07172106206417084, + 0.10473576188087463, + 0.18532460927963257, + 0.13177165389060974, + 0.12630507349967957 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1171631813049316, + "mean_weights": [ + 0.09575992077589035, + 0.10795485228300095, + 0.1064416766166687, + 0.12594115734100342, + 0.11225643754005432, + 0.11817369610071182, + 0.13458982110023499, + 0.09386932849884033, + 0.10501311719417572 + ], + "sources": 9 + }, + { + "entropy_mean": 2.047362804412842, + "mean_weights": [ + 0.09324105083942413, + 0.1109590083360672, + 0.08460511267185211, + 0.0896209180355072, + 0.06694252789020538, + 0.07065871357917786, + 0.21396517753601074, + 0.1462516188621521, + 0.12375585734844208 + ], + "sources": 9 + }, + { + "entropy_mean": 2.142190933227539, + "mean_weights": [ + 0.09088978171348572, + 0.10934954881668091, + 0.11475005745887756, + 0.13721464574337006, + 0.09275093674659729, + 0.1193506196141243, + 0.1372617483139038, + 0.09967370331287384, + 0.09875896573066711 + ], + "sources": 9 + }, + { + "entropy_mean": 2.053262710571289, + "mean_weights": [ + 0.08806650340557098, + 0.11618921160697937, + 0.09243421256542206, + 0.10599718987941742, + 0.08111069351434708, + 0.08612693101167679, + 0.18951518833637238, + 0.11790518462657928, + 0.12265488505363464 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1595213413238525, + "mean_weights": [ + 0.10617947578430176, + 0.10042236745357513, + 0.11166078597307205, + 0.11633507907390594, + 0.12016797065734863, + 0.10907766222953796, + 0.1113375872373581, + 0.10447067022323608, + 0.12034839391708374 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1123452186584473, + "mean_weights": [ + 0.09050309658050537, + 0.12232520431280136, + 0.10561330616474152, + 0.1267215609550476, + 0.08666616678237915, + 0.09135445952415466, + 0.18464890122413635, + 0.11031270027160645, + 0.08185460418462753 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05918491259217262, + 0.055924948304891586, + 0.06616533547639847, + 0.09222625195980072, + 0.12831835448741913, + 0.1777380406856537, + 0.1443139910697937, + 0.08581677079200745, + 0.14541059732437134, + 0.10501810908317566, + 0.11102382093667984, + 0.1160506084561348, + 0.10497897863388062, + 0.11055652052164078, + 0.10352807492017746, + 0.10864093899726868, + 0.10219033062458038, + 0.0698123574256897, + 0.07480212301015854, + 0.07436435669660568, + 0.07544071972370148, + 0.06532841175794601, + 0.07406888157129288, + 0.07164090871810913, + 0.08180233836174011, + 0.05616654455661774, + 0.059604547917842865, + 0.055941179394721985, + 0.07082673162221909, + 0.06061339005827904, + 0.0829368531703949, + 0.06586005538702011, + 0.07953620702028275, + 0.052934784442186356, + 0.0612504668533802, + 0.050185948610305786, + 0.06369364261627197, + 0.05508765950798988, + 0.06613913923501968, + 0.061504993587732315, + 0.07593764364719391, + 0.0479552336037159, + 0.05495677888393402, + 0.05040038004517555, + 0.05983974039554596, + 0.05726739019155502, + 0.06222398206591606, + 0.06598220020532608, + 0.058408647775650024, + 0.050458863377571106, + 0.05151406675577164, + 0.051624953746795654, + 0.053354412317276, + 0.05360622704029083, + 0.056061357259750366, + 0.05612650886178017, + 0.06349223107099533, + 0.06260828673839569, + 0.05886351689696312, + 0.07101194560527802, + 0.06255334615707397, + 0.07260199636220932, + 0.07205114513635635, + 0.06936988234519958 + ], + "loss_nats": 1.321450114250183, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.913226842880249, + "mean_weights": [ + 0.1031094342470169, + 0.0850210189819336, + 0.07202162593603134, + 0.05865488201379776, + 0.06886604428291321, + 0.04668717458844185, + 0.10652348399162292, + 0.14514580368995667, + 0.31397056579589844 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.07185374200344086, + 0.12654753029346466, + 0.23357203602790833, + 0.23922233283519745, + 0.2709117829799652, + 0.27656757831573486, + 0.27930477261543274, + 0.29357627034187317, + 0.1399001181125641, + 0.14313384890556335, + 0.15757356584072113, + 0.1585245579481125, + 0.16797132790088654, + 0.16924446821212769, + 0.1751146912574768, + 0.17900532484054565, + 0.04883766546845436, + 0.056742530316114426, + 0.07011433690786362, + 0.0834232047200203, + 0.09868786484003067, + 0.11722027510404587, + 0.1431102603673935, + 0.1701633483171463, + 0.09360788762569427, + 0.10779952257871628, + 0.1321963220834732, + 0.146687850356102, + 0.15773653984069824, + 0.17411917448043823, + 0.19743746519088745, + 0.2125592827796936, + 0.03604442626237869, + 0.054119229316711426, + 0.06793603301048279, + 0.08672213554382324, + 0.09488140046596527, + 0.11636935919523239, + 0.1266157180070877, + 0.15662381052970886, + 0.04557310417294502, + 0.06710280478000641, + 0.07621937990188599, + 0.10124816000461578, + 0.11287613213062286, + 0.1392347514629364, + 0.15132750570774078, + 0.18585234880447388, + 0.06149710342288017, + 0.08930521458387375, + 0.11077401041984558, + 0.1409701108932495, + 0.1504175066947937, + 0.18493154644966125, + 0.1877913922071457, + 0.2363770306110382, + 0.015523440204560757, + 0.07579561322927475, + 0.0795246958732605, + 0.15241676568984985, + 0.1549522429704666, + 0.23702558875083923, + 0.23988524079322815, + 0.3547191917896271 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00011256998550379649, + 0.00011608834029175341, + 0.00011307936802040786, + 0.00010960452345898375, + 0.0001336085406364873, + 0.00013472123828250915, + 0.0001259810960618779, + 0.00011922900011995807, + 0.00012442414299584925, + 0.00011725658259820193, + 0.00010905973613262177, + 0.0001026458412525244, + 8.79938670550473e-05, + 8.631208766018972e-05, + 8.753449947107583e-05, + 8.726862870389596e-05, + 0.00014698492304887623, + 0.0001351345854345709, + 0.00011712588457157835, + 0.00010417791781947017, + 0.00016132646123878658, + 0.00013268974726088345, + 0.00010458255565026775, + 8.047515439102426e-05, + 5.884338679607026e-05, + 5.559486089623533e-05, + 5.261701880954206e-05, + 5.040612450102344e-05, + 6.887591007398441e-05, + 5.42616871825885e-05, + 4.7548299335176125e-05, + 4.5572312956210226e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00012061026154697174, + "first_to_last_ratio": 2.2246679447974205, + "imbalance_abs_log_ratio": 0.7996076661877672, + "last_quartile_mean": 5.4214950068853796e-05, + "mean": 9.948732213160838e-05, + "normalized": [ + 1.1315008092677528, + 1.1668656649355191, + 1.1366208839234715, + 1.101693372689152, + 1.3429705189947834, + 1.3541548349677262, + 1.2663030159282187, + 1.1984341076366907, + 1.2506532523938356, + 1.178608289839053, + 1.0962174254559829, + 1.031747955952998, + 0.8844731687384572, + 0.8675687093679173, + 0.8798558207776407, + 0.8771834122588129, + 1.4774236545882187, + 1.3583096070854734, + 1.177294574444737, + 1.0471476725613014, + 1.6215780843449914, + 1.3337352379969853, + 1.0512149026578388, + 0.8088985879483863, + 0.5914661841860449, + 0.5588135222163362, + 0.5288816472508608, + 0.5066587724046175, + 0.6923084127530423, + 0.5454130839988593, + 0.47793325135715387, + 0.45807156107714075 + ], + "population_cv": 0.31692820338644817 + }, + "activation_output_rms_by_block": [ + 0.11427237838506699, + 0.23817487061023712, + 0.27317890524864197, + 0.2844410836696625, + 0.13613075017929077, + 0.14841797947883606, + 0.15770810842514038, + 0.1659657508134842, + 0.048475202172994614, + 0.07034122943878174, + 0.10131242871284485, + 0.1543440818786621, + 0.0956718772649765, + 0.12488728016614914, + 0.1484198421239853, + 0.1830996870994568, + 0.04907182976603508, + 0.07541443407535553, + 0.10293498635292053, + 0.13807444274425507, + 0.060822080820798874, + 0.09160740673542023, + 0.12607654929161072, + 0.1690700650215149, + 0.08542407304048538, + 0.12935929000377655, + 0.16742807626724243, + 0.20706304907798767, + 0.06557665020227432, + 0.12835223972797394, + 0.1965756118297577, + 0.284595787525177 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.189786228351295, + "first_to_last_ratio": 1.2008226149549288, + "imbalance_abs_log_ratio": 0.1830068344000524, + "last_quartile_mean": 0.15804684720933437, + "mean": 0.1413215008797124, + "normalized": [ + 0.8085986751749218, + 1.6853406532454158, + 1.9330314463696623, + 2.0127233428674676, + 0.9632699152775075, + 1.0502151375052542, + 1.1159526854967077, + 1.174384292413849, + 0.3430136382025471, + 0.4977390489126886, + 0.7168932404636589, + 1.092148617994328, + 0.6769803368166096, + 0.8837104006732036, + 1.0502283177017397, + 1.2956251239880647, + 0.3472354132992346, + 0.5336373701518037, + 0.7283745623430291, + 0.9770236084725628, + 0.4303809430425478, + 0.6482198827862226, + 0.8921257452460994, + 1.1963506187598518, + 0.6044662171624908, + 0.9153546289738484, + 1.1847318010707442, + 1.4651914095805707, + 0.46402458078966147, + 0.9082286766627435, + 1.3909816312881897, + 2.0138180372667733 + ], + "population_cv": 0.45198740397417647 + }, + "bits_per_byte": 1.8755791567289717, + "branch_output_rms_by_sublayer": [ + 0.06924261152744293, + 0.08715036511421204, + 0.1563301533460617, + 0.010194960050284863, + 0.04894939064979553, + 0.015863409265875816, + 0.024761321023106575, + 0.02872019074857235, + 0.13285663723945618, + 0.011080419644713402, + 0.024652149528265, + 0.004756811074912548, + 0.024704737588763237, + 0.008602344430983067, + 0.02762894332408905, + 0.021304847672581673, + 0.04170539230108261, + 0.02205050364136696, + 0.026622971519827843, + 0.02331046760082245, + 0.030255690217018127, + 0.03148031234741211, + 0.050311315804719925, + 0.039536088705062866, + 0.08088748157024384, + 0.035828687250614166, + 0.02678149938583374, + 0.03196560963988304, + 0.02690052054822445, + 0.03224668651819229, + 0.04079823195934296, + 0.031973257660865784, + 0.03209294006228447, + 0.03466026484966278, + 0.027836447581648827, + 0.03666257485747337, + 0.03129557520151138, + 0.04683101922273636, + 0.03393841162323952, + 0.06432909518480301, + 0.03914104029536247, + 0.0440673828125, + 0.028756819665431976, + 0.053904641419649124, + 0.0382247194647789, + 0.06515713781118393, + 0.04269610345363617, + 0.08360167592763901, + 0.05708961561322212, + 0.058046840131282806, + 0.04346603527665138, + 0.06114881858229637, + 0.03982488811016083, + 0.06687389314174652, + 0.024579433724284172, + 0.08207371830940247, + 0.013750901445746422, + 0.06377240270376205, + 0.01623797044157982, + 0.08086662739515305, + 0.015543250367045403, + 0.08651969581842422, + 0.021196292713284492, + 0.1121927872300148 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0004436480430024554, + 0.000290036674007586, + 0.00020860017779762794, + 0.00023245217764663882, + 0.00027479178399362033, + 0.00013709769545517192, + 0.00015740296649897437, + 0.00024061210626211127, + 0.0003391664662389327, + 0.00026191767940717664, + 0.00031600822080415713, + 0.0003744741622146375, + 0.0002902464308626841, + 0.0002520977616836866, + 0.00027306405354830804, + 0.0003520249576836477, + 0.0004960279458685245, + 0.0005352096190275505, + 0.0005967733567124423, + 0.0006392271142212413, + 0.0008647342238653121, + 0.0008051198506940602, + 0.0007113117068853425, + 0.0006533319845843197, + 0.0003553082109604543, + 0.00030422111069073227, + 0.0002946392003127672, + 0.00028707150256358863, + 0.00038378384006631914, + 0.0003413762489815915, + 0.00029335142547013934, + 0.0002815937044687921 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00024808020308302324, + "first_to_last_ratio": 0.7809413654949368, + "imbalance_abs_log_ratio": 0.2472552081516714, + "last_quartile_mean": 0.00031766815543929805, + "mean": 0.00038396007507751856, + "normalized": [ + 1.1554535791589433, + 0.7553823765375342, + 0.5432861157648049, + 0.6054071575012285, + 0.7156780140178428, + 0.35706237276751485, + 0.4099461811679142, + 0.6266591812014142, + 0.8833378474844229, + 0.682148213858687, + 0.8230236457276385, + 0.975294533263736, + 0.7559286751469814, + 0.656572852353959, + 0.7111782481373318, + 0.9168269841925013, + 1.2918737599693935, + 1.3939199770171309, + 1.5542588811921616, + 1.66482703726992, + 2.252146199551422, + 2.0968842933253224, + 1.8525668501906982, + 1.7015622899136507, + 0.9253780119943991, + 0.7923248546852492, + 0.7673693684253027, + 0.7476597729741331, + 0.9995410069362971, + 0.8890930884224624, + 0.7640154393940931, + 0.7333931904559101 + ], + "population_cv": 0.4720193871762268 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.21751394867897034, + "mean_weights": [ + 0.9370211362838745, + 0.06297890841960907 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5407636165618896, + "mean_weights": [ + 0.7099307775497437, + 0.29006922245025635 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6493605375289917, + "mean_weights": [ + 0.6113038063049316, + 0.38869616389274597 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6895085573196411, + "mean_weights": [ + 0.4795704483985901, + 0.5204295516014099 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6044203042984009, + "mean_weights": [ + 0.33118173480033875, + 0.6688182353973389 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6896922588348389, + "mean_weights": [ + 0.5031631588935852, + 0.4968368411064148 + ], + "sources": 2 + }, + { + "entropy_mean": 0.558347761631012, + "mean_weights": [ + 0.7392898201942444, + 0.2607101798057556 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6835589408874512, + "mean_weights": [ + 0.5489000082015991, + 0.4510999619960785 + ], + "sources": 2 + }, + { + "entropy_mean": 1.036144495010376, + "mean_weights": [ + 0.27323150634765625, + 0.24991267919540405, + 0.4768558144569397 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0787163972854614, + "mean_weights": [ + 0.29843032360076904, + 0.27880463004112244, + 0.42276501655578613 + ], + "sources": 3 + }, + { + "entropy_mean": 0.8596357703208923, + "mean_weights": [ + 0.18808063864707947, + 0.15459778904914856, + 0.657321572303772 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0663710832595825, + "mean_weights": [ + 0.3278513550758362, + 0.23661038279533386, + 0.43553826212882996 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9620242118835449, + "mean_weights": [ + 0.26144468784332275, + 0.1769636571407318, + 0.561591625213623 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0630332231521606, + "mean_weights": [ + 0.3378422260284424, + 0.22928667068481445, + 0.43287110328674316 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9962137937545776, + "mean_weights": [ + 0.2828536033630371, + 0.1889738142490387, + 0.5281726121902466 + ], + "sources": 3 + }, + { + "entropy_mean": 1.083638310432434, + "mean_weights": [ + 0.37394392490386963, + 0.25705721974372864, + 0.36899885535240173 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3319507837295532, + "mean_weights": [ + 0.24367234110832214, + 0.14257535338401794, + 0.2770358622074127, + 0.3367164433002472 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3682777881622314, + "mean_weights": [ + 0.29398924112319946, + 0.1932728886604309, + 0.2261681705713272, + 0.28656965494155884 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3582814931869507, + "mean_weights": [ + 0.2546596825122833, + 0.1722031682729721, + 0.2767754793167114, + 0.29636165499687195 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3632707595825195, + "mean_weights": [ + 0.2934684455394745, + 0.20407642424106598, + 0.21240410208702087, + 0.29005101323127747 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3069984912872314, + "mean_weights": [ + 0.29863882064819336, + 0.1513044685125351, + 0.19813954830169678, + 0.3519171178340912 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3630049228668213, + "mean_weights": [ + 0.2708645761013031, + 0.18405304849147797, + 0.24000081419944763, + 0.3050815761089325 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3509695529937744, + "mean_weights": [ + 0.2596704363822937, + 0.19961732625961304, + 0.2078961580991745, + 0.33281606435775757 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3828163146972656, + "mean_weights": [ + 0.2659085988998413, + 0.24258729815483093, + 0.2402249574661255, + 0.25127917528152466 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5766849517822266, + "mean_weights": [ + 0.1814059615135193, + 0.14477907121181488, + 0.18362656235694885, + 0.27523452043533325, + 0.21495386958122253 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5842468738555908, + "mean_weights": [ + 0.1911981999874115, + 0.17350733280181885, + 0.16918015480041504, + 0.17833834886550903, + 0.2877759635448456 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5882223844528198, + "mean_weights": [ + 0.225559800863266, + 0.1412772536277771, + 0.18970009684562683, + 0.22256594896316528, + 0.220896914601326 + ], + "sources": 5 + }, + { + "entropy_mean": 1.545775055885315, + "mean_weights": [ + 0.19951410591602325, + 0.18934985995292664, + 0.11679990589618683, + 0.16236910223960876, + 0.3319670557975769 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5807313919067383, + "mean_weights": [ + 0.1891961395740509, + 0.1419243961572647, + 0.19001564383506775, + 0.2432972490787506, + 0.23556658625602722 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5314936637878418, + "mean_weights": [ + 0.20433203876018524, + 0.22880440950393677, + 0.10845310986042023, + 0.1300453543663025, + 0.3283650875091553 + ], + "sources": 5 + }, + { + "entropy_mean": 1.583519458770752, + "mean_weights": [ + 0.1890762746334076, + 0.14884406328201294, + 0.19678419828414917, + 0.2067156732082367, + 0.2585797905921936 + ], + "sources": 5 + }, + { + "entropy_mean": 1.584309697151184, + "mean_weights": [ + 0.212642639875412, + 0.21519626677036285, + 0.13616301119327545, + 0.1728803515434265, + 0.2631177604198456 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7621970176696777, + "mean_weights": [ + 0.13663369417190552, + 0.12357592582702637, + 0.16730815172195435, + 0.18850678205490112, + 0.15588003396987915, + 0.2280954122543335 + ], + "sources": 6 + }, + { + "entropy_mean": 1.701179027557373, + "mean_weights": [ + 0.14903563261032104, + 0.1626545786857605, + 0.08506090939044952, + 0.11331439018249512, + 0.1763274371623993, + 0.31360703706741333 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6943695545196533, + "mean_weights": [ + 0.12271308153867722, + 0.10987471044063568, + 0.14610394835472107, + 0.16883134841918945, + 0.12254531681537628, + 0.3299315869808197 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7172635793685913, + "mean_weights": [ + 0.15381330251693726, + 0.17127496004104614, + 0.09106636047363281, + 0.1169445663690567, + 0.1724638044834137, + 0.2944370210170746 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6638290882110596, + "mean_weights": [ + 0.12103515863418579, + 0.11075270920991898, + 0.13912361860275269, + 0.15793362259864807, + 0.11660546064376831, + 0.3545494079589844 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7515745162963867, + "mean_weights": [ + 0.16375267505645752, + 0.18288594484329224, + 0.10830142349004745, + 0.12405762076377869, + 0.17245593667030334, + 0.24854639172554016 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6730718612670898, + "mean_weights": [ + 0.12389004230499268, + 0.12033841013908386, + 0.1418823003768921, + 0.1629519760608673, + 0.11382497847080231, + 0.33711230754852295 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7427812814712524, + "mean_weights": [ + 0.1725234091281891, + 0.2093750536441803, + 0.08763118088245392, + 0.11911962926387787, + 0.20339494943618774, + 0.20795577764511108 + ], + "sources": 6 + }, + { + "entropy_mean": 1.8161566257476807, + "mean_weights": [ + 0.09697173535823822, + 0.09447111934423447, + 0.10528635233640671, + 0.12399692833423615, + 0.08241759240627289, + 0.20165753364562988, + 0.2951987385749817 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8578764200210571, + "mean_weights": [ + 0.12829428911209106, + 0.15037661790847778, + 0.08424612879753113, + 0.08993658423423767, + 0.1407640278339386, + 0.12182091176509857, + 0.2845614552497864 + ], + "sources": 7 + }, + { + "entropy_mean": 1.729842185974121, + "mean_weights": [ + 0.09100021421909332, + 0.0852670669555664, + 0.10036946088075638, + 0.11213124543428421, + 0.07580001652240753, + 0.1619945764541626, + 0.37343740463256836 + ], + "sources": 7 + }, + { + "entropy_mean": 1.867231845855713, + "mean_weights": [ + 0.13960297405719757, + 0.16482335329055786, + 0.08323060721158981, + 0.08897126466035843, + 0.14854834973812103, + 0.11251068115234375, + 0.26231276988983154 + ], + "sources": 7 + }, + { + "entropy_mean": 1.759962558746338, + "mean_weights": [ + 0.10778215527534485, + 0.092461958527565, + 0.10151682049036026, + 0.11454993486404419, + 0.08033804595470428, + 0.13504932820796967, + 0.36830174922943115 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8995733261108398, + "mean_weights": [ + 0.14892423152923584, + 0.1685381829738617, + 0.09900127351284027, + 0.09578512609004974, + 0.15339402854442596, + 0.11228728294372559, + 0.2220698744058609 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7808358669281006, + "mean_weights": [ + 0.10197260975837708, + 0.10992246866226196, + 0.11207077652215958, + 0.1229957714676857, + 0.0817628800868988, + 0.12744472920894623, + 0.34383076429367065 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0509724617004395, + "mean_weights": [ + 0.10499924421310425, + 0.12799815833568573, + 0.1185382828116417, + 0.12561362981796265, + 0.1268092542886734, + 0.12648361921310425, + 0.14848148822784424, + 0.12107636034488678 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0520524978637695, + "mean_weights": [ + 0.0824132114648819, + 0.10357530415058136, + 0.08798205852508545, + 0.0907672867178917, + 0.06845161318778992, + 0.10302996635437012, + 0.17704220116138458, + 0.14631247520446777, + 0.1404258906841278 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1290132999420166, + "mean_weights": [ + 0.09601300209760666, + 0.112395741045475, + 0.11105199158191681, + 0.11652921885251999, + 0.11739972233772278, + 0.11520169675350189, + 0.1253741979598999, + 0.09306395053863525, + 0.1129704937338829 + ], + "sources": 9 + }, + { + "entropy_mean": 2.01992130279541, + "mean_weights": [ + 0.09216365218162537, + 0.09879494458436966, + 0.08027730882167816, + 0.08336887508630753, + 0.06550291180610657, + 0.06227210536599159, + 0.20018666982650757, + 0.16262760758399963, + 0.15480594336986542 + ], + "sources": 9 + }, + { + "entropy_mean": 2.148375988006592, + "mean_weights": [ + 0.09072244167327881, + 0.11055954545736313, + 0.12024322897195816, + 0.13449642062187195, + 0.09748931229114532, + 0.11299417912960052, + 0.13041159510612488, + 0.09668298810720444, + 0.10640028119087219 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0205559730529785, + "mean_weights": [ + 0.08509345352649689, + 0.10695987939834595, + 0.089454285800457, + 0.09759271144866943, + 0.0741405040025711, + 0.08196006715297699, + 0.18127255141735077, + 0.12566132843494415, + 0.1578652262687683 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1612603664398193, + "mean_weights": [ + 0.10732871294021606, + 0.10928969085216522, + 0.11351121962070465, + 0.10611467063426971, + 0.12792731821537018, + 0.10549595952033997, + 0.10168985277414322, + 0.10327720642089844, + 0.12536534667015076 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1142916679382324, + "mean_weights": [ + 0.09036671370267868, + 0.11588644981384277, + 0.1058499813079834, + 0.11694025993347168, + 0.08344843983650208, + 0.0905958041548729, + 0.17790725827217102, + 0.11846087127923965, + 0.10054422914981842 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06211438775062561, + 0.05887576937675476, + 0.0625971183180809, + 0.1053200289607048, + 0.13316068053245544, + 0.18203993141651154, + 0.1463746279478073, + 0.09312358498573303, + 0.14036449790000916, + 0.09766179323196411, + 0.10165993869304657, + 0.10703272372484207, + 0.09639595448970795, + 0.10139063745737076, + 0.09578371793031693, + 0.09860112518072128, + 0.09399279952049255, + 0.062042053788900375, + 0.0675555169582367, + 0.0667034164071083, + 0.06799062341451645, + 0.05742833390831947, + 0.06505316495895386, + 0.0616331472992897, + 0.07167881727218628, + 0.046929772943258286, + 0.050851304084062576, + 0.04679100960493088, + 0.06160381808876991, + 0.0506686232984066, + 0.07409743219614029, + 0.055399734526872635, + 0.06967058777809143, + 0.042987193912267685, + 0.052712973207235336, + 0.04142794758081436, + 0.056203652173280716, + 0.04678492248058319, + 0.05908435583114624, + 0.053378067910671234, + 0.06883536279201508, + 0.04117409139871597, + 0.049047794193029404, + 0.04354740306735039, + 0.05438254401087761, + 0.050159431993961334, + 0.05549543723464012, + 0.058089159429073334, + 0.050311364233493805, + 0.04327835887670517, + 0.044068675488233566, + 0.04417750611901283, + 0.04559645056724548, + 0.04590814933180809, + 0.04793088510632515, + 0.047960035502910614, + 0.054211847484111786, + 0.055152423679828644, + 0.049323346465826035, + 0.0626484602689743, + 0.052339378744363785, + 0.06438160687685013, + 0.05978427082300186, + 0.06028405949473381 + ], + "loss_nats": 1.3000524044036865, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.7828166484832764, + "mean_weights": [ + 0.0955134779214859, + 0.07318860292434692, + 0.06981553137302399, + 0.048903122544288635, + 0.05929988622665405, + 0.039753444492816925, + 0.08527521789073944, + 0.13367652893066406, + 0.3945741653442383 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.06924261152744293, + 0.11427237838506699, + 0.2347099334001541, + 0.23817487061023712, + 0.26708653569221497, + 0.27317890524864197, + 0.2754039168357849, + 0.2844410836696625, + 0.13285663723945618, + 0.13613075017929077, + 0.1473916918039322, + 0.14841797947883606, + 0.15650348365306854, + 0.15770810842514038, + 0.1623157411813736, + 0.1659657508134842, + 0.04170539230108261, + 0.048475202172994614, + 0.05908121168613434, + 0.07034122943878174, + 0.08480077236890793, + 0.10131242871284485, + 0.12928204238414764, + 0.1543440818786621, + 0.08088748157024384, + 0.0956718772649765, + 0.1105017215013504, + 0.12488728016614914, + 0.13256269693374634, + 0.1484198421239853, + 0.16828303039073944, + 0.1830996870994568, + 0.03209294006228447, + 0.04907182976603508, + 0.058096010237932205, + 0.07541443407535553, + 0.0825863927602768, + 0.10293498635292053, + 0.1106458455324173, + 0.13807444274425507, + 0.03914104029536247, + 0.060822080820798874, + 0.06762295961380005, + 0.09160740673542023, + 0.10062053054571152, + 0.12607654929161072, + 0.13570784032344818, + 0.1690700650215149, + 0.05708961561322212, + 0.08542407304048538, + 0.10116130858659744, + 0.12935929000377655, + 0.13832047581672668, + 0.16742807626724243, + 0.16957785189151764, + 0.20706304907798767, + 0.013750901445746422, + 0.06557665020227432, + 0.06806214153766632, + 0.12835223972797394, + 0.13001596927642822, + 0.1965756118297577, + 0.19671563804149628, + 0.284595787525177 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 8.061194200882085, + "cross_entropy_nats": 5.587594032287598, + "step": 0 + }, + { + "bits_per_byte": 5.192306155424083, + "cross_entropy_nats": 3.599032372236252, + "step": 100 + }, + { + "bits_per_byte": 3.098279494079058, + "cross_entropy_nats": 2.1475636959075928, + "step": 500 + }, + { + "bits_per_byte": 1.9542788692182516, + "cross_entropy_nats": 1.354602888226509, + "step": 2000 + }, + { + "bits_per_byte": 1.7924308335033179, + "cross_entropy_nats": 1.2424183785915375, + "step": 4000 + }, + { + "bits_per_byte": 1.7072041429376497, + "cross_entropy_nats": 1.1833437383174896, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.48.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.48.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "48": true, + "49": true, + "50": true, + "51": true, + "52": true, + "53": true, + "54": true, + "55": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_group_7_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "74f9829851f9a4834c4112f9329028e7eb3ebbedf188e4312704cb4b59743179", + "final_model_state": "aaa5722d8bed3244b0a50f88228f9b9391669a4397946a42f6e451878e59527e", + "final_optimizer_state": "f138e7912ae2ac41883d3efee8f1bb6d61d22262eeb340f2bbd1af55b79705b4", + "final_public_parameters": "1e090fe4ce6412afc9e0ea3a0369c64aec60c40faf54ab21e3e8b18a9425f345", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "3f93c11463780b386fff00c53b0424f687e1136b1b80f7b626f058adef96c764" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "3eff5c07fa0cfbd5ceed5922cbd09525ad514f801cc0a440c03fd2fa15df4c84", + "1": "e83e4001c5811b19c69a54c7ea7a802af996a34aa57072cde43a76bc8369e2d0", + "7999": "d04c4ca879f0209183a48ac54a0c4ffe3a0f62a78888479d4641f6c304426442" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "94a866492867637158b7d5521b4c6178b9752253196893e6b5fbe80e0d887b98", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073003, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 209.38021183748953, + "measured_steps": 7980, + "median_ms": 210.02767949539702, + "p95_ms": 211.76163228228688, + "peak_allocated_bytes": 11621543936, + "peak_reserved_bytes": 12083789824, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 8.082576459898554, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.602415084838867, + "step": 1, + "unclipped_grad_norm": 19.082693099975586 + }, + { + "bits_per_byte": 7.337154120055694, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.085727691650391, + "step": 10, + "unclipped_grad_norm": 11.735821723937988 + }, + { + "bits_per_byte": 6.794642415145602, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.709687232971191, + "step": 20, + "unclipped_grad_norm": 3.3706324100494385 + }, + { + "bits_per_byte": 6.66971077790584, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.623091220855713, + "step": 30, + "unclipped_grad_norm": 3.0377907752990723 + }, + { + "bits_per_byte": 6.4886357482572175, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.497579574584961, + "step": 40, + "unclipped_grad_norm": 3.071356773376465 + }, + { + "bits_per_byte": 6.170624004408005, + "learning_rate": 3.75e-05, + "loss_nats": 4.277150630950928, + "step": 50, + "unclipped_grad_norm": 2.825165033340454 + }, + { + "bits_per_byte": 5.945986457226435, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.121443748474121, + "step": 60, + "unclipped_grad_norm": 2.6026313304901123 + }, + { + "bits_per_byte": 5.764125122898067, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.995387077331543, + "step": 70, + "unclipped_grad_norm": 2.5196492671966553 + }, + { + "bits_per_byte": 5.5957678663154455, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.878690719604492, + "step": 80, + "unclipped_grad_norm": 2.5942959785461426 + }, + { + "bits_per_byte": 5.405339044251696, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.7466955184936523, + "step": 90, + "unclipped_grad_norm": 2.3170742988586426 + }, + { + "bits_per_byte": 5.1783412071696855, + "learning_rate": 7.5e-05, + "loss_nats": 3.589352607727051, + "step": 100, + "unclipped_grad_norm": 2.216233253479004 + }, + { + "bits_per_byte": 4.995879997355107, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.4628801345825195, + "step": 110, + "unclipped_grad_norm": 2.0336647033691406 + }, + { + "bits_per_byte": 4.799493008334866, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.3267550468444824, + "step": 120, + "unclipped_grad_norm": 1.8059049844741821 + }, + { + "bits_per_byte": 4.634010269778766, + "learning_rate": 9.75e-05, + "loss_nats": 3.2120511531829834, + "step": 130, + "unclipped_grad_norm": 1.5959006547927856 + }, + { + "bits_per_byte": 4.477679759970982, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.1036911010742188, + "step": 140, + "unclipped_grad_norm": 1.3925493955612183 + }, + { + "bits_per_byte": 4.276178694844685, + "learning_rate": 0.0001125, + "loss_nats": 2.9640212059020996, + "step": 150, + "unclipped_grad_norm": 1.2069321870803833 + }, + { + "bits_per_byte": 4.1298936916737965, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.862624168395996, + "step": 160, + "unclipped_grad_norm": 1.0484689474105835 + }, + { + "bits_per_byte": 4.022712039864284, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.7883315086364746, + "step": 170, + "unclipped_grad_norm": 0.9141635894775391 + }, + { + "bits_per_byte": 3.9202863962817314, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.7173354625701904, + "step": 180, + "unclipped_grad_norm": 0.8749651312828064 + }, + { + "bits_per_byte": 3.8317696776050574, + "learning_rate": 0.0001425, + "loss_nats": 2.655980348587036, + "step": 190, + "unclipped_grad_norm": 0.8055938482284546 + }, + { + "bits_per_byte": 3.7637522590153254, + "learning_rate": 0.00015, + "loss_nats": 2.6088342666625977, + "step": 200, + "unclipped_grad_norm": 0.710497260093689 + }, + { + "bits_per_byte": 3.6853770132947297, + "learning_rate": 0.0001575, + "loss_nats": 2.554508686065674, + "step": 210, + "unclipped_grad_norm": 1.044273853302002 + }, + { + "bits_per_byte": 3.6512680381144107, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.5308661460876465, + "step": 220, + "unclipped_grad_norm": 0.800299346446991 + }, + { + "bits_per_byte": 3.6104348852506574, + "learning_rate": 0.0001725, + "loss_nats": 2.5025627613067627, + "step": 230, + "unclipped_grad_norm": 0.8922969698905945 + }, + { + "bits_per_byte": 3.5875567211369814, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.4867048263549805, + "step": 240, + "unclipped_grad_norm": 0.7792111039161682 + }, + { + "bits_per_byte": 3.505000233208609, + "learning_rate": 0.0001875, + "loss_nats": 2.429481029510498, + "step": 250, + "unclipped_grad_norm": 0.7236195206642151 + }, + { + "bits_per_byte": 3.5166881741612457, + "learning_rate": 0.000195, + "loss_nats": 2.437582492828369, + "step": 260, + "unclipped_grad_norm": 0.9388204216957092 + }, + { + "bits_per_byte": 3.4999542622322943, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.425983428955078, + "step": 270, + "unclipped_grad_norm": 0.9595193862915039 + }, + { + "bits_per_byte": 3.466532185759521, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.4028170108795166, + "step": 280, + "unclipped_grad_norm": 1.7195367813110352 + }, + { + "bits_per_byte": 3.4330932549869635, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.379638910293579, + "step": 290, + "unclipped_grad_norm": 0.9839808940887451 + }, + { + "bits_per_byte": 3.4398528610964756, + "learning_rate": 0.000225, + "loss_nats": 2.384324312210083, + "step": 300, + "unclipped_grad_norm": 1.5129112005233765 + }, + { + "bits_per_byte": 3.323415446982828, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.3036160469055176, + "step": 310, + "unclipped_grad_norm": 1.0021251440048218 + }, + { + "bits_per_byte": 3.4068445748816782, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.3614447116851807, + "step": 320, + "unclipped_grad_norm": 1.3444116115570068 + }, + { + "bits_per_byte": 3.4036677113549403, + "learning_rate": 0.0002475, + "loss_nats": 2.3592426776885986, + "step": 330, + "unclipped_grad_norm": 0.9060816764831543 + }, + { + "bits_per_byte": 3.3321019467126067, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.3096370697021484, + "step": 340, + "unclipped_grad_norm": 1.0138038396835327 + }, + { + "bits_per_byte": 3.3095976728819068, + "learning_rate": 0.0002625, + "loss_nats": 2.2940382957458496, + "step": 350, + "unclipped_grad_norm": 0.8970586657524109 + }, + { + "bits_per_byte": 3.3025002928461, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.289118766784668, + "step": 360, + "unclipped_grad_norm": 0.9033161401748657 + }, + { + "bits_per_byte": 3.3116422026353636, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.2954554557800293, + "step": 370, + "unclipped_grad_norm": 1.0622698068618774 + }, + { + "bits_per_byte": 3.19993362348984, + "learning_rate": 0.000285, + "loss_nats": 2.218024969100952, + "step": 380, + "unclipped_grad_norm": 1.0439220666885376 + }, + { + "bits_per_byte": 3.2829947085158437, + "learning_rate": 0.0002925, + "loss_nats": 2.2755985260009766, + "step": 390, + "unclipped_grad_norm": 0.9923666715621948 + }, + { + "bits_per_byte": 3.289365289869104, + "learning_rate": 0.0003, + "loss_nats": 2.2800142765045166, + "step": 400, + "unclipped_grad_norm": 1.445814847946167 + }, + { + "bits_per_byte": 3.2716975121459675, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.267767906188965, + "step": 410, + "unclipped_grad_norm": 1.1496127843856812 + }, + { + "bits_per_byte": 3.2052606141176647, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.221717357635498, + "step": 420, + "unclipped_grad_norm": 0.8571957349777222 + }, + { + "bits_per_byte": 3.2375586119843986, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.2441046237945557, + "step": 430, + "unclipped_grad_norm": 1.1314860582351685 + }, + { + "bits_per_byte": 3.1798329791875983, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.204092264175415, + "step": 440, + "unclipped_grad_norm": 1.2115997076034546 + }, + { + "bits_per_byte": 3.24455933777042, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.2489571571350098, + "step": 450, + "unclipped_grad_norm": 1.1346222162246704 + }, + { + "bits_per_byte": 3.1812569955367427, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.2050793170928955, + "step": 460, + "unclipped_grad_norm": 1.071357011795044 + }, + { + "bits_per_byte": 3.1433829761637346, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.1788270473480225, + "step": 470, + "unclipped_grad_norm": 0.9839321970939636 + }, + { + "bits_per_byte": 3.1200604088802293, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.162661075592041, + "step": 480, + "unclipped_grad_norm": 1.0880863666534424 + }, + { + "bits_per_byte": 3.097947223597591, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.1473333835601807, + "step": 490, + "unclipped_grad_norm": 0.9460000395774841 + }, + { + "bits_per_byte": 3.049732575358806, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.1139135360717773, + "step": 500, + "unclipped_grad_norm": 1.3816719055175781 + }, + { + "bits_per_byte": 3.122747121852008, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.1645233631134033, + "step": 510, + "unclipped_grad_norm": 1.5173277854919434 + }, + { + "bits_per_byte": 3.1293818685569934, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.1691222190856934, + "step": 520, + "unclipped_grad_norm": 0.8952300548553467 + }, + { + "bits_per_byte": 2.987284642934615, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.0706279277801514, + "step": 530, + "unclipped_grad_norm": 1.0767766237258911 + }, + { + "bits_per_byte": 2.9921679183232137, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.0740127563476562, + "step": 540, + "unclipped_grad_norm": 0.9647241234779358 + }, + { + "bits_per_byte": 2.987321103256598, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.070653200149536, + "step": 550, + "unclipped_grad_norm": 0.7745878100395203 + }, + { + "bits_per_byte": 3.0156015863991765, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.0902557373046875, + "step": 560, + "unclipped_grad_norm": 1.306320309638977 + }, + { + "bits_per_byte": 2.9631527252961494, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.053900957107544, + "step": 570, + "unclipped_grad_norm": 0.9239676594734192 + }, + { + "bits_per_byte": 2.9684549504222497, + "learning_rate": 0.000299626473885547, + "loss_nats": 2.0575761795043945, + "step": 580, + "unclipped_grad_norm": 1.0172964334487915 + }, + { + "bits_per_byte": 2.914413186006931, + "learning_rate": 0.00029958384005397226, + "loss_nats": 2.0201172828674316, + "step": 590, + "unclipped_grad_norm": 1.2077929973602295 + }, + { + "bits_per_byte": 2.866491080257264, + "learning_rate": 0.0002995389065559004, + "loss_nats": 1.9869002103805542, + "step": 600, + "unclipped_grad_norm": 0.8019208908081055 + }, + { + "bits_per_byte": 2.8983864667383274, + "learning_rate": 0.000299491674159121, + "loss_nats": 2.0090084075927734, + "step": 610, + "unclipped_grad_norm": 1.3821065425872803 + }, + { + "bits_per_byte": 2.8774090548819875, + "learning_rate": 0.00029944214367070535, + "loss_nats": 1.9944679737091064, + "step": 620, + "unclipped_grad_norm": 1.0848106145858765 + }, + { + "bits_per_byte": 2.7905815498020883, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.93428373336792, + "step": 630, + "unclipped_grad_norm": 0.8434648513793945 + }, + { + "bits_per_byte": 2.8091942002090646, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.9471850395202637, + "step": 640, + "unclipped_grad_norm": 1.7573950290679932 + }, + { + "bits_per_byte": 2.792602001984424, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.9356842041015625, + "step": 650, + "unclipped_grad_norm": 1.4679571390151978 + }, + { + "bits_per_byte": 2.7952562102351903, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.9375239610671997, + "step": 660, + "unclipped_grad_norm": 1.1164823770523071 + }, + { + "bits_per_byte": 2.785295147079867, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.930619478225708, + "step": 670, + "unclipped_grad_norm": 1.137392520904541 + }, + { + "bits_per_byte": 2.7441477819129876, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.902098298072815, + "step": 680, + "unclipped_grad_norm": 0.8734592199325562 + }, + { + "bits_per_byte": 2.7310586983037717, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.8930256366729736, + "step": 690, + "unclipped_grad_norm": 0.9858389496803284 + }, + { + "bits_per_byte": 2.728585071836412, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.8913110494613647, + "step": 700, + "unclipped_grad_norm": 1.4443389177322388 + }, + { + "bits_per_byte": 2.6905791417701925, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.8649673461914062, + "step": 710, + "unclipped_grad_norm": 0.9958661794662476 + }, + { + "bits_per_byte": 2.705867883483944, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.875564694404602, + "step": 720, + "unclipped_grad_norm": 1.23285710811615 + }, + { + "bits_per_byte": 2.6709229006377804, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.8513426780700684, + "step": 730, + "unclipped_grad_norm": 1.1192034482955933 + }, + { + "bits_per_byte": 2.6287004719203573, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.8220763206481934, + "step": 740, + "unclipped_grad_norm": 1.1653774976730347 + }, + { + "bits_per_byte": 2.6386451967238487, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.8289694786071777, + "step": 750, + "unclipped_grad_norm": 0.8755648732185364 + }, + { + "bits_per_byte": 2.5565719800444007, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.772080659866333, + "step": 760, + "unclipped_grad_norm": 1.4887887239456177 + }, + { + "bits_per_byte": 2.569090253235403, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.7807576656341553, + "step": 770, + "unclipped_grad_norm": 0.9356563091278076 + }, + { + "bits_per_byte": 2.520539550896832, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.7471048831939697, + "step": 780, + "unclipped_grad_norm": 0.9265742897987366 + }, + { + "bits_per_byte": 2.516579478378065, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.7443599700927734, + "step": 790, + "unclipped_grad_norm": 1.0105865001678467 + }, + { + "bits_per_byte": 2.439560831807475, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.6909747123718262, + "step": 800, + "unclipped_grad_norm": 0.9081717133522034 + }, + { + "bits_per_byte": 2.4827769762815834, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7209298610687256, + "step": 810, + "unclipped_grad_norm": 1.0066841840744019 + }, + { + "bits_per_byte": 2.4593405564832067, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.7046849727630615, + "step": 820, + "unclipped_grad_norm": 0.9108051657676697 + }, + { + "bits_per_byte": 2.464562809676656, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.708304762840271, + "step": 830, + "unclipped_grad_norm": 0.9891977310180664 + }, + { + "bits_per_byte": 2.457934942277705, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.703710675239563, + "step": 840, + "unclipped_grad_norm": 1.14935302734375 + }, + { + "bits_per_byte": 2.3798235179435325, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.64956796169281, + "step": 850, + "unclipped_grad_norm": 1.021597981452942 + }, + { + "bits_per_byte": 2.448699473926372, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.697309136390686, + "step": 860, + "unclipped_grad_norm": 1.0554877519607544 + }, + { + "bits_per_byte": 2.4405493880846345, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.691659927368164, + "step": 870, + "unclipped_grad_norm": 1.2054848670959473 + }, + { + "bits_per_byte": 2.3861560911309523, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.6539573669433594, + "step": 880, + "unclipped_grad_norm": 0.8521014451980591 + }, + { + "bits_per_byte": 2.3284662307255317, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.6139698028564453, + "step": 890, + "unclipped_grad_norm": 0.9111976027488708 + }, + { + "bits_per_byte": 2.394481255311268, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.659727931022644, + "step": 900, + "unclipped_grad_norm": 1.0503509044647217 + }, + { + "bits_per_byte": 2.360193246191401, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.6359612941741943, + "step": 910, + "unclipped_grad_norm": 1.0133107900619507 + }, + { + "bits_per_byte": 2.3543332813285525, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.6318994760513306, + "step": 920, + "unclipped_grad_norm": 0.8582392930984497 + }, + { + "bits_per_byte": 2.2954381665058854, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.5910764932632446, + "step": 930, + "unclipped_grad_norm": 1.212068796157837 + }, + { + "bits_per_byte": 2.3388321410409856, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.6211549043655396, + "step": 940, + "unclipped_grad_norm": 0.8815436363220215 + }, + { + "bits_per_byte": 2.36333846491038, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.638141393661499, + "step": 950, + "unclipped_grad_norm": 0.9888424873352051 + }, + { + "bits_per_byte": 2.305671994145477, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.5981700420379639, + "step": 960, + "unclipped_grad_norm": 1.0667964220046997 + }, + { + "bits_per_byte": 2.2414655391162137, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5536655187606812, + "step": 970, + "unclipped_grad_norm": 1.0290824174880981 + }, + { + "bits_per_byte": 2.389095274634198, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.6559946537017822, + "step": 980, + "unclipped_grad_norm": 1.004488229751587 + }, + { + "bits_per_byte": 2.2314580406451574, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.5467288494110107, + "step": 990, + "unclipped_grad_norm": 0.9296519756317139 + }, + { + "bits_per_byte": 2.2316207362328737, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.5468416213989258, + "step": 1000, + "unclipped_grad_norm": 0.9619799256324768 + }, + { + "bits_per_byte": 2.232828914355185, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.54767906665802, + "step": 1010, + "unclipped_grad_norm": 0.9616303443908691 + }, + { + "bits_per_byte": 2.196473157823994, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.5224791765213013, + "step": 1020, + "unclipped_grad_norm": 0.9614624381065369 + }, + { + "bits_per_byte": 2.197589497210745, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.5232529640197754, + "step": 1030, + "unclipped_grad_norm": 0.8855695128440857 + }, + { + "bits_per_byte": 2.2920181195108276, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.5887058973312378, + "step": 1040, + "unclipped_grad_norm": 1.023490071296692 + }, + { + "bits_per_byte": 2.1928355527755947, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5199577808380127, + "step": 1050, + "unclipped_grad_norm": 0.9809900522232056 + }, + { + "bits_per_byte": 2.225750968358926, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.5427730083465576, + "step": 1060, + "unclipped_grad_norm": 0.8663541674613953 + }, + { + "bits_per_byte": 2.2728384423039563, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.5754115581512451, + "step": 1070, + "unclipped_grad_norm": 0.9538322687149048 + }, + { + "bits_per_byte": 2.2517033223567746, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.56076180934906, + "step": 1080, + "unclipped_grad_norm": 1.0214945077896118 + }, + { + "bits_per_byte": 2.1790556148578633, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.510406255722046, + "step": 1090, + "unclipped_grad_norm": 0.9988153576850891 + }, + { + "bits_per_byte": 2.191657127651883, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5191409587860107, + "step": 1100, + "unclipped_grad_norm": 0.9650821685791016 + }, + { + "bits_per_byte": 2.1293075693039856, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.4759235382080078, + "step": 1110, + "unclipped_grad_norm": 1.123518943786621 + }, + { + "bits_per_byte": 2.1815233939150938, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.51211678981781, + "step": 1120, + "unclipped_grad_norm": 0.9894219040870667 + }, + { + "bits_per_byte": 2.2029309343812415, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.5269553661346436, + "step": 1130, + "unclipped_grad_norm": 0.7673927545547485 + }, + { + "bits_per_byte": 2.122561377841241, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.4712474346160889, + "step": 1140, + "unclipped_grad_norm": 0.9915676116943359 + }, + { + "bits_per_byte": 2.1845762579505594, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.514232873916626, + "step": 1150, + "unclipped_grad_norm": 0.9133602976799011 + }, + { + "bits_per_byte": 2.095483397393495, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.4524784088134766, + "step": 1160, + "unclipped_grad_norm": 0.8488816022872925 + }, + { + "bits_per_byte": 2.1551027311589457, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.4938033819198608, + "step": 1170, + "unclipped_grad_norm": 0.8874648213386536 + }, + { + "bits_per_byte": 2.1490795547604282, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.4896284341812134, + "step": 1180, + "unclipped_grad_norm": 0.8033651113510132 + }, + { + "bits_per_byte": 2.116765906472466, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.4672303199768066, + "step": 1190, + "unclipped_grad_norm": 0.9034426808357239 + }, + { + "bits_per_byte": 2.1365709125978745, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.480958104133606, + "step": 1200, + "unclipped_grad_norm": 0.8197165131568909 + }, + { + "bits_per_byte": 2.150597989584896, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.4906809329986572, + "step": 1210, + "unclipped_grad_norm": 0.934878945350647 + }, + { + "bits_per_byte": 2.1005948937597885, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4560214281082153, + "step": 1220, + "unclipped_grad_norm": 0.809873104095459 + }, + { + "bits_per_byte": 2.1189614369933807, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.468752145767212, + "step": 1230, + "unclipped_grad_norm": 0.8442389369010925 + }, + { + "bits_per_byte": 2.1036283237557067, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.458124041557312, + "step": 1240, + "unclipped_grad_norm": 1.0173314809799194 + }, + { + "bits_per_byte": 2.015139122303171, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.3967880010604858, + "step": 1250, + "unclipped_grad_norm": 0.8291606307029724 + }, + { + "bits_per_byte": 2.073891663508273, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.4375121593475342, + "step": 1260, + "unclipped_grad_norm": 0.959783136844635 + }, + { + "bits_per_byte": 2.1003090585940547, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.455823302268982, + "step": 1270, + "unclipped_grad_norm": 0.8533931374549866 + }, + { + "bits_per_byte": 2.134419409635581, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.4794667959213257, + "step": 1280, + "unclipped_grad_norm": 0.8699606657028198 + }, + { + "bits_per_byte": 2.072432390715702, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4365006685256958, + "step": 1290, + "unclipped_grad_norm": 0.7947449088096619 + }, + { + "bits_per_byte": 2.0300015190254332, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.4070898294448853, + "step": 1300, + "unclipped_grad_norm": 0.818013608455658 + }, + { + "bits_per_byte": 2.0570910223107868, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.4258668422698975, + "step": 1310, + "unclipped_grad_norm": 0.97151118516922 + }, + { + "bits_per_byte": 2.040350403058452, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.414263129234314, + "step": 1320, + "unclipped_grad_norm": 0.8405282497406006 + }, + { + "bits_per_byte": 2.0419696197163253, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.4153854846954346, + "step": 1330, + "unclipped_grad_norm": 0.9559363126754761 + }, + { + "bits_per_byte": 2.0423992323781808, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.4156832695007324, + "step": 1340, + "unclipped_grad_norm": 0.8715131878852844 + }, + { + "bits_per_byte": 2.053634514993746, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.4234709739685059, + "step": 1350, + "unclipped_grad_norm": 0.7974268198013306 + }, + { + "bits_per_byte": 2.018255619919456, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.3989481925964355, + "step": 1360, + "unclipped_grad_norm": 0.8314212560653687 + }, + { + "bits_per_byte": 2.0558355489594886, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4249966144561768, + "step": 1370, + "unclipped_grad_norm": 0.9514848589897156 + }, + { + "bits_per_byte": 2.0135680607875392, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.3956990242004395, + "step": 1380, + "unclipped_grad_norm": 0.9944908618927002 + }, + { + "bits_per_byte": 2.061265385212529, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.428760290145874, + "step": 1390, + "unclipped_grad_norm": 0.7803435325622559 + }, + { + "bits_per_byte": 2.1674244281800115, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.5023441314697266, + "step": 1400, + "unclipped_grad_norm": 0.8709531426429749 + }, + { + "bits_per_byte": 2.0337632955477556, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.4096972942352295, + "step": 1410, + "unclipped_grad_norm": 0.767693817615509 + }, + { + "bits_per_byte": 2.0457202173663416, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.417985200881958, + "step": 1420, + "unclipped_grad_norm": 0.7609179019927979 + }, + { + "bits_per_byte": 1.9944559727624571, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.3824515342712402, + "step": 1430, + "unclipped_grad_norm": 0.873127818107605 + }, + { + "bits_per_byte": 2.0188871402134243, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.399385929107666, + "step": 1440, + "unclipped_grad_norm": 0.8898225426673889 + }, + { + "bits_per_byte": 1.9799427008214465, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.372391700744629, + "step": 1450, + "unclipped_grad_norm": 0.7384155988693237 + }, + { + "bits_per_byte": 2.1192472721591145, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.4689502716064453, + "step": 1460, + "unclipped_grad_norm": 0.9373573064804077 + }, + { + "bits_per_byte": 2.088472352648422, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.4476187229156494, + "step": 1470, + "unclipped_grad_norm": 0.7584673762321472 + }, + { + "bits_per_byte": 2.027154002275098, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.405116081237793, + "step": 1480, + "unclipped_grad_norm": 0.733923614025116 + }, + { + "bits_per_byte": 2.0522404236258525, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.4225046634674072, + "step": 1490, + "unclipped_grad_norm": 1.1139967441558838 + }, + { + "bits_per_byte": 1.9184776492469364, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.3297873735427856, + "step": 1500, + "unclipped_grad_norm": 0.779400110244751 + }, + { + "bits_per_byte": 1.942636224196286, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.3465328216552734, + "step": 1510, + "unclipped_grad_norm": 0.8135190010070801 + }, + { + "bits_per_byte": 1.9983152634478185, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.3851265907287598, + "step": 1520, + "unclipped_grad_norm": 0.7962142825126648 + }, + { + "bits_per_byte": 2.0252962456804777, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.4038283824920654, + "step": 1530, + "unclipped_grad_norm": 0.7650490999221802 + }, + { + "bits_per_byte": 2.0162607931520986, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.397565484046936, + "step": 1540, + "unclipped_grad_norm": 0.8527859449386597 + }, + { + "bits_per_byte": 2.065826021148109, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.4319214820861816, + "step": 1550, + "unclipped_grad_norm": 0.8282930254936218 + }, + { + "bits_per_byte": 1.934683402455088, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.3410203456878662, + "step": 1560, + "unclipped_grad_norm": 0.7039616107940674 + }, + { + "bits_per_byte": 1.9524430189137851, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.353330373764038, + "step": 1570, + "unclipped_grad_norm": 0.8072934746742249 + }, + { + "bits_per_byte": 1.9687994289240924, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.3646677732467651, + "step": 1580, + "unclipped_grad_norm": 0.8038614988327026 + }, + { + "bits_per_byte": 1.9835293990051903, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3748778104782104, + "step": 1590, + "unclipped_grad_norm": 0.8896909952163696 + }, + { + "bits_per_byte": 1.9517003978273595, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.3528156280517578, + "step": 1600, + "unclipped_grad_norm": 0.7759017944335938 + }, + { + "bits_per_byte": 1.9172675793154657, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.3289486169815063, + "step": 1610, + "unclipped_grad_norm": 0.7024519443511963 + }, + { + "bits_per_byte": 1.940532704393583, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.3450747728347778, + "step": 1620, + "unclipped_grad_norm": 0.7787846922874451 + }, + { + "bits_per_byte": 1.9715432401359578, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.3665696382522583, + "step": 1630, + "unclipped_grad_norm": 0.7470934391021729 + }, + { + "bits_per_byte": 1.935878337913283, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.341848611831665, + "step": 1640, + "unclipped_grad_norm": 0.9124332070350647 + }, + { + "bits_per_byte": 1.922601277266674, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.3326456546783447, + "step": 1650, + "unclipped_grad_norm": 0.7344956994056702 + }, + { + "bits_per_byte": 1.9518749602179852, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.3529366254806519, + "step": 1660, + "unclipped_grad_norm": 0.824038028717041 + }, + { + "bits_per_byte": 1.8903545302298974, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.3102939128875732, + "step": 1670, + "unclipped_grad_norm": 0.8769912719726562 + }, + { + "bits_per_byte": 1.9172161565028578, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.3289129734039307, + "step": 1680, + "unclipped_grad_norm": 0.6885949373245239 + }, + { + "bits_per_byte": 1.9556260738159539, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.355536699295044, + "step": 1690, + "unclipped_grad_norm": 0.8208780884742737 + }, + { + "bits_per_byte": 1.884375381390001, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.3061494827270508, + "step": 1700, + "unclipped_grad_norm": 0.7479352951049805 + }, + { + "bits_per_byte": 1.9522509142927713, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3531972169876099, + "step": 1710, + "unclipped_grad_norm": 0.7551807165145874 + }, + { + "bits_per_byte": 1.861953487249064, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.2906078100204468, + "step": 1720, + "unclipped_grad_norm": 0.9263664484024048 + }, + { + "bits_per_byte": 1.9520467708861973, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.353055715560913, + "step": 1730, + "unclipped_grad_norm": 0.7876571416854858 + }, + { + "bits_per_byte": 1.9559189602703733, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.355739712715149, + "step": 1740, + "unclipped_grad_norm": 0.7307856678962708 + }, + { + "bits_per_byte": 1.9186683780067433, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.3299195766448975, + "step": 1750, + "unclipped_grad_norm": 0.7305265069007874 + }, + { + "bits_per_byte": 1.9445503911003892, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.3478596210479736, + "step": 1760, + "unclipped_grad_norm": 0.7500777840614319 + }, + { + "bits_per_byte": 1.9404293428204145, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.3450031280517578, + "step": 1770, + "unclipped_grad_norm": 0.722295880317688 + }, + { + "bits_per_byte": 1.994646873504915, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.3825838565826416, + "step": 1780, + "unclipped_grad_norm": 0.8118055462837219 + }, + { + "bits_per_byte": 1.877266306533936, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3012218475341797, + "step": 1790, + "unclipped_grad_norm": 0.7757477760314941 + }, + { + "bits_per_byte": 2.0109250314090796, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.393867015838623, + "step": 1800, + "unclipped_grad_norm": 0.8003209233283997 + }, + { + "bits_per_byte": 1.8799791608686458, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.3031022548675537, + "step": 1810, + "unclipped_grad_norm": 0.8457723259925842 + }, + { + "bits_per_byte": 1.9361008834634994, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.3420028686523438, + "step": 1820, + "unclipped_grad_norm": 0.7704673409461975 + }, + { + "bits_per_byte": 1.9706171135610615, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.3659276962280273, + "step": 1830, + "unclipped_grad_norm": 0.7859576344490051 + }, + { + "bits_per_byte": 1.9457071464100923, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.3486614227294922, + "step": 1840, + "unclipped_grad_norm": 0.8324850797653198 + }, + { + "bits_per_byte": 1.8801090077700473, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.3031922578811646, + "step": 1850, + "unclipped_grad_norm": 0.7206872701644897 + }, + { + "bits_per_byte": 1.891747589701886, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.3112595081329346, + "step": 1860, + "unclipped_grad_norm": 0.6775411367416382 + }, + { + "bits_per_byte": 1.9766686670969713, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.3701223134994507, + "step": 1870, + "unclipped_grad_norm": 0.7119731903076172 + }, + { + "bits_per_byte": 1.884168314278362, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.3060059547424316, + "step": 1880, + "unclipped_grad_norm": 0.7594051361083984 + }, + { + "bits_per_byte": 1.9087649290394593, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3230550289154053, + "step": 1890, + "unclipped_grad_norm": 0.710192859172821 + }, + { + "bits_per_byte": 1.9218746505667788, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.3321419954299927, + "step": 1900, + "unclipped_grad_norm": 0.7209277749061584 + }, + { + "bits_per_byte": 1.9477162477374717, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.3500540256500244, + "step": 1910, + "unclipped_grad_norm": 0.7663269639015198 + }, + { + "bits_per_byte": 1.871915754282943, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.2975131273269653, + "step": 1920, + "unclipped_grad_norm": 0.772055447101593 + }, + { + "bits_per_byte": 1.921152323433155, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.3316413164138794, + "step": 1930, + "unclipped_grad_norm": 0.7362596988677979 + }, + { + "bits_per_byte": 1.9340063067586413, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.3405510187149048, + "step": 1940, + "unclipped_grad_norm": 0.7237302660942078 + }, + { + "bits_per_byte": 1.87446058156776, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.2992770671844482, + "step": 1950, + "unclipped_grad_norm": 0.7382659316062927 + }, + { + "bits_per_byte": 1.8619564109541287, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.2906098365783691, + "step": 1960, + "unclipped_grad_norm": 0.6941271424293518 + }, + { + "bits_per_byte": 1.8691700512619183, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.295609951019287, + "step": 1970, + "unclipped_grad_norm": 0.748450517654419 + }, + { + "bits_per_byte": 1.8888388471278434, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.3092433214187622, + "step": 1980, + "unclipped_grad_norm": 0.7122690677642822 + }, + { + "bits_per_byte": 1.8615305819105923, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.2903146743774414, + "step": 1990, + "unclipped_grad_norm": 0.7516839504241943 + }, + { + "bits_per_byte": 1.948412777473466, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.350536823272705, + "step": 2000, + "unclipped_grad_norm": 0.73175448179245 + }, + { + "bits_per_byte": 1.9454290504636471, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.3484686613082886, + "step": 2010, + "unclipped_grad_norm": 0.7510569095611572 + }, + { + "bits_per_byte": 1.8521819489749918, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.28383469581604, + "step": 2020, + "unclipped_grad_norm": 0.7182886600494385 + }, + { + "bits_per_byte": 1.866794970853503, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.2939636707305908, + "step": 2030, + "unclipped_grad_norm": 0.7515308856964111 + }, + { + "bits_per_byte": 1.8781530490817846, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.3018364906311035, + "step": 2040, + "unclipped_grad_norm": 0.717312216758728 + }, + { + "bits_per_byte": 1.8450453568947887, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.2788879871368408, + "step": 2050, + "unclipped_grad_norm": 0.7280142307281494 + }, + { + "bits_per_byte": 1.8191229278781902, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.2609199285507202, + "step": 2060, + "unclipped_grad_norm": 0.6870599985122681 + }, + { + "bits_per_byte": 1.944784975436166, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.348022222518921, + "step": 2070, + "unclipped_grad_norm": 0.7366598844528198 + }, + { + "bits_per_byte": 1.9226645668821916, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.3326895236968994, + "step": 2080, + "unclipped_grad_norm": 0.7822546362876892 + }, + { + "bits_per_byte": 1.8412273420456344, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.2762415409088135, + "step": 2090, + "unclipped_grad_norm": 0.6948415637016296 + }, + { + "bits_per_byte": 1.7741069849628026, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.2297172546386719, + "step": 2100, + "unclipped_grad_norm": 0.6751590371131897 + }, + { + "bits_per_byte": 1.8268810652586114, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.266297459602356, + "step": 2110, + "unclipped_grad_norm": 0.7959526777267456 + }, + { + "bits_per_byte": 1.8303243299115357, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.2686841487884521, + "step": 2120, + "unclipped_grad_norm": 0.7053372859954834 + }, + { + "bits_per_byte": 1.8791598355199355, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.3025343418121338, + "step": 2130, + "unclipped_grad_norm": 0.775160014629364 + }, + { + "bits_per_byte": 1.9031885635678836, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.3191897869110107, + "step": 2140, + "unclipped_grad_norm": 0.7063625454902649 + }, + { + "bits_per_byte": 1.8163512554768853, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2589987516403198, + "step": 2150, + "unclipped_grad_norm": 0.7413393259048462 + }, + { + "bits_per_byte": 1.8981219546734653, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.3156778812408447, + "step": 2160, + "unclipped_grad_norm": 0.7872826457023621 + }, + { + "bits_per_byte": 1.816059400918371, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.2587964534759521, + "step": 2170, + "unclipped_grad_norm": 0.7131244540214539 + }, + { + "bits_per_byte": 1.88075136297102, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.3036375045776367, + "step": 2180, + "unclipped_grad_norm": 0.6994345784187317 + }, + { + "bits_per_byte": 1.8732353771630135, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.2984278202056885, + "step": 2190, + "unclipped_grad_norm": 0.7243929505348206 + }, + { + "bits_per_byte": 1.8392175527876515, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.274848461151123, + "step": 2200, + "unclipped_grad_norm": 0.7677517533302307 + }, + { + "bits_per_byte": 1.8263757802303766, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.2659472227096558, + "step": 2210, + "unclipped_grad_norm": 0.6946780681610107 + }, + { + "bits_per_byte": 1.8265114745419073, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2660412788391113, + "step": 2220, + "unclipped_grad_norm": 0.6991197466850281 + }, + { + "bits_per_byte": 1.7854169080488427, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.2375566959381104, + "step": 2230, + "unclipped_grad_norm": 0.6679069399833679 + }, + { + "bits_per_byte": 1.8801078038914913, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.3031914234161377, + "step": 2240, + "unclipped_grad_norm": 0.6876413226127625 + }, + { + "bits_per_byte": 1.852575445280166, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2841074466705322, + "step": 2250, + "unclipped_grad_norm": 0.757318377494812 + }, + { + "bits_per_byte": 1.9019741940701413, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.3183480501174927, + "step": 2260, + "unclipped_grad_norm": 0.7510946989059448 + }, + { + "bits_per_byte": 1.789246101770303, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.2402108907699585, + "step": 2270, + "unclipped_grad_norm": 0.7097837924957275 + }, + { + "bits_per_byte": 1.8526122495674506, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.284132957458496, + "step": 2280, + "unclipped_grad_norm": 0.6768771409988403 + }, + { + "bits_per_byte": 1.809469025737308, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.2542283535003662, + "step": 2290, + "unclipped_grad_norm": 0.72145015001297 + }, + { + "bits_per_byte": 1.794269371036704, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.2436927556991577, + "step": 2300, + "unclipped_grad_norm": 0.6949226260185242 + }, + { + "bits_per_byte": 1.8055887531685415, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.2515387535095215, + "step": 2310, + "unclipped_grad_norm": 0.70436030626297 + }, + { + "bits_per_byte": 1.868977430692952, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.2954764366149902, + "step": 2320, + "unclipped_grad_norm": 0.7594719529151917 + }, + { + "bits_per_byte": 1.7922619895358334, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.242301344871521, + "step": 2330, + "unclipped_grad_norm": 0.7139930129051208 + }, + { + "bits_per_byte": 1.8139828823918533, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.257357120513916, + "step": 2340, + "unclipped_grad_norm": 0.7163569927215576 + }, + { + "bits_per_byte": 1.8694283692035143, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2957890033721924, + "step": 2350, + "unclipped_grad_norm": 0.7508989572525024 + }, + { + "bits_per_byte": 1.800800412203219, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.2482197284698486, + "step": 2360, + "unclipped_grad_norm": 0.7678603529930115 + }, + { + "bits_per_byte": 1.7359448946495997, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.2032653093338013, + "step": 2370, + "unclipped_grad_norm": 0.7002173662185669 + }, + { + "bits_per_byte": 1.8322637782653157, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.2700284719467163, + "step": 2380, + "unclipped_grad_norm": 0.7012119293212891 + }, + { + "bits_per_byte": 1.8257991224020336, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.265547513961792, + "step": 2390, + "unclipped_grad_norm": 0.6579994559288025 + }, + { + "bits_per_byte": 1.8631334602166338, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.2914257049560547, + "step": 2400, + "unclipped_grad_norm": 0.7961483597755432 + }, + { + "bits_per_byte": 1.8018949097933101, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2489783763885498, + "step": 2410, + "unclipped_grad_norm": 0.7746272087097168 + }, + { + "bits_per_byte": 1.8263054393261737, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.2658984661102295, + "step": 2420, + "unclipped_grad_norm": 0.755566418170929 + }, + { + "bits_per_byte": 1.839629451236468, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.2751339673995972, + "step": 2430, + "unclipped_grad_norm": 0.7083949446678162 + }, + { + "bits_per_byte": 1.7635778631116796, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.222419023513794, + "step": 2440, + "unclipped_grad_norm": 0.6766149401664734 + }, + { + "bits_per_byte": 1.8957399949590155, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.3140268325805664, + "step": 2450, + "unclipped_grad_norm": 0.7580212950706482 + }, + { + "bits_per_byte": 1.7801528630712338, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.23390793800354, + "step": 2460, + "unclipped_grad_norm": 0.7442304491996765 + }, + { + "bits_per_byte": 1.776217556054191, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.231180191040039, + "step": 2470, + "unclipped_grad_norm": 0.7592154741287231 + }, + { + "bits_per_byte": 1.8665510994545795, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.2937946319580078, + "step": 2480, + "unclipped_grad_norm": 0.7538792490959167 + }, + { + "bits_per_byte": 1.8518842470063483, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.2836283445358276, + "step": 2490, + "unclipped_grad_norm": 0.6985123753547668 + }, + { + "bits_per_byte": 1.8006941269249859, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2481460571289062, + "step": 2500, + "unclipped_grad_norm": 0.6396581530570984 + }, + { + "bits_per_byte": 1.7663020683013466, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.2243072986602783, + "step": 2510, + "unclipped_grad_norm": 0.6943463087081909 + }, + { + "bits_per_byte": 1.7859471305614527, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.2379242181777954, + "step": 2520, + "unclipped_grad_norm": 0.742425799369812 + }, + { + "bits_per_byte": 1.8146033957961665, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.2577872276306152, + "step": 2530, + "unclipped_grad_norm": 0.729271411895752 + }, + { + "bits_per_byte": 1.8070444143254443, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.2525477409362793, + "step": 2540, + "unclipped_grad_norm": 0.6923004388809204 + }, + { + "bits_per_byte": 1.7607227791247062, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.2204400300979614, + "step": 2550, + "unclipped_grad_norm": 0.6993430852890015 + }, + { + "bits_per_byte": 1.7299002204197245, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.19907546043396, + "step": 2560, + "unclipped_grad_norm": 0.6633782982826233 + }, + { + "bits_per_byte": 1.7508332607521422, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2135851383209229, + "step": 2570, + "unclipped_grad_norm": 0.6786762475967407 + }, + { + "bits_per_byte": 1.7533039635144372, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.2152976989746094, + "step": 2580, + "unclipped_grad_norm": 0.7012450695037842 + }, + { + "bits_per_byte": 1.7516618731639992, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.2141594886779785, + "step": 2590, + "unclipped_grad_norm": 0.6811529397964478 + }, + { + "bits_per_byte": 1.7663887475573814, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.224367380142212, + "step": 2600, + "unclipped_grad_norm": 0.688772976398468 + }, + { + "bits_per_byte": 1.8021886561609837, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.2491819858551025, + "step": 2610, + "unclipped_grad_norm": 0.7485259175300598 + }, + { + "bits_per_byte": 1.7182456441013554, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.1909971237182617, + "step": 2620, + "unclipped_grad_norm": 0.6495457291603088 + }, + { + "bits_per_byte": 1.7352368420759976, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.2027745246887207, + "step": 2630, + "unclipped_grad_norm": 0.6808093190193176 + }, + { + "bits_per_byte": 1.7014212692980504, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.179335355758667, + "step": 2640, + "unclipped_grad_norm": 0.682691752910614 + }, + { + "bits_per_byte": 1.7906155996191235, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2411601543426514, + "step": 2650, + "unclipped_grad_norm": 0.7770785689353943 + }, + { + "bits_per_byte": 1.6922384276378815, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.1729702949523926, + "step": 2660, + "unclipped_grad_norm": 0.6747376322746277 + }, + { + "bits_per_byte": 1.7668259274558746, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.22467041015625, + "step": 2670, + "unclipped_grad_norm": 0.6915087699890137 + }, + { + "bits_per_byte": 1.8654545380726781, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.293034553527832, + "step": 2680, + "unclipped_grad_norm": 0.729944109916687 + }, + { + "bits_per_byte": 1.7845595745342915, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.2369624376296997, + "step": 2690, + "unclipped_grad_norm": 0.7563508749008179 + }, + { + "bits_per_byte": 1.7057284028062585, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.1823208332061768, + "step": 2700, + "unclipped_grad_norm": 0.6877232789993286 + }, + { + "bits_per_byte": 1.7767699643287624, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.2315630912780762, + "step": 2710, + "unclipped_grad_norm": 0.6931856870651245 + }, + { + "bits_per_byte": 1.8387277461979943, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.2745089530944824, + "step": 2720, + "unclipped_grad_norm": 0.7578797340393066 + }, + { + "bits_per_byte": 1.7169079630429445, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.1900699138641357, + "step": 2730, + "unclipped_grad_norm": 0.674341082572937 + }, + { + "bits_per_byte": 1.799657071540283, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.247427225112915, + "step": 2740, + "unclipped_grad_norm": 0.7396203875541687 + }, + { + "bits_per_byte": 1.7517456287149693, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.2142175436019897, + "step": 2750, + "unclipped_grad_norm": 0.6496230363845825 + }, + { + "bits_per_byte": 1.867953274007064, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.2947665452957153, + "step": 2760, + "unclipped_grad_norm": 0.8206448554992676 + }, + { + "bits_per_byte": 1.7911425544613673, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.241525411605835, + "step": 2770, + "unclipped_grad_norm": 0.7205994725227356 + }, + { + "bits_per_byte": 1.816867891360077, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.2593568563461304, + "step": 2780, + "unclipped_grad_norm": 0.7271172404289246 + }, + { + "bits_per_byte": 1.7547021824659519, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2162668704986572, + "step": 2790, + "unclipped_grad_norm": 0.6990965604782104 + }, + { + "bits_per_byte": 1.7849284773203922, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.2372181415557861, + "step": 2800, + "unclipped_grad_norm": 0.716502845287323 + }, + { + "bits_per_byte": 1.70861100401737, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.1843189001083374, + "step": 2810, + "unclipped_grad_norm": 0.6464852094650269 + }, + { + "bits_per_byte": 1.793900812215905, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2434372901916504, + "step": 2820, + "unclipped_grad_norm": 0.7103920578956604 + }, + { + "bits_per_byte": 1.7296209205947233, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.1988818645477295, + "step": 2830, + "unclipped_grad_norm": 0.6629278659820557 + }, + { + "bits_per_byte": 1.7820828523792163, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.235245704650879, + "step": 2840, + "unclipped_grad_norm": 0.7148314714431763 + }, + { + "bits_per_byte": 1.701043079448803, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.1790732145309448, + "step": 2850, + "unclipped_grad_norm": 0.6930469870567322 + }, + { + "bits_per_byte": 1.7389011044352805, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.2053143978118896, + "step": 2860, + "unclipped_grad_norm": 0.7037895917892456 + }, + { + "bits_per_byte": 1.732908196983315, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.2011604309082031, + "step": 2870, + "unclipped_grad_norm": 0.6840003132820129 + }, + { + "bits_per_byte": 1.7525499915730547, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2147750854492188, + "step": 2880, + "unclipped_grad_norm": 0.6827396154403687 + }, + { + "bits_per_byte": 1.7033835913443949, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.1806955337524414, + "step": 2890, + "unclipped_grad_norm": 0.6346778869628906 + }, + { + "bits_per_byte": 1.7414430080150327, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.2070763111114502, + "step": 2900, + "unclipped_grad_norm": 0.6748653650283813 + }, + { + "bits_per_byte": 1.702566329787495, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.180129051208496, + "step": 2910, + "unclipped_grad_norm": 0.6806477308273315 + }, + { + "bits_per_byte": 1.7610091302383928, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.2206385135650635, + "step": 2920, + "unclipped_grad_norm": 0.6575880646705627 + }, + { + "bits_per_byte": 1.7111752653417345, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.1860963106155396, + "step": 2930, + "unclipped_grad_norm": 0.6358208060264587 + }, + { + "bits_per_byte": 1.7147781298946596, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.1885936260223389, + "step": 2940, + "unclipped_grad_norm": 0.685723602771759 + }, + { + "bits_per_byte": 1.7823922491681183, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2354601621627808, + "step": 2950, + "unclipped_grad_norm": 0.7353365421295166 + }, + { + "bits_per_byte": 1.7777860378300598, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.2322673797607422, + "step": 2960, + "unclipped_grad_norm": 0.7349781394004822 + }, + { + "bits_per_byte": 1.7572372067396698, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.2180240154266357, + "step": 2970, + "unclipped_grad_norm": 0.6859810948371887 + }, + { + "bits_per_byte": 1.7782865073440706, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.232614278793335, + "step": 2980, + "unclipped_grad_norm": 0.6771033406257629 + }, + { + "bits_per_byte": 1.715672611661797, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.1892136335372925, + "step": 2990, + "unclipped_grad_norm": 0.711289644241333 + }, + { + "bits_per_byte": 1.7986152006413563, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.2467050552368164, + "step": 3000, + "unclipped_grad_norm": 0.6865803003311157 + }, + { + "bits_per_byte": 1.6810693583428982, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.1652284860610962, + "step": 3010, + "unclipped_grad_norm": 0.6814430952072144 + }, + { + "bits_per_byte": 1.7627141662390466, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.22182035446167, + "step": 3020, + "unclipped_grad_norm": 0.6967481970787048 + }, + { + "bits_per_byte": 1.6877378136474528, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.1698507070541382, + "step": 3030, + "unclipped_grad_norm": 0.6906926035881042 + }, + { + "bits_per_byte": 1.6779291271207941, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.1630518436431885, + "step": 3040, + "unclipped_grad_norm": 0.6687314510345459 + }, + { + "bits_per_byte": 1.8226993071028823, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.2633988857269287, + "step": 3050, + "unclipped_grad_norm": 0.7547245621681213 + }, + { + "bits_per_byte": 1.764374830715778, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.2229714393615723, + "step": 3060, + "unclipped_grad_norm": 0.7102962732315063 + }, + { + "bits_per_byte": 1.673027793553855, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.1596544981002808, + "step": 3070, + "unclipped_grad_norm": 0.6981297135353088 + }, + { + "bits_per_byte": 1.7714842495371448, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.2278993129730225, + "step": 3080, + "unclipped_grad_norm": 0.6815842390060425 + }, + { + "bits_per_byte": 1.7457427462692539, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.2100566625595093, + "step": 3090, + "unclipped_grad_norm": 0.6700278520584106 + }, + { + "bits_per_byte": 1.7052649095621832, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.1819995641708374, + "step": 3100, + "unclipped_grad_norm": 0.6952568292617798 + }, + { + "bits_per_byte": 1.8498873564471803, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.2822442054748535, + "step": 3110, + "unclipped_grad_norm": 0.778131365776062 + }, + { + "bits_per_byte": 1.7310824291617555, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.199894905090332, + "step": 3120, + "unclipped_grad_norm": 0.7198062539100647 + }, + { + "bits_per_byte": 1.6679954092069582, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.1561663150787354, + "step": 3130, + "unclipped_grad_norm": 0.6881610751152039 + }, + { + "bits_per_byte": 1.7174983794833565, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.1904791593551636, + "step": 3140, + "unclipped_grad_norm": 0.7455202341079712 + }, + { + "bits_per_byte": 1.6904300300640591, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.1717168092727661, + "step": 3150, + "unclipped_grad_norm": 0.6183355450630188 + }, + { + "bits_per_byte": 1.7123161982475583, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.1868871450424194, + "step": 3160, + "unclipped_grad_norm": 0.687978982925415 + }, + { + "bits_per_byte": 1.7704819346479164, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.2272045612335205, + "step": 3170, + "unclipped_grad_norm": 0.7254087328910828 + }, + { + "bits_per_byte": 1.7492910923218554, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.212516188621521, + "step": 3180, + "unclipped_grad_norm": 0.7444673180580139 + }, + { + "bits_per_byte": 1.7348829017805218, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.2025291919708252, + "step": 3190, + "unclipped_grad_norm": 0.6939688920974731 + }, + { + "bits_per_byte": 1.7072838139006619, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.183398962020874, + "step": 3200, + "unclipped_grad_norm": 0.65944504737854 + }, + { + "bits_per_byte": 1.6983775203430804, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.1772255897521973, + "step": 3210, + "unclipped_grad_norm": 0.6629296541213989 + }, + { + "bits_per_byte": 1.6779934486322168, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.1630964279174805, + "step": 3220, + "unclipped_grad_norm": 0.6869114637374878 + }, + { + "bits_per_byte": 1.7325317269605764, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.2008994817733765, + "step": 3230, + "unclipped_grad_norm": 0.6663589477539062 + }, + { + "bits_per_byte": 1.7515870607108739, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.214107632637024, + "step": 3240, + "unclipped_grad_norm": 0.7918242812156677 + }, + { + "bits_per_byte": 1.710349920600244, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.1855242252349854, + "step": 3250, + "unclipped_grad_norm": 0.6762316823005676 + }, + { + "bits_per_byte": 1.7160965488961737, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.1895074844360352, + "step": 3260, + "unclipped_grad_norm": 0.7212422490119934 + }, + { + "bits_per_byte": 1.7369279474819315, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.2039467096328735, + "step": 3270, + "unclipped_grad_norm": 0.6828325390815735 + }, + { + "bits_per_byte": 1.755544553489878, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.216850757598877, + "step": 3280, + "unclipped_grad_norm": 0.7061439156532288 + }, + { + "bits_per_byte": 1.702583700035232, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.1801410913467407, + "step": 3290, + "unclipped_grad_norm": 0.7226800918579102 + }, + { + "bits_per_byte": 1.7062997291724247, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.1827168464660645, + "step": 3300, + "unclipped_grad_norm": 0.6950116157531738 + }, + { + "bits_per_byte": 1.6953563011153734, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.1751314401626587, + "step": 3310, + "unclipped_grad_norm": 0.6799318790435791 + }, + { + "bits_per_byte": 1.6966527063375771, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.1760300397872925, + "step": 3320, + "unclipped_grad_norm": 0.695996880531311 + }, + { + "bits_per_byte": 1.6994520679456713, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.1779704093933105, + "step": 3330, + "unclipped_grad_norm": 0.6822677254676819 + }, + { + "bits_per_byte": 1.638563846182155, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.1357659101486206, + "step": 3340, + "unclipped_grad_norm": 0.7137103080749512 + }, + { + "bits_per_byte": 1.6748845182525698, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.160941481590271, + "step": 3350, + "unclipped_grad_norm": 0.6746564507484436 + }, + { + "bits_per_byte": 1.7036979756301722, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.1809134483337402, + "step": 3360, + "unclipped_grad_norm": 0.6956479549407959 + }, + { + "bits_per_byte": 1.6626458888518707, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.1524583101272583, + "step": 3370, + "unclipped_grad_norm": 0.6469524502754211 + }, + { + "bits_per_byte": 1.6887040121799999, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.1705204248428345, + "step": 3380, + "unclipped_grad_norm": 0.6685478687286377 + }, + { + "bits_per_byte": 1.707417616403033, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.1834917068481445, + "step": 3390, + "unclipped_grad_norm": 0.7688858509063721 + }, + { + "bits_per_byte": 1.699863450446535, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.178255558013916, + "step": 3400, + "unclipped_grad_norm": 0.6692017912864685 + }, + { + "bits_per_byte": 1.6794356951423524, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.1640961170196533, + "step": 3410, + "unclipped_grad_norm": 0.6743058562278748 + }, + { + "bits_per_byte": 1.66517128209714, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.154208779335022, + "step": 3420, + "unclipped_grad_norm": 0.7006316184997559 + }, + { + "bits_per_byte": 1.7044275260351323, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1814191341400146, + "step": 3430, + "unclipped_grad_norm": 0.6554458737373352 + }, + { + "bits_per_byte": 1.7083179455803, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.1841157674789429, + "step": 3440, + "unclipped_grad_norm": 0.6529005765914917 + }, + { + "bits_per_byte": 1.7821077898635913, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.2352629899978638, + "step": 3450, + "unclipped_grad_norm": 0.7037137150764465 + }, + { + "bits_per_byte": 1.7706030104341237, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.2272884845733643, + "step": 3460, + "unclipped_grad_norm": 0.7114999294281006 + }, + { + "bits_per_byte": 1.741074277211583, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.2068207263946533, + "step": 3470, + "unclipped_grad_norm": 0.7143293619155884 + }, + { + "bits_per_byte": 1.713788541721595, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.1879076957702637, + "step": 3480, + "unclipped_grad_norm": 0.7445624470710754 + }, + { + "bits_per_byte": 1.6338074939898932, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.1324690580368042, + "step": 3490, + "unclipped_grad_norm": 0.6983089447021484 + }, + { + "bits_per_byte": 1.642275575753076, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1383386850357056, + "step": 3500, + "unclipped_grad_norm": 0.6774524450302124 + }, + { + "bits_per_byte": 1.694654783882504, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.174645185470581, + "step": 3510, + "unclipped_grad_norm": 0.714634358882904 + }, + { + "bits_per_byte": 1.6603831131144686, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.1508898735046387, + "step": 3520, + "unclipped_grad_norm": 0.6618956327438354 + }, + { + "bits_per_byte": 1.6681309315358381, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.1562602519989014, + "step": 3530, + "unclipped_grad_norm": 0.6907511949539185 + }, + { + "bits_per_byte": 1.6889970706170703, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.170723557472229, + "step": 3540, + "unclipped_grad_norm": 0.6799629330635071 + }, + { + "bits_per_byte": 1.7214059972936104, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.1931877136230469, + "step": 3550, + "unclipped_grad_norm": 0.7092427015304565 + }, + { + "bits_per_byte": 1.6594820960065984, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.1502653360366821, + "step": 3560, + "unclipped_grad_norm": 0.6571233868598938 + }, + { + "bits_per_byte": 1.7163211582382005, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.1896631717681885, + "step": 3570, + "unclipped_grad_norm": 0.7452999949455261 + }, + { + "bits_per_byte": 1.7727455702985726, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.228773593902588, + "step": 3580, + "unclipped_grad_norm": 0.7941396832466125 + }, + { + "bits_per_byte": 1.702409481609908, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.1800203323364258, + "step": 3590, + "unclipped_grad_norm": 0.7076472640037537 + }, + { + "bits_per_byte": 1.6799853516945098, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.1644771099090576, + "step": 3600, + "unclipped_grad_norm": 0.7179877758026123 + }, + { + "bits_per_byte": 1.6602706364608044, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.1508119106292725, + "step": 3610, + "unclipped_grad_norm": 0.6538001894950867 + }, + { + "bits_per_byte": 1.6646417475151336, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.1538417339324951, + "step": 3620, + "unclipped_grad_norm": 0.7317078113555908 + }, + { + "bits_per_byte": 1.691806407218914, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.172670841217041, + "step": 3630, + "unclipped_grad_norm": 0.7144780158996582 + }, + { + "bits_per_byte": 1.634294032909184, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.1328063011169434, + "step": 3640, + "unclipped_grad_norm": 0.7394706606864929 + }, + { + "bits_per_byte": 1.7127525182327974, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.1871895790100098, + "step": 3650, + "unclipped_grad_norm": 0.7001412510871887 + }, + { + "bits_per_byte": 1.6134873998251507, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.1183842420578003, + "step": 3660, + "unclipped_grad_norm": 0.7242536544799805 + }, + { + "bits_per_byte": 1.6276379603554916, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.128192663192749, + "step": 3670, + "unclipped_grad_norm": 0.6606935262680054 + }, + { + "bits_per_byte": 1.6941637734142907, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.1743048429489136, + "step": 3680, + "unclipped_grad_norm": 0.7007681727409363 + }, + { + "bits_per_byte": 1.65955415673731, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.150315284729004, + "step": 3690, + "unclipped_grad_norm": 0.6606731414794922 + }, + { + "bits_per_byte": 1.601972645401933, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.1104028224945068, + "step": 3700, + "unclipped_grad_norm": 0.6781966686248779 + }, + { + "bits_per_byte": 1.6705926912002884, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.1579666137695312, + "step": 3710, + "unclipped_grad_norm": 0.7430207133293152 + }, + { + "bits_per_byte": 1.6675221129517839, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.1558382511138916, + "step": 3720, + "unclipped_grad_norm": 0.6966750025749207 + }, + { + "bits_per_byte": 1.7169203457938067, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.1900784969329834, + "step": 3730, + "unclipped_grad_norm": 0.7088687419891357 + }, + { + "bits_per_byte": 1.647948251509135, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.1422706842422485, + "step": 3740, + "unclipped_grad_norm": 0.67493736743927 + }, + { + "bits_per_byte": 1.6302926845542098, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1300327777862549, + "step": 3750, + "unclipped_grad_norm": 0.7076202630996704 + }, + { + "bits_per_byte": 1.625935160129299, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.1270123720169067, + "step": 3760, + "unclipped_grad_norm": 0.7026046514511108 + }, + { + "bits_per_byte": 1.6823263795380545, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.1660997867584229, + "step": 3770, + "unclipped_grad_norm": 0.6940341591835022 + }, + { + "bits_per_byte": 1.6300350545432174, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.1298542022705078, + "step": 3780, + "unclipped_grad_norm": 0.6567591428756714 + }, + { + "bits_per_byte": 1.6707065437151596, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.1580455303192139, + "step": 3790, + "unclipped_grad_norm": 0.6438860893249512 + }, + { + "bits_per_byte": 1.6207524629455483, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.123420000076294, + "step": 3800, + "unclipped_grad_norm": 0.7110574841499329 + }, + { + "bits_per_byte": 1.6212701307246453, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.1237788200378418, + "step": 3810, + "unclipped_grad_norm": 0.693913459777832 + }, + { + "bits_per_byte": 1.7239960560156042, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1949830055236816, + "step": 3820, + "unclipped_grad_norm": 0.7051780223846436 + }, + { + "bits_per_byte": 1.6371957241945414, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.1348176002502441, + "step": 3830, + "unclipped_grad_norm": 0.7036773562431335 + }, + { + "bits_per_byte": 1.6366801202072547, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.134460210800171, + "step": 3840, + "unclipped_grad_norm": 0.7120051383972168 + }, + { + "bits_per_byte": 1.68266036984603, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1663312911987305, + "step": 3850, + "unclipped_grad_norm": 0.7358297109603882 + }, + { + "bits_per_byte": 1.623509172856228, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.1253308057785034, + "step": 3860, + "unclipped_grad_norm": 0.7661007642745972 + }, + { + "bits_per_byte": 1.6792086780432134, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.1639387607574463, + "step": 3870, + "unclipped_grad_norm": 0.7317741513252258 + }, + { + "bits_per_byte": 1.6338644202473287, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1325085163116455, + "step": 3880, + "unclipped_grad_norm": 0.732395350933075 + }, + { + "bits_per_byte": 1.673113096948683, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.159713625907898, + "step": 3890, + "unclipped_grad_norm": 0.7054989337921143 + }, + { + "bits_per_byte": 1.656017677487618, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.1478639841079712, + "step": 3900, + "unclipped_grad_norm": 0.7187857627868652 + }, + { + "bits_per_byte": 1.6837639825166166, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1670962572097778, + "step": 3910, + "unclipped_grad_norm": 0.7014420628547668 + }, + { + "bits_per_byte": 1.6495148414758443, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.1433565616607666, + "step": 3920, + "unclipped_grad_norm": 0.6903359889984131 + }, + { + "bits_per_byte": 1.6226803884617202, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.1247563362121582, + "step": 3930, + "unclipped_grad_norm": 0.7052710652351379 + }, + { + "bits_per_byte": 1.6317954689574492, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.1310744285583496, + "step": 3940, + "unclipped_grad_norm": 0.7147054672241211 + }, + { + "bits_per_byte": 1.7156851663953099, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1892223358154297, + "step": 3950, + "unclipped_grad_norm": 0.740327000617981 + }, + { + "bits_per_byte": 1.715005834924402, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.188751459121704, + "step": 3960, + "unclipped_grad_norm": 0.7351598739624023 + }, + { + "bits_per_byte": 1.692333018095856, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1730358600616455, + "step": 3970, + "unclipped_grad_norm": 0.7323389053344727 + }, + { + "bits_per_byte": 1.6663987222763479, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.155059576034546, + "step": 3980, + "unclipped_grad_norm": 0.7073854207992554 + }, + { + "bits_per_byte": 1.7083384115157525, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1841299533843994, + "step": 3990, + "unclipped_grad_norm": 0.706539511680603 + }, + { + "bits_per_byte": 1.6987269890896335, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.1774678230285645, + "step": 4000, + "unclipped_grad_norm": 0.7108194231987 + }, + { + "bits_per_byte": 1.7080341742063763, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.183919072151184, + "step": 4010, + "unclipped_grad_norm": 0.7395947575569153 + }, + { + "bits_per_byte": 1.5881593428499339, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.1008281707763672, + "step": 4020, + "unclipped_grad_norm": 0.6720413565635681 + }, + { + "bits_per_byte": 1.6311694521083084, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.1306405067443848, + "step": 4030, + "unclipped_grad_norm": 0.7062451839447021 + }, + { + "bits_per_byte": 1.6259845191500966, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1270465850830078, + "step": 4040, + "unclipped_grad_norm": 0.703016996383667 + }, + { + "bits_per_byte": 1.62622839054902, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.1272156238555908, + "step": 4050, + "unclipped_grad_norm": 0.7064345479011536 + }, + { + "bits_per_byte": 1.6446391333238837, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.139976978302002, + "step": 4060, + "unclipped_grad_norm": 0.7367279529571533 + }, + { + "bits_per_byte": 1.61614435979833, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.1202259063720703, + "step": 4070, + "unclipped_grad_norm": 0.7090343236923218 + }, + { + "bits_per_byte": 1.6452950751542745, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.1404316425323486, + "step": 4080, + "unclipped_grad_norm": 0.7135412693023682 + }, + { + "bits_per_byte": 1.594535255665371, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.1052476167678833, + "step": 4090, + "unclipped_grad_norm": 0.7324045300483704 + }, + { + "bits_per_byte": 1.5804180597692972, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.0954623222351074, + "step": 4100, + "unclipped_grad_norm": 0.6755955815315247 + }, + { + "bits_per_byte": 1.6819213603952725, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.1658190488815308, + "step": 4110, + "unclipped_grad_norm": 0.7271538376808167 + }, + { + "bits_per_byte": 1.6281143522983816, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.1285228729248047, + "step": 4120, + "unclipped_grad_norm": 0.7020561099052429 + }, + { + "bits_per_byte": 1.6030465050739204, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.111147165298462, + "step": 4130, + "unclipped_grad_norm": 0.6916652917861938 + }, + { + "bits_per_byte": 1.6260969958037608, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.127124547958374, + "step": 4140, + "unclipped_grad_norm": 0.7178624272346497 + }, + { + "bits_per_byte": 1.66229693605327, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1522164344787598, + "step": 4150, + "unclipped_grad_norm": 0.7373496890068054 + }, + { + "bits_per_byte": 1.6359983809792344, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.1339876651763916, + "step": 4160, + "unclipped_grad_norm": 0.7243641018867493 + }, + { + "bits_per_byte": 1.5862039001096238, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.0994727611541748, + "step": 4170, + "unclipped_grad_norm": 0.7503612637519836 + }, + { + "bits_per_byte": 1.573626120921423, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.090754508972168, + "step": 4180, + "unclipped_grad_norm": 0.7170337438583374 + }, + { + "bits_per_byte": 1.71302012323754, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.1873750686645508, + "step": 4190, + "unclipped_grad_norm": 0.7240293025970459 + }, + { + "bits_per_byte": 1.61204016581814, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1173810958862305, + "step": 4200, + "unclipped_grad_norm": 0.7353889346122742 + }, + { + "bits_per_byte": 1.6176657183278624, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.1212804317474365, + "step": 4210, + "unclipped_grad_norm": 0.7255803942680359 + }, + { + "bits_per_byte": 1.6676867003486595, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.1559523344039917, + "step": 4220, + "unclipped_grad_norm": 0.7396818995475769 + }, + { + "bits_per_byte": 1.635438749433327, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.1335997581481934, + "step": 4230, + "unclipped_grad_norm": 0.6928678154945374 + }, + { + "bits_per_byte": 1.6800538007895534, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.1645245552062988, + "step": 4240, + "unclipped_grad_norm": 0.7652084827423096 + }, + { + "bits_per_byte": 1.695609975525396, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.175307273864746, + "step": 4250, + "unclipped_grad_norm": 0.7278174161911011 + }, + { + "bits_per_byte": 1.6773165249184208, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1626272201538086, + "step": 4260, + "unclipped_grad_norm": 0.7652578353881836 + }, + { + "bits_per_byte": 1.6427101759118063, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.1386399269104004, + "step": 4270, + "unclipped_grad_norm": 0.7085094451904297 + }, + { + "bits_per_byte": 1.6708496332806775, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1581447124481201, + "step": 4280, + "unclipped_grad_norm": 0.7225905060768127 + }, + { + "bits_per_byte": 1.6421914762368042, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.1382803916931152, + "step": 4290, + "unclipped_grad_norm": 0.7149664759635925 + }, + { + "bits_per_byte": 1.5671523499776452, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.0862672328948975, + "step": 4300, + "unclipped_grad_norm": 0.747035801410675 + }, + { + "bits_per_byte": 1.7047494775575474, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.1816422939300537, + "step": 4310, + "unclipped_grad_norm": 0.8344160914421082 + }, + { + "bits_per_byte": 1.5834053984147842, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.0975329875946045, + "step": 4320, + "unclipped_grad_norm": 0.7258288264274597 + }, + { + "bits_per_byte": 1.5862886875564992, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.0995315313339233, + "step": 4330, + "unclipped_grad_norm": 0.6705095767974854 + }, + { + "bits_per_byte": 1.5959855853600968, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.106252908706665, + "step": 4340, + "unclipped_grad_norm": 0.7307996153831482 + }, + { + "bits_per_byte": 1.5162458292875556, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.0509815216064453, + "step": 4350, + "unclipped_grad_norm": 0.670310378074646 + }, + { + "bits_per_byte": 1.6824087592278176, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.1661568880081177, + "step": 4360, + "unclipped_grad_norm": 0.7400433421134949 + }, + { + "bits_per_byte": 1.6143763781474605, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.1190004348754883, + "step": 4370, + "unclipped_grad_norm": 0.7156907916069031 + }, + { + "bits_per_byte": 1.588213173419654, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.1008654832839966, + "step": 4380, + "unclipped_grad_norm": 0.7376554012298584 + }, + { + "bits_per_byte": 1.6062364392821238, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1133582592010498, + "step": 4390, + "unclipped_grad_norm": 0.6973915100097656 + }, + { + "bits_per_byte": 1.5779759061270453, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.0937695503234863, + "step": 4400, + "unclipped_grad_norm": 0.7829853892326355 + }, + { + "bits_per_byte": 1.5883323573967019, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.1009480953216553, + "step": 4410, + "unclipped_grad_norm": 0.7512783408164978 + }, + { + "bits_per_byte": 1.5805138541058277, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.0955287218093872, + "step": 4420, + "unclipped_grad_norm": 0.7326459884643555 + }, + { + "bits_per_byte": 1.587561187190233, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.1004135608673096, + "step": 4430, + "unclipped_grad_norm": 0.711580753326416 + }, + { + "bits_per_byte": 1.5918457909711783, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.1033834218978882, + "step": 4440, + "unclipped_grad_norm": 0.7045962810516357 + }, + { + "bits_per_byte": 1.5762326899778998, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.0925612449645996, + "step": 4450, + "unclipped_grad_norm": 0.7150362730026245 + }, + { + "bits_per_byte": 1.5507498486169067, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.0748978853225708, + "step": 4460, + "unclipped_grad_norm": 0.6975631713867188 + }, + { + "bits_per_byte": 1.6550151906157387, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1471691131591797, + "step": 4470, + "unclipped_grad_norm": 0.7837245464324951 + }, + { + "bits_per_byte": 1.5628410888858164, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.0832788944244385, + "step": 4480, + "unclipped_grad_norm": 0.7012354135513306 + }, + { + "bits_per_byte": 1.5268850200352306, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.0583560466766357, + "step": 4490, + "unclipped_grad_norm": 0.7023672461509705 + }, + { + "bits_per_byte": 1.6051426296226363, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.1126000881195068, + "step": 4500, + "unclipped_grad_norm": 0.7257990837097168 + }, + { + "bits_per_byte": 1.6039222407321139, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.1117541790008545, + "step": 4510, + "unclipped_grad_norm": 0.7441514730453491 + }, + { + "bits_per_byte": 1.5741592671390978, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.0911240577697754, + "step": 4520, + "unclipped_grad_norm": 0.7071590423583984 + }, + { + "bits_per_byte": 1.5861113734434595, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.0994086265563965, + "step": 4530, + "unclipped_grad_norm": 0.7145944237709045 + }, + { + "bits_per_byte": 1.5484860409835994, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.0733287334442139, + "step": 4540, + "unclipped_grad_norm": 0.7082009315490723 + }, + { + "bits_per_byte": 1.6243245426039687, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.1258959770202637, + "step": 4550, + "unclipped_grad_norm": 0.7257193922996521 + }, + { + "bits_per_byte": 1.636255839007576, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.1341661214828491, + "step": 4560, + "unclipped_grad_norm": 0.7607755064964294 + }, + { + "bits_per_byte": 1.562168292755641, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.0828125476837158, + "step": 4570, + "unclipped_grad_norm": 0.7378687858581543 + }, + { + "bits_per_byte": 1.5338551329093986, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.0631873607635498, + "step": 4580, + "unclipped_grad_norm": 0.7281880974769592 + }, + { + "bits_per_byte": 1.6492379494079552, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.1431646347045898, + "step": 4590, + "unclipped_grad_norm": 0.7851492762565613 + }, + { + "bits_per_byte": 1.6188895468714022, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.1221287250518799, + "step": 4600, + "unclipped_grad_norm": 0.7287366390228271 + }, + { + "bits_per_byte": 1.5871797296706194, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.100149154663086, + "step": 4610, + "unclipped_grad_norm": 0.7211630344390869 + }, + { + "bits_per_byte": 1.5282429950464436, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.0592973232269287, + "step": 4620, + "unclipped_grad_norm": 0.7168827056884766 + }, + { + "bits_per_byte": 1.6070251517189806, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.1139049530029297, + "step": 4630, + "unclipped_grad_norm": 0.734682023525238 + }, + { + "bits_per_byte": 1.5797129309007596, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.0949735641479492, + "step": 4640, + "unclipped_grad_norm": 0.7415030002593994 + }, + { + "bits_per_byte": 1.5466974214146263, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.0720889568328857, + "step": 4650, + "unclipped_grad_norm": 0.7430700659751892 + }, + { + "bits_per_byte": 1.5810812248710242, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.0959219932556152, + "step": 4660, + "unclipped_grad_norm": 0.7610417008399963 + }, + { + "bits_per_byte": 1.61094188460973, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.1166198253631592, + "step": 4670, + "unclipped_grad_norm": 0.7573997974395752 + }, + { + "bits_per_byte": 1.5733005577633399, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.0905288457870483, + "step": 4680, + "unclipped_grad_norm": 0.7657695412635803 + }, + { + "bits_per_byte": 1.5466183093952295, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.0720341205596924, + "step": 4690, + "unclipped_grad_norm": 0.7456971406936646 + }, + { + "bits_per_byte": 1.6421185555928384, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.1382298469543457, + "step": 4700, + "unclipped_grad_norm": 0.7399928569793701 + }, + { + "bits_per_byte": 1.5643676068948746, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.0843369960784912, + "step": 4710, + "unclipped_grad_norm": 0.7822249531745911 + }, + { + "bits_per_byte": 1.5271698233050595, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.0585534572601318, + "step": 4720, + "unclipped_grad_norm": 0.7349926233291626 + }, + { + "bits_per_byte": 1.6522901255128175, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.1452802419662476, + "step": 4730, + "unclipped_grad_norm": 0.7539346814155579 + }, + { + "bits_per_byte": 1.591391068842297, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.103068232536316, + "step": 4740, + "unclipped_grad_norm": 0.7638260722160339 + }, + { + "bits_per_byte": 1.6377562156537033, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.1352061033248901, + "step": 4750, + "unclipped_grad_norm": 0.791678786277771 + }, + { + "bits_per_byte": 1.5235296385168973, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.0560302734375, + "step": 4760, + "unclipped_grad_norm": 0.742056131362915 + }, + { + "bits_per_byte": 1.6220997750324075, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.1243538856506348, + "step": 4770, + "unclipped_grad_norm": 0.7348089814186096 + }, + { + "bits_per_byte": 1.5642214216416412, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.084235668182373, + "step": 4780, + "unclipped_grad_norm": 0.7018063068389893 + }, + { + "bits_per_byte": 1.599996908708821, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.1090333461761475, + "step": 4790, + "unclipped_grad_norm": 0.7742942571640015 + }, + { + "bits_per_byte": 1.5731965082595678, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.0904567241668701, + "step": 4800, + "unclipped_grad_norm": 0.7357653379440308 + }, + { + "bits_per_byte": 1.545303674012034, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.0711228847503662, + "step": 4810, + "unclipped_grad_norm": 0.7454521059989929 + }, + { + "bits_per_byte": 1.6152151375357184, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.1195818185806274, + "step": 4820, + "unclipped_grad_norm": 0.8018742203712463 + }, + { + "bits_per_byte": 1.5972073501118262, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.1070997714996338, + "step": 4830, + "unclipped_grad_norm": 0.7799034118652344 + }, + { + "bits_per_byte": 1.608833549292803, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.1151584386825562, + "step": 4840, + "unclipped_grad_norm": 0.7678232192993164 + }, + { + "bits_per_byte": 1.5401933815242965, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.0675806999206543, + "step": 4850, + "unclipped_grad_norm": 0.7227927446365356 + }, + { + "bits_per_byte": 1.559549684913604, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.0809974670410156, + "step": 4860, + "unclipped_grad_norm": 0.7678463459014893 + }, + { + "bits_per_byte": 1.554354776961642, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.0773966312408447, + "step": 4870, + "unclipped_grad_norm": 0.7430142164230347 + }, + { + "bits_per_byte": 1.7230030281895226, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.1942946910858154, + "step": 4880, + "unclipped_grad_norm": 0.8216635584831238 + }, + { + "bits_per_byte": 1.5145824130884105, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.0498285293579102, + "step": 4890, + "unclipped_grad_norm": 0.707672119140625 + }, + { + "bits_per_byte": 1.6109649302849458, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.116635799407959, + "step": 4900, + "unclipped_grad_norm": 0.7757799625396729 + }, + { + "bits_per_byte": 1.5986612914422205, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.108107566833496, + "step": 4910, + "unclipped_grad_norm": 0.7893052697181702 + }, + { + "bits_per_byte": 1.524328669912806, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.056584119796753, + "step": 4920, + "unclipped_grad_norm": 0.7137652635574341 + }, + { + "bits_per_byte": 1.5309622127392353, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.0611821413040161, + "step": 4930, + "unclipped_grad_norm": 0.6897748708724976 + }, + { + "bits_per_byte": 1.5586048122297638, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.0803425312042236, + "step": 4940, + "unclipped_grad_norm": 0.8026672005653381 + }, + { + "bits_per_byte": 1.5822476112091757, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.0967304706573486, + "step": 4950, + "unclipped_grad_norm": 0.7695291042327881 + }, + { + "bits_per_byte": 1.5960444034266918, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.1062936782836914, + "step": 4960, + "unclipped_grad_norm": 0.7572632431983948 + }, + { + "bits_per_byte": 1.5527746003655145, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.0763013362884521, + "step": 4970, + "unclipped_grad_norm": 0.7408527731895447 + }, + { + "bits_per_byte": 1.6221358913890886, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.1243789196014404, + "step": 4980, + "unclipped_grad_norm": 0.7835325002670288 + }, + { + "bits_per_byte": 1.5745128634692718, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.0913691520690918, + "step": 4990, + "unclipped_grad_norm": 0.7745388746261597 + }, + { + "bits_per_byte": 1.5787211069532339, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.0942860841751099, + "step": 5000, + "unclipped_grad_norm": 0.7623274922370911 + }, + { + "bits_per_byte": 1.6236954300671127, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.125459909439087, + "step": 5010, + "unclipped_grad_norm": 0.7771487832069397 + }, + { + "bits_per_byte": 1.571700603162363, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.0894198417663574, + "step": 5020, + "unclipped_grad_norm": 0.7782252430915833 + }, + { + "bits_per_byte": 1.5111066437144731, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.0474193096160889, + "step": 5030, + "unclipped_grad_norm": 0.6962376832962036 + }, + { + "bits_per_byte": 1.5788894779684286, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.09440279006958, + "step": 5040, + "unclipped_grad_norm": 0.763558566570282 + }, + { + "bits_per_byte": 1.5379975070380802, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.06605863571167, + "step": 5050, + "unclipped_grad_norm": 0.7815753221511841 + }, + { + "bits_per_byte": 1.6093620519789045, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.1155247688293457, + "step": 5060, + "unclipped_grad_norm": 0.7478274703025818 + }, + { + "bits_per_byte": 1.5594974021877417, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.0809612274169922, + "step": 5070, + "unclipped_grad_norm": 0.7443169355392456 + }, + { + "bits_per_byte": 1.6276885232548453, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.128227710723877, + "step": 5080, + "unclipped_grad_norm": 0.7993897199630737 + }, + { + "bits_per_byte": 1.5557554036702688, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.0783674716949463, + "step": 5090, + "unclipped_grad_norm": 0.7544270157814026 + }, + { + "bits_per_byte": 1.5420122700398216, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.0688414573669434, + "step": 5100, + "unclipped_grad_norm": 0.7675668001174927 + }, + { + "bits_per_byte": 1.5878975852553208, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.100646734237671, + "step": 5110, + "unclipped_grad_norm": 0.8086073398590088 + }, + { + "bits_per_byte": 1.490120460742939, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0328727960586548, + "step": 5120, + "unclipped_grad_norm": 0.7168148159980774 + }, + { + "bits_per_byte": 1.6268485599880314, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.127645492553711, + "step": 5130, + "unclipped_grad_norm": 0.7648565173149109 + }, + { + "bits_per_byte": 1.5892301068342058, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.1015703678131104, + "step": 5140, + "unclipped_grad_norm": 0.7664040923118591 + }, + { + "bits_per_byte": 1.5422606129876675, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.0690135955810547, + "step": 5150, + "unclipped_grad_norm": 0.7540826797485352 + }, + { + "bits_per_byte": 1.5575560620248023, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.079615592956543, + "step": 5160, + "unclipped_grad_norm": 0.7752884030342102 + }, + { + "bits_per_byte": 1.6103987633983055, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.1162433624267578, + "step": 5170, + "unclipped_grad_norm": 0.8419158458709717 + }, + { + "bits_per_byte": 1.569538437275716, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.087921142578125, + "step": 5180, + "unclipped_grad_norm": 0.783911943435669 + }, + { + "bits_per_byte": 1.4686841991740969, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.0180143117904663, + "step": 5190, + "unclipped_grad_norm": 0.7424947619438171 + }, + { + "bits_per_byte": 1.5454698092527674, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.0712380409240723, + "step": 5200, + "unclipped_grad_norm": 0.7612892985343933 + }, + { + "bits_per_byte": 1.5198113736052434, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.053452968597412, + "step": 5210, + "unclipped_grad_norm": 0.7524287104606628 + }, + { + "bits_per_byte": 1.535099599371042, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.0640499591827393, + "step": 5220, + "unclipped_grad_norm": 0.7870495319366455 + }, + { + "bits_per_byte": 1.59927114192218, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.1085302829742432, + "step": 5230, + "unclipped_grad_norm": 0.7915166616439819 + }, + { + "bits_per_byte": 1.5792341312007576, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0946416854858398, + "step": 5240, + "unclipped_grad_norm": 0.8550464510917664 + }, + { + "bits_per_byte": 1.5696868583034105, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0880240201950073, + "step": 5250, + "unclipped_grad_norm": 0.7794579267501831 + }, + { + "bits_per_byte": 1.5047501649385837, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.043013334274292, + "step": 5260, + "unclipped_grad_norm": 0.77144455909729 + }, + { + "bits_per_byte": 1.558467570074375, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.080247402191162, + "step": 5270, + "unclipped_grad_norm": 0.7968428730964661 + }, + { + "bits_per_byte": 1.5069816398335287, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.0445600748062134, + "step": 5280, + "unclipped_grad_norm": 0.8603785037994385 + }, + { + "bits_per_byte": 1.516083477665141, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.0508689880371094, + "step": 5290, + "unclipped_grad_norm": 0.7612236738204956 + }, + { + "bits_per_byte": 1.5016152651786565, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.0408403873443604, + "step": 5300, + "unclipped_grad_norm": 0.823062539100647 + }, + { + "bits_per_byte": 1.49876757644567, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.0388665199279785, + "step": 5310, + "unclipped_grad_norm": 0.7297431230545044 + }, + { + "bits_per_byte": 1.5199682217828303, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0535616874694824, + "step": 5320, + "unclipped_grad_norm": 0.7908921241760254 + }, + { + "bits_per_byte": 1.5529090907984893, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.0763945579528809, + "step": 5330, + "unclipped_grad_norm": 0.7975842356681824 + }, + { + "bits_per_byte": 1.622400056740814, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.1245620250701904, + "step": 5340, + "unclipped_grad_norm": 0.824028730392456 + }, + { + "bits_per_byte": 1.4850545397791242, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.029361367225647, + "step": 5350, + "unclipped_grad_norm": 0.768308699131012 + }, + { + "bits_per_byte": 1.5148875103110413, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.0500400066375732, + "step": 5360, + "unclipped_grad_norm": 0.7570281624794006 + }, + { + "bits_per_byte": 1.5959381181484587, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.106220006942749, + "step": 5370, + "unclipped_grad_norm": 0.8137096166610718 + }, + { + "bits_per_byte": 1.514726190584532, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.0499281883239746, + "step": 5380, + "unclipped_grad_norm": 0.7670685648918152 + }, + { + "bits_per_byte": 1.5036640944983852, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.0422605276107788, + "step": 5390, + "unclipped_grad_norm": 0.7815292477607727 + }, + { + "bits_per_byte": 1.56020270303893, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.08145010471344, + "step": 5400, + "unclipped_grad_norm": 0.8050987720489502 + }, + { + "bits_per_byte": 1.465311791373328, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.015676736831665, + "step": 5410, + "unclipped_grad_norm": 0.7766015529632568 + }, + { + "bits_per_byte": 1.6169760678979026, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.120802402496338, + "step": 5420, + "unclipped_grad_norm": 0.8383106589317322 + }, + { + "bits_per_byte": 1.5418623011682693, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.068737506866455, + "step": 5430, + "unclipped_grad_norm": 0.7962902188301086 + }, + { + "bits_per_byte": 1.533710667482674, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.0630872249603271, + "step": 5440, + "unclipped_grad_norm": 0.7506768107414246 + }, + { + "bits_per_byte": 1.5536898920334064, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.0769357681274414, + "step": 5450, + "unclipped_grad_norm": 0.7962613701820374 + }, + { + "bits_per_byte": 1.5897008233496173, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.1018966436386108, + "step": 5460, + "unclipped_grad_norm": 0.810023844242096 + }, + { + "bits_per_byte": 1.544394917684875, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0704929828643799, + "step": 5470, + "unclipped_grad_norm": 0.7931766510009766 + }, + { + "bits_per_byte": 1.4817567724488296, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.0270755290985107, + "step": 5480, + "unclipped_grad_norm": 0.7624560594558716 + }, + { + "bits_per_byte": 1.5425423205697808, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.0692088603973389, + "step": 5490, + "unclipped_grad_norm": 0.8255633115768433 + }, + { + "bits_per_byte": 1.5607936354272947, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.0818597078323364, + "step": 5500, + "unclipped_grad_norm": 0.7694008946418762 + }, + { + "bits_per_byte": 1.5467926998032042, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.0721549987792969, + "step": 5510, + "unclipped_grad_norm": 0.8112620115280151 + }, + { + "bits_per_byte": 1.5456720608501822, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.071378231048584, + "step": 5520, + "unclipped_grad_norm": 0.8474589586257935 + }, + { + "bits_per_byte": 1.554130167619615, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.0772409439086914, + "step": 5530, + "unclipped_grad_norm": 0.7864557504653931 + }, + { + "bits_per_byte": 1.592154327846826, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.1035972833633423, + "step": 5540, + "unclipped_grad_norm": 0.8304275870323181 + }, + { + "bits_per_byte": 1.5296828337994668, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.0602953433990479, + "step": 5550, + "unclipped_grad_norm": 0.7590808272361755 + }, + { + "bits_per_byte": 1.5210346862008306, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.0543009042739868, + "step": 5560, + "unclipped_grad_norm": 0.7927854657173157 + }, + { + "bits_per_byte": 1.5755475110968624, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.0920863151550293, + "step": 5570, + "unclipped_grad_norm": 0.8423728346824646 + }, + { + "bits_per_byte": 1.470332308917315, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.0191566944122314, + "step": 5580, + "unclipped_grad_norm": 0.7690621614456177 + }, + { + "bits_per_byte": 1.5324204536359012, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.0621929168701172, + "step": 5590, + "unclipped_grad_norm": 0.8284049034118652 + }, + { + "bits_per_byte": 1.527149185386956, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.0585391521453857, + "step": 5600, + "unclipped_grad_norm": 0.7675554156303406 + }, + { + "bits_per_byte": 1.5376208650326906, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.0657975673675537, + "step": 5610, + "unclipped_grad_norm": 0.794812798500061 + }, + { + "bits_per_byte": 1.4823258630405347, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.0274699926376343, + "step": 5620, + "unclipped_grad_norm": 0.7831655144691467 + }, + { + "bits_per_byte": 1.484573160339359, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0290277004241943, + "step": 5630, + "unclipped_grad_norm": 0.7998319268226624 + }, + { + "bits_per_byte": 1.46376738716858, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.014606237411499, + "step": 5640, + "unclipped_grad_norm": 0.7806196212768555 + }, + { + "bits_per_byte": 1.5825248472423665, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.0969226360321045, + "step": 5650, + "unclipped_grad_norm": 0.8651771545410156 + }, + { + "bits_per_byte": 1.540739254458135, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.0679590702056885, + "step": 5660, + "unclipped_grad_norm": 0.7913312315940857 + }, + { + "bits_per_byte": 1.523426104961078, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.0559585094451904, + "step": 5670, + "unclipped_grad_norm": 0.7947834730148315 + }, + { + "bits_per_byte": 1.5327657947988336, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0624322891235352, + "step": 5680, + "unclipped_grad_norm": 0.8122745156288147 + }, + { + "bits_per_byte": 1.5735372058909272, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.0906928777694702, + "step": 5690, + "unclipped_grad_norm": 0.8397650122642517 + }, + { + "bits_per_byte": 1.5103817368410866, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.0469168424606323, + "step": 5700, + "unclipped_grad_norm": 0.8185856938362122 + }, + { + "bits_per_byte": 1.5500077434784336, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.0743834972381592, + "step": 5710, + "unclipped_grad_norm": 0.799609363079071 + }, + { + "bits_per_byte": 1.5679782106670885, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0868396759033203, + "step": 5720, + "unclipped_grad_norm": 0.8444621562957764 + }, + { + "bits_per_byte": 1.4725883773313333, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.0207204818725586, + "step": 5730, + "unclipped_grad_norm": 0.8228028416633606 + }, + { + "bits_per_byte": 1.5415226354328153, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.0685020685195923, + "step": 5740, + "unclipped_grad_norm": 0.8184993267059326 + }, + { + "bits_per_byte": 1.5100396633485202, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.0466797351837158, + "step": 5750, + "unclipped_grad_norm": 0.826136589050293 + }, + { + "bits_per_byte": 1.4968293319704462, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.0375230312347412, + "step": 5760, + "unclipped_grad_norm": 0.8505411744117737 + }, + { + "bits_per_byte": 1.466365013127212, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.016406774520874, + "step": 5770, + "unclipped_grad_norm": 0.7837311625480652 + }, + { + "bits_per_byte": 1.544782222614618, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.0707614421844482, + "step": 5780, + "unclipped_grad_norm": 0.7713605761528015 + }, + { + "bits_per_byte": 1.4965718739421046, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 1.0373445749282837, + "step": 5790, + "unclipped_grad_norm": 0.7760441303253174 + }, + { + "bits_per_byte": 1.5005422654199234, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.040096640586853, + "step": 5800, + "unclipped_grad_norm": 0.8235262632369995 + }, + { + "bits_per_byte": 1.5234587816647418, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.055981159210205, + "step": 5810, + "unclipped_grad_norm": 0.7965627908706665 + }, + { + "bits_per_byte": 1.5230797319022402, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.0557184219360352, + "step": 5820, + "unclipped_grad_norm": 0.8052477240562439 + }, + { + "bits_per_byte": 1.4965534717984623, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.0373318195343018, + "step": 5830, + "unclipped_grad_norm": 0.8209868669509888 + }, + { + "bits_per_byte": 1.4688030391858429, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.018096685409546, + "step": 5840, + "unclipped_grad_norm": 0.8250015377998352 + }, + { + "bits_per_byte": 1.4644099143522034, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.0150516033172607, + "step": 5850, + "unclipped_grad_norm": 0.789702296257019 + }, + { + "bits_per_byte": 1.5003106047892112, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.0399360656738281, + "step": 5860, + "unclipped_grad_norm": 0.8312520980834961 + }, + { + "bits_per_byte": 1.547288353802991, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0724985599517822, + "step": 5870, + "unclipped_grad_norm": 0.8582806587219238 + }, + { + "bits_per_byte": 1.5568450857461353, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.07912278175354, + "step": 5880, + "unclipped_grad_norm": 0.8060404062271118 + }, + { + "bits_per_byte": 1.5306435288871865, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.0609612464904785, + "step": 5890, + "unclipped_grad_norm": 0.8349488973617554 + }, + { + "bits_per_byte": 1.510061333162529, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.0466947555541992, + "step": 5900, + "unclipped_grad_norm": 0.8609278202056885 + }, + { + "bits_per_byte": 1.5008079786155064, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.040280818939209, + "step": 5910, + "unclipped_grad_norm": 0.8607969284057617 + }, + { + "bits_per_byte": 1.5046342486319022, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.0429329872131348, + "step": 5920, + "unclipped_grad_norm": 0.866912305355072 + }, + { + "bits_per_byte": 1.5553089367086288, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.0780580043792725, + "step": 5930, + "unclipped_grad_norm": 0.8537927269935608 + }, + { + "bits_per_byte": 1.5512890142273614, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.0752716064453125, + "step": 5940, + "unclipped_grad_norm": 0.8828614950180054 + }, + { + "bits_per_byte": 1.5302648230899867, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.0606987476348877, + "step": 5950, + "unclipped_grad_norm": 0.8564047813415527 + }, + { + "bits_per_byte": 1.517704586132174, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.051992654800415, + "step": 5960, + "unclipped_grad_norm": 0.8294140696525574 + }, + { + "bits_per_byte": 1.5227837497601053, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.0555132627487183, + "step": 5970, + "unclipped_grad_norm": 0.8628323674201965 + }, + { + "bits_per_byte": 1.4738115179442697, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.0215682983398438, + "step": 5980, + "unclipped_grad_norm": 0.8240107893943787 + }, + { + "bits_per_byte": 1.4540943949534522, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0079014301300049, + "step": 5990, + "unclipped_grad_norm": 0.8460492491722107 + }, + { + "bits_per_byte": 1.5271196043710076, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.058518648147583, + "step": 6000, + "unclipped_grad_norm": 0.8315730094909668 + }, + { + "bits_per_byte": 1.5098174617636055, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0465257167816162, + "step": 6010, + "unclipped_grad_norm": 0.8524832129478455 + }, + { + "bits_per_byte": 1.4802374777111076, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0260224342346191, + "step": 6020, + "unclipped_grad_norm": 0.7963995337486267 + }, + { + "bits_per_byte": 1.5432371304792665, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.069690465927124, + "step": 6030, + "unclipped_grad_norm": 0.8414739966392517 + }, + { + "bits_per_byte": 1.4839968464763178, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.0286282300949097, + "step": 6040, + "unclipped_grad_norm": 0.8407770395278931 + }, + { + "bits_per_byte": 1.5055753376974235, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.0435853004455566, + "step": 6050, + "unclipped_grad_norm": 0.8566180467605591 + }, + { + "bits_per_byte": 1.5383170508033834, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.0662801265716553, + "step": 6060, + "unclipped_grad_norm": 0.8632009029388428 + }, + { + "bits_per_byte": 1.5072059052102538, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 1.0447155237197876, + "step": 6070, + "unclipped_grad_norm": 0.8236702084541321 + }, + { + "bits_per_byte": 1.4889342963999381, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.032050609588623, + "step": 6080, + "unclipped_grad_norm": 0.8699520826339722 + }, + { + "bits_per_byte": 1.49392987645955, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.0355132818222046, + "step": 6090, + "unclipped_grad_norm": 0.8228705525398254 + }, + { + "bits_per_byte": 1.4927075957598681, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.0346660614013672, + "step": 6100, + "unclipped_grad_norm": 0.7981337308883667 + }, + { + "bits_per_byte": 1.466816123620425, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.0167194604873657, + "step": 6110, + "unclipped_grad_norm": 0.8225575685501099 + }, + { + "bits_per_byte": 1.472174071125405, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0204333066940308, + "step": 6120, + "unclipped_grad_norm": 0.828834056854248 + }, + { + "bits_per_byte": 1.5135465615822639, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.0491105318069458, + "step": 6130, + "unclipped_grad_norm": 0.9135096669197083 + }, + { + "bits_per_byte": 1.537121427414585, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.0654513835906982, + "step": 6140, + "unclipped_grad_norm": 0.8410868644714355 + }, + { + "bits_per_byte": 1.4654332111248372, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.015760898590088, + "step": 6150, + "unclipped_grad_norm": 0.8130381107330322 + }, + { + "bits_per_byte": 1.5610335512252482, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 1.0820260047912598, + "step": 6160, + "unclipped_grad_norm": 0.8469728231430054 + }, + { + "bits_per_byte": 1.428924905982331, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 0.9904552698135376, + "step": 6170, + "unclipped_grad_norm": 0.7920248508453369 + }, + { + "bits_per_byte": 1.4972811303942628, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0378361940383911, + "step": 6180, + "unclipped_grad_norm": 0.8312132358551025 + }, + { + "bits_per_byte": 1.5007237071165838, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.040222406387329, + "step": 6190, + "unclipped_grad_norm": 0.8636819124221802 + }, + { + "bits_per_byte": 1.4630880556976718, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.0141353607177734, + "step": 6200, + "unclipped_grad_norm": 0.8371090888977051 + }, + { + "bits_per_byte": 1.480064119199038, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.025902271270752, + "step": 6210, + "unclipped_grad_norm": 0.8231096863746643 + }, + { + "bits_per_byte": 1.4642454989379785, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0149376392364502, + "step": 6220, + "unclipped_grad_norm": 0.7931125164031982 + }, + { + "bits_per_byte": 1.4462203412663492, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.002443552017212, + "step": 6230, + "unclipped_grad_norm": 0.827806830406189 + }, + { + "bits_per_byte": 1.4693497720329358, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.0184756517410278, + "step": 6240, + "unclipped_grad_norm": 0.8484618067741394 + }, + { + "bits_per_byte": 1.4599823929883913, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.0119826793670654, + "step": 6250, + "unclipped_grad_norm": 0.8607235550880432 + }, + { + "bits_per_byte": 1.5265252323296257, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 1.0581066608428955, + "step": 6260, + "unclipped_grad_norm": 0.8813891410827637 + }, + { + "bits_per_byte": 1.4863702070582245, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.0302733182907104, + "step": 6270, + "unclipped_grad_norm": 0.8320657014846802 + }, + { + "bits_per_byte": 1.5771629441364168, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.0932060480117798, + "step": 6280, + "unclipped_grad_norm": 0.9096292853355408 + }, + { + "bits_per_byte": 1.4605702296890404, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 1.01239013671875, + "step": 6290, + "unclipped_grad_norm": 0.8256506323814392 + }, + { + "bits_per_byte": 1.5054986334351388, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.043532133102417, + "step": 6300, + "unclipped_grad_norm": 0.8480525612831116 + }, + { + "bits_per_byte": 1.530533803955936, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.0608851909637451, + "step": 6310, + "unclipped_grad_norm": 0.8752952218055725 + }, + { + "bits_per_byte": 1.490752153019558, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 1.0333106517791748, + "step": 6320, + "unclipped_grad_norm": 0.8671753406524658 + }, + { + "bits_per_byte": 1.4320883548623013, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 0.9926480054855347, + "step": 6330, + "unclipped_grad_norm": 0.8273700475692749 + }, + { + "bits_per_byte": 1.503170332307758, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 1.0419182777404785, + "step": 6340, + "unclipped_grad_norm": 0.8535099625587463 + }, + { + "bits_per_byte": 1.4947909935924202, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.0361101627349854, + "step": 6350, + "unclipped_grad_norm": 0.8587582111358643 + }, + { + "bits_per_byte": 1.4430169924113787, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.000223159790039, + "step": 6360, + "unclipped_grad_norm": 0.8159511089324951 + }, + { + "bits_per_byte": 1.5079970254042228, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 1.0452638864517212, + "step": 6370, + "unclipped_grad_norm": 0.9028456211090088 + }, + { + "bits_per_byte": 1.459066241407245, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.0113476514816284, + "step": 6380, + "unclipped_grad_norm": 0.8239596486091614 + }, + { + "bits_per_byte": 1.4987479704234719, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0388529300689697, + "step": 6390, + "unclipped_grad_norm": 0.8929301500320435 + }, + { + "bits_per_byte": 1.4856807286109155, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.0297954082489014, + "step": 6400, + "unclipped_grad_norm": 0.8465076088905334 + }, + { + "bits_per_byte": 1.4722609223640906, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.020493507385254, + "step": 6410, + "unclipped_grad_norm": 0.8801109790802002 + }, + { + "bits_per_byte": 1.4474988602928633, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0033297538757324, + "step": 6420, + "unclipped_grad_norm": 0.8338649272918701 + }, + { + "bits_per_byte": 1.47521885197628, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.0225437879562378, + "step": 6430, + "unclipped_grad_norm": 0.8950762152671814 + }, + { + "bits_per_byte": 1.4767813143593687, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.0236268043518066, + "step": 6440, + "unclipped_grad_norm": 0.8455358147621155 + }, + { + "bits_per_byte": 1.461988398628055, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0133731365203857, + "step": 6450, + "unclipped_grad_norm": 0.8632203340530396 + }, + { + "bits_per_byte": 1.4492522234184098, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.0045450925827026, + "step": 6460, + "unclipped_grad_norm": 1.074549913406372 + }, + { + "bits_per_byte": 1.4832698758111207, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.0281243324279785, + "step": 6470, + "unclipped_grad_norm": 0.8713725805282593 + }, + { + "bits_per_byte": 1.5020514131812446, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 1.0411427021026611, + "step": 6480, + "unclipped_grad_norm": 0.8867150545120239 + }, + { + "bits_per_byte": 1.4877496799007952, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0312294960021973, + "step": 6490, + "unclipped_grad_norm": 0.8849058151245117 + }, + { + "bits_per_byte": 1.4807867902979635, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.0264031887054443, + "step": 6500, + "unclipped_grad_norm": 0.8547548055648804 + }, + { + "bits_per_byte": 1.4419339316675701, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 0.999472439289093, + "step": 6510, + "unclipped_grad_norm": 0.815456748008728 + }, + { + "bits_per_byte": 1.3917721818470419, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 0.9647029638290405, + "step": 6520, + "unclipped_grad_norm": 0.842738151550293 + }, + { + "bits_per_byte": 1.5060266201732875, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.043898105621338, + "step": 6530, + "unclipped_grad_norm": 0.9138928651809692 + }, + { + "bits_per_byte": 1.4607893355862396, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.0125420093536377, + "step": 6540, + "unclipped_grad_norm": 0.8454874753952026 + }, + { + "bits_per_byte": 1.5088760287327825, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 1.0458731651306152, + "step": 6550, + "unclipped_grad_norm": 0.874169647693634 + }, + { + "bits_per_byte": 1.476713209229627, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.0235795974731445, + "step": 6560, + "unclipped_grad_norm": 0.8644656538963318 + }, + { + "bits_per_byte": 1.401555242958722, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 0.9714840650558472, + "step": 6570, + "unclipped_grad_norm": 0.8851974010467529 + }, + { + "bits_per_byte": 1.4843509587544443, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.0288736820220947, + "step": 6580, + "unclipped_grad_norm": 0.8569743633270264 + }, + { + "bits_per_byte": 1.4728688810348907, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.020914912223816, + "step": 6590, + "unclipped_grad_norm": 0.9000326991081238 + }, + { + "bits_per_byte": 1.3973316070275075, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9685564637184143, + "step": 6600, + "unclipped_grad_norm": 0.8176273107528687 + }, + { + "bits_per_byte": 1.5125024549088757, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.048386812210083, + "step": 6610, + "unclipped_grad_norm": 0.8706966042518616 + }, + { + "bits_per_byte": 1.5169427029888518, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 1.051464557647705, + "step": 6620, + "unclipped_grad_norm": 0.8870038986206055 + }, + { + "bits_per_byte": 1.5062022144598184, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.0440198183059692, + "step": 6630, + "unclipped_grad_norm": 0.8916358351707458 + }, + { + "bits_per_byte": 1.4749689611845764, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.0223705768585205, + "step": 6640, + "unclipped_grad_norm": 0.9267131686210632 + }, + { + "bits_per_byte": 1.441712847969886, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 0.9993191957473755, + "step": 6650, + "unclipped_grad_norm": 0.8645026087760925 + }, + { + "bits_per_byte": 1.5116846774040231, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.047819972038269, + "step": 6660, + "unclipped_grad_norm": 0.8734310865402222 + }, + { + "bits_per_byte": 1.4845685168077858, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.0290244817733765, + "step": 6670, + "unclipped_grad_norm": 0.8921259641647339 + }, + { + "bits_per_byte": 1.4975330849777768, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.038010835647583, + "step": 6680, + "unclipped_grad_norm": 0.9257161021232605 + }, + { + "bits_per_byte": 1.4613221378386125, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.012911319732666, + "step": 6690, + "unclipped_grad_norm": 0.8868398070335388 + }, + { + "bits_per_byte": 1.5080212749579944, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.0452806949615479, + "step": 6700, + "unclipped_grad_norm": 0.9118692278862 + }, + { + "bits_per_byte": 1.4510437666924476, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.0057868957519531, + "step": 6710, + "unclipped_grad_norm": 0.8764793276786804 + }, + { + "bits_per_byte": 1.4431949944550218, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.0003465414047241, + "step": 6720, + "unclipped_grad_norm": 0.8701503276824951 + }, + { + "bits_per_byte": 1.432308320672755, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 0.9928004741668701, + "step": 6730, + "unclipped_grad_norm": 0.9072649478912354 + }, + { + "bits_per_byte": 1.4651253621797926, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 1.015547513961792, + "step": 6740, + "unclipped_grad_norm": 0.8810498118400574 + }, + { + "bits_per_byte": 1.4246036698967521, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 0.9874600172042847, + "step": 6750, + "unclipped_grad_norm": 0.8828034996986389 + }, + { + "bits_per_byte": 1.4152031841919164, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 0.9809440970420837, + "step": 6760, + "unclipped_grad_norm": 0.8337298035621643 + }, + { + "bits_per_byte": 1.433150691696681, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 0.9933843612670898, + "step": 6770, + "unclipped_grad_norm": 0.8472889065742493 + }, + { + "bits_per_byte": 1.4394755256648117, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 0.9977684020996094, + "step": 6780, + "unclipped_grad_norm": 0.9014667272567749 + }, + { + "bits_per_byte": 1.5342224878516415, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.0634419918060303, + "step": 6790, + "unclipped_grad_norm": 0.8911973834037781 + }, + { + "bits_per_byte": 1.512064415097128, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0480831861495972, + "step": 6800, + "unclipped_grad_norm": 0.9676592946052551 + }, + { + "bits_per_byte": 1.488135952934633, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0314972400665283, + "step": 6810, + "unclipped_grad_norm": 0.9102917909622192 + }, + { + "bits_per_byte": 1.4292499531924616, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 0.9906805753707886, + "step": 6820, + "unclipped_grad_norm": 0.8522918820381165 + }, + { + "bits_per_byte": 1.456734672609498, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.0097315311431885, + "step": 6830, + "unclipped_grad_norm": 0.9276469945907593 + }, + { + "bits_per_byte": 1.4824316323708153, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 1.027543306350708, + "step": 6840, + "unclipped_grad_norm": 0.8685769438743591 + }, + { + "bits_per_byte": 1.4432789219886426, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.000404715538025, + "step": 6850, + "unclipped_grad_norm": 0.8829621076583862 + }, + { + "bits_per_byte": 1.4517922351890027, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.0063056945800781, + "step": 6860, + "unclipped_grad_norm": 0.8671189546585083 + }, + { + "bits_per_byte": 1.4771954485826464, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 1.023913860321045, + "step": 6870, + "unclipped_grad_norm": 0.9415659308433533 + }, + { + "bits_per_byte": 1.5107442762691052, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 1.0471681356430054, + "step": 6880, + "unclipped_grad_norm": 0.9159059524536133 + }, + { + "bits_per_byte": 1.403604588226403, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 0.9729045629501343, + "step": 6890, + "unclipped_grad_norm": 0.8648654222488403 + }, + { + "bits_per_byte": 1.4290599983545837, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 0.9905489087104797, + "step": 6900, + "unclipped_grad_norm": 0.8992753028869629 + }, + { + "bits_per_byte": 1.6027094190782292, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.1109135150909424, + "step": 6910, + "unclipped_grad_norm": 0.9864886999130249 + }, + { + "bits_per_byte": 1.4240951171981506, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 0.987107515335083, + "step": 6920, + "unclipped_grad_norm": 0.8857905864715576 + }, + { + "bits_per_byte": 1.4308625485182769, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.9917983412742615, + "step": 6930, + "unclipped_grad_norm": 0.9176548719406128 + }, + { + "bits_per_byte": 1.4697052601722693, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 1.0187220573425293, + "step": 6940, + "unclipped_grad_norm": 0.8998448252677917 + }, + { + "bits_per_byte": 1.4541215682122886, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 1.007920265197754, + "step": 6950, + "unclipped_grad_norm": 0.9375615119934082 + }, + { + "bits_per_byte": 1.4816054277160704, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 1.026970624923706, + "step": 6960, + "unclipped_grad_norm": 0.9188227653503418 + }, + { + "bits_per_byte": 1.469793315289511, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 1.0187830924987793, + "step": 6970, + "unclipped_grad_norm": 0.9197005033493042 + }, + { + "bits_per_byte": 1.5084147712631686, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.04555344581604, + "step": 6980, + "unclipped_grad_norm": 0.9444351196289062 + }, + { + "bits_per_byte": 1.466378427773979, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.016416072845459, + "step": 6990, + "unclipped_grad_norm": 0.8996309041976929 + }, + { + "bits_per_byte": 1.4121636488118925, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 0.978837251663208, + "step": 7000, + "unclipped_grad_norm": 0.8954524993896484 + }, + { + "bits_per_byte": 1.4552707562853537, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 1.0087168216705322, + "step": 7010, + "unclipped_grad_norm": 0.9172784090042114 + }, + { + "bits_per_byte": 1.4154891053489758, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 0.9811422824859619, + "step": 7020, + "unclipped_grad_norm": 0.8922209739685059 + }, + { + "bits_per_byte": 1.4280571675174027, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 0.9898537993431091, + "step": 7030, + "unclipped_grad_norm": 0.8567910194396973 + }, + { + "bits_per_byte": 1.5196896098884325, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.0533685684204102, + "step": 7040, + "unclipped_grad_norm": 0.978830873966217 + }, + { + "bits_per_byte": 1.475627654737381, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.0228271484375, + "step": 7050, + "unclipped_grad_norm": 0.9045489430427551 + }, + { + "bits_per_byte": 1.4846789296696397, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 1.029101014137268, + "step": 7060, + "unclipped_grad_norm": 0.981572687625885 + }, + { + "bits_per_byte": 1.4421326576206421, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 0.999610185623169, + "step": 7070, + "unclipped_grad_norm": 0.9034438729286194 + }, + { + "bits_per_byte": 1.3479317403503015, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 0.9343150854110718, + "step": 7080, + "unclipped_grad_norm": 0.894311249256134 + }, + { + "bits_per_byte": 1.423906452230154, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 0.9869767427444458, + "step": 7090, + "unclipped_grad_norm": 0.9467347860336304 + }, + { + "bits_per_byte": 1.3992750969735828, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 0.9699035882949829, + "step": 7100, + "unclipped_grad_norm": 0.8752198219299316 + }, + { + "bits_per_byte": 1.4687469728416618, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 1.0180578231811523, + "step": 7110, + "unclipped_grad_norm": 0.9269235730171204 + }, + { + "bits_per_byte": 1.4782280324184267, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.0246295928955078, + "step": 7120, + "unclipped_grad_norm": 0.9651775360107422 + }, + { + "bits_per_byte": 1.4157605799633628, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 0.9813304543495178, + "step": 7130, + "unclipped_grad_norm": 0.9211539626121521 + }, + { + "bits_per_byte": 1.5238395512537521, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.0562450885772705, + "step": 7140, + "unclipped_grad_norm": 0.967028021812439 + }, + { + "bits_per_byte": 1.5371535881702965, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 1.0654736757278442, + "step": 7150, + "unclipped_grad_norm": 0.9308844208717346 + }, + { + "bits_per_byte": 1.4517695334790888, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0062899589538574, + "step": 7160, + "unclipped_grad_norm": 0.958120584487915 + }, + { + "bits_per_byte": 1.451617500815726, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.0061845779418945, + "step": 7170, + "unclipped_grad_norm": 0.8951361775398254 + }, + { + "bits_per_byte": 1.3804372352853014, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 0.9568461775779724, + "step": 7180, + "unclipped_grad_norm": 0.861808180809021 + }, + { + "bits_per_byte": 1.3823468446491565, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 0.9581698179244995, + "step": 7190, + "unclipped_grad_norm": 0.8709425926208496 + }, + { + "bits_per_byte": 1.470358794245548, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 1.0191750526428223, + "step": 7200, + "unclipped_grad_norm": 0.9395002126693726 + }, + { + "bits_per_byte": 1.4741793888344652, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 1.0218232870101929, + "step": 7210, + "unclipped_grad_norm": 0.8699914813041687 + }, + { + "bits_per_byte": 1.4377833883629725, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 0.9965955018997192, + "step": 7220, + "unclipped_grad_norm": 0.9003393650054932 + }, + { + "bits_per_byte": 1.4334180387274473, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 0.9935696721076965, + "step": 7230, + "unclipped_grad_norm": 0.9436356425285339 + }, + { + "bits_per_byte": 1.4019962924668596, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 0.9717897772789001, + "step": 7240, + "unclipped_grad_norm": 0.8605899810791016 + }, + { + "bits_per_byte": 1.44070502963583, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 0.9986206293106079, + "step": 7250, + "unclipped_grad_norm": 0.9447914958000183 + }, + { + "bits_per_byte": 1.439576479480868, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 0.9978383779525757, + "step": 7260, + "unclipped_grad_norm": 0.9320752620697021 + }, + { + "bits_per_byte": 1.4894024331755868, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 1.0323750972747803, + "step": 7270, + "unclipped_grad_norm": 0.9317567348480225 + }, + { + "bits_per_byte": 1.4449125851891884, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.0015370845794678, + "step": 7280, + "unclipped_grad_norm": 0.9382103085517883 + }, + { + "bits_per_byte": 1.4531961295679956, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 1.0072788000106812, + "step": 7290, + "unclipped_grad_norm": 0.9036419987678528 + }, + { + "bits_per_byte": 1.4402098915839958, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 0.9982774257659912, + "step": 7300, + "unclipped_grad_norm": 0.930662214756012 + }, + { + "bits_per_byte": 1.4815598523135918, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 1.026939034461975, + "step": 7310, + "unclipped_grad_norm": 0.9238929748535156 + }, + { + "bits_per_byte": 1.471397913422061, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 1.019895315170288, + "step": 7320, + "unclipped_grad_norm": 0.9481417536735535 + }, + { + "bits_per_byte": 1.4661044594111547, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 1.0162261724472046, + "step": 7330, + "unclipped_grad_norm": 0.9988716244697571 + }, + { + "bits_per_byte": 1.437381034951279, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.9963166117668152, + "step": 7340, + "unclipped_grad_norm": 0.948367714881897 + }, + { + "bits_per_byte": 1.483790983243235, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 1.0284855365753174, + "step": 7350, + "unclipped_grad_norm": 0.9437121152877808 + }, + { + "bits_per_byte": 1.455043567203564, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 1.0085593461990356, + "step": 7360, + "unclipped_grad_norm": 0.9004260897636414 + }, + { + "bits_per_byte": 1.4613682291890437, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 1.0129432678222656, + "step": 7370, + "unclipped_grad_norm": 0.9012579321861267 + }, + { + "bits_per_byte": 1.5126371173245012, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 1.0484801530838013, + "step": 7380, + "unclipped_grad_norm": 0.9875184297561646 + }, + { + "bits_per_byte": 1.4880092017209472, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 1.0314093828201294, + "step": 7390, + "unclipped_grad_norm": 0.9245239496231079 + }, + { + "bits_per_byte": 1.4203576762209256, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.9845169186592102, + "step": 7400, + "unclipped_grad_norm": 0.9230930805206299 + }, + { + "bits_per_byte": 1.439935837229846, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 0.998087465763092, + "step": 7410, + "unclipped_grad_norm": 0.9358386993408203 + }, + { + "bits_per_byte": 1.5044543547790998, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0428082942962646, + "step": 7420, + "unclipped_grad_norm": 0.9640387892723083 + }, + { + "bits_per_byte": 1.4269394522694452, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 0.9890790581703186, + "step": 7430, + "unclipped_grad_norm": 0.986886203289032 + }, + { + "bits_per_byte": 1.451731697295899, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 1.0062637329101562, + "step": 7440, + "unclipped_grad_norm": 0.9600938558578491 + }, + { + "bits_per_byte": 1.3840149043798748, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 0.9593260288238525, + "step": 7450, + "unclipped_grad_norm": 0.9776710867881775 + }, + { + "bits_per_byte": 1.4117471928228282, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 0.9785485863685608, + "step": 7460, + "unclipped_grad_norm": 0.9104099869728088 + }, + { + "bits_per_byte": 1.4644616811301132, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 1.0150874853134155, + "step": 7470, + "unclipped_grad_norm": 0.9258345365524292 + }, + { + "bits_per_byte": 1.3958006174695268, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 0.9674952626228333, + "step": 7480, + "unclipped_grad_norm": 0.9340448379516602 + }, + { + "bits_per_byte": 1.4188382954918781, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 0.9834637641906738, + "step": 7490, + "unclipped_grad_norm": 0.9439316391944885 + }, + { + "bits_per_byte": 1.48245846166435, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.027561902999878, + "step": 7500, + "unclipped_grad_norm": 0.9230331778526306 + }, + { + "bits_per_byte": 1.470812656461175, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 1.0194896459579468, + "step": 7510, + "unclipped_grad_norm": 0.9684584140777588 + }, + { + "bits_per_byte": 1.4304457485639108, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 0.9915094375610352, + "step": 7520, + "unclipped_grad_norm": 0.9270921945571899 + }, + { + "bits_per_byte": 1.46936456254091, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.0184859037399292, + "step": 7530, + "unclipped_grad_norm": 0.9356089234352112 + }, + { + "bits_per_byte": 1.4997451258331744, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.0395441055297852, + "step": 7540, + "unclipped_grad_norm": 0.9384937882423401 + }, + { + "bits_per_byte": 1.4198691595011497, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 0.9841783046722412, + "step": 7550, + "unclipped_grad_norm": 0.9434585571289062 + }, + { + "bits_per_byte": 1.4646037387997257, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 1.0151859521865845, + "step": 7560, + "unclipped_grad_norm": 0.9542449712753296 + }, + { + "bits_per_byte": 1.4674665620059881, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 1.0171703100204468, + "step": 7570, + "unclipped_grad_norm": 0.931070864200592 + }, + { + "bits_per_byte": 1.4288311754376106, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 0.9903903007507324, + "step": 7580, + "unclipped_grad_norm": 0.9063198566436768 + }, + { + "bits_per_byte": 1.4370147978962666, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 0.9960627555847168, + "step": 7590, + "unclipped_grad_norm": 0.9074749946594238 + }, + { + "bits_per_byte": 1.3998171862891022, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 0.970279335975647, + "step": 7600, + "unclipped_grad_norm": 0.9185620546340942 + }, + { + "bits_per_byte": 1.3694875298614935, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 0.949256420135498, + "step": 7610, + "unclipped_grad_norm": 0.9378254413604736 + }, + { + "bits_per_byte": 1.465760322126778, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 1.0159876346588135, + "step": 7620, + "unclipped_grad_norm": 0.9552015662193298 + }, + { + "bits_per_byte": 1.4240919355191097, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 0.9871053099632263, + "step": 7630, + "unclipped_grad_norm": 0.9403524398803711 + }, + { + "bits_per_byte": 1.4139078968569432, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.980046272277832, + "step": 7640, + "unclipped_grad_norm": 0.932920515537262 + }, + { + "bits_per_byte": 1.5112564406033746, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 1.0475231409072876, + "step": 7650, + "unclipped_grad_norm": 1.002869963645935 + }, + { + "bits_per_byte": 1.448756741401274, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 1.0042016506195068, + "step": 7660, + "unclipped_grad_norm": 0.9678865671157837 + }, + { + "bits_per_byte": 1.41361827807289, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 0.9798455238342285, + "step": 7670, + "unclipped_grad_norm": 0.9324528574943542 + }, + { + "bits_per_byte": 1.39204589223589, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 0.9648926854133606, + "step": 7680, + "unclipped_grad_norm": 0.9411377906799316 + }, + { + "bits_per_byte": 1.4387559502536016, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 0.9972696304321289, + "step": 7690, + "unclipped_grad_norm": 0.930465579032898 + }, + { + "bits_per_byte": 1.4490628705198099, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 1.0044138431549072, + "step": 7700, + "unclipped_grad_norm": 1.0089294910430908 + }, + { + "bits_per_byte": 1.3723721948644154, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 0.9512559175491333, + "step": 7710, + "unclipped_grad_norm": 0.9348894357681274 + }, + { + "bits_per_byte": 1.3721833579137679, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 0.9511250257492065, + "step": 7720, + "unclipped_grad_norm": 0.906726062297821 + }, + { + "bits_per_byte": 1.435828461570615, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 0.9952404499053955, + "step": 7730, + "unclipped_grad_norm": 0.9059075117111206 + }, + { + "bits_per_byte": 1.4228678490015938, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 0.9862568378448486, + "step": 7740, + "unclipped_grad_norm": 0.9165619015693665 + }, + { + "bits_per_byte": 1.436527915011674, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9957252740859985, + "step": 7750, + "unclipped_grad_norm": 0.9504069089889526 + }, + { + "bits_per_byte": 1.4125389149560752, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 0.9790973663330078, + "step": 7760, + "unclipped_grad_norm": 0.9769607782363892 + }, + { + "bits_per_byte": 1.4097558916998132, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 0.9771683216094971, + "step": 7770, + "unclipped_grad_norm": 0.98009192943573 + }, + { + "bits_per_byte": 1.5439238572041618, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 1.0701664686203003, + "step": 7780, + "unclipped_grad_norm": 1.0303176641464233 + }, + { + "bits_per_byte": 1.4162461729780738, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 0.9816670417785645, + "step": 7790, + "unclipped_grad_norm": 0.9548059105873108 + }, + { + "bits_per_byte": 1.4585669757717903, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 1.0110015869140625, + "step": 7800, + "unclipped_grad_norm": 0.9941151738166809 + }, + { + "bits_per_byte": 1.393949654189616, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.966212272644043, + "step": 7810, + "unclipped_grad_norm": 0.9603933095932007 + }, + { + "bits_per_byte": 1.4039762427349176, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 0.9731621742248535, + "step": 7820, + "unclipped_grad_norm": 0.962434709072113 + }, + { + "bits_per_byte": 1.4612421659059613, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 1.012855887413025, + "step": 7830, + "unclipped_grad_norm": 0.9587858319282532 + }, + { + "bits_per_byte": 1.4172268180532936, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.982346773147583, + "step": 7840, + "unclipped_grad_norm": 0.8982013463973999 + }, + { + "bits_per_byte": 1.3995034899339285, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.9700618982315063, + "step": 7850, + "unclipped_grad_norm": 0.8969244956970215 + }, + { + "bits_per_byte": 1.396125148731705, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 0.9677202105522156, + "step": 7860, + "unclipped_grad_norm": 0.950086236000061 + }, + { + "bits_per_byte": 1.4527285087402997, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0069546699523926, + "step": 7870, + "unclipped_grad_norm": 0.9996808171272278 + }, + { + "bits_per_byte": 1.431299900399421, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9921014904975891, + "step": 7880, + "unclipped_grad_norm": 0.9725462198257446 + }, + { + "bits_per_byte": 1.4870098105367833, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 1.0307166576385498, + "step": 7890, + "unclipped_grad_norm": 0.9916442632675171 + }, + { + "bits_per_byte": 1.4671337755765685, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 1.016939640045166, + "step": 7900, + "unclipped_grad_norm": 0.9673606157302856 + }, + { + "bits_per_byte": 1.4691740057637541, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.018353819847107, + "step": 7910, + "unclipped_grad_norm": 1.0204181671142578 + }, + { + "bits_per_byte": 1.3751287327924444, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 0.9531666040420532, + "step": 7920, + "unclipped_grad_norm": 0.9728531837463379 + }, + { + "bits_per_byte": 1.3945271719312131, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.9666125774383545, + "step": 7930, + "unclipped_grad_norm": 0.965191125869751 + }, + { + "bits_per_byte": 1.4383021740293003, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 0.9969550967216492, + "step": 7940, + "unclipped_grad_norm": 1.0432325601577759 + }, + { + "bits_per_byte": 1.3947921971961927, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 0.9667962789535522, + "step": 7950, + "unclipped_grad_norm": 0.9746297597885132 + }, + { + "bits_per_byte": 1.4245434759689501, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 0.9874182939529419, + "step": 7960, + "unclipped_grad_norm": 0.9362404346466064 + }, + { + "bits_per_byte": 1.4076147076965713, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 0.9756841659545898, + "step": 7970, + "unclipped_grad_norm": 0.9664047360420227 + }, + { + "bits_per_byte": 1.434604375053099, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 0.9943919777870178, + "step": 7980, + "unclipped_grad_norm": 0.9625164866447449 + }, + { + "bits_per_byte": 1.4485152777594625, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 1.0040342807769775, + "step": 7990, + "unclipped_grad_norm": 0.9863426089286804 + }, + { + "bits_per_byte": 1.3826528877763669, + "learning_rate": 3e-05, + "loss_nats": 0.9583819508552551, + "step": 8000, + "unclipped_grad_norm": 0.9372552037239075 + } + ], + "variant": "uniform_group_7_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073001.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073001.json new file mode 100644 index 0000000..1a76a07 --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073001.json @@ -0,0 +1,14029 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "a0d9f32c901c9a4d8e3d5fe13d9db3aefbbae5fae13ba2ceb20aa3baa48991c7", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.0002677410375326872, + 0.0002473295899108052, + 0.00023034414334688336, + 0.00021438254043459892, + 0.00020044577831868082, + 0.00018819731485564262, + 0.00017973134526982903, + 0.000171217747265473, + 0.0001640921109355986, + 0.00016061548376455903, + 0.00015463090676348656, + 0.00015024171443656087, + 0.0001478576596127823, + 0.00014403100067283958, + 0.00013951722939964384, + 0.00013661097909789532, + 0.00013379570737015456, + 0.0001311724045081064, + 0.0001286817860091105, + 0.00012687484559137374, + 0.0001249351626029238, + 0.00012369138130452484, + 0.00012227990373503417, + 0.00012101328320568427, + 0.00011953213106608018, + 0.0001181897969217971, + 0.0001171672047348693, + 0.00011609455395955592, + 0.00011492414341773838, + 0.0001140913664130494, + 0.00011346775136189535, + 0.00011285000800853595 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00021242368711682502, + "first_to_last_ratio": 1.8345658968465297, + "imbalance_abs_log_ratio": 0.6068078850437921, + "last_quartile_mean": 0.0001157896194854402, + "mean": 0.0001511171253696375, + "normalized": [ + 1.7717451736709768, + 1.636674793183293, + 1.5242755762025908, + 1.4186515254985965, + 1.3264266232459345, + 1.2453738409548603, + 1.1893512719369177, + 1.1330135273991528, + 1.085860457802012, + 1.0628542818802846, + 1.023252039669589, + 0.9942070699734702, + 0.9784308644775869, + 0.9531083940389613, + 0.9232390376562555, + 0.9040072643238835, + 0.8853775311228679, + 0.8680181295617843, + 0.8515367513400655, + 0.8395795332993111, + 0.8267439067367663, + 0.8185133286645813, + 0.8091730400239778, + 0.800791325997512, + 0.7909899739933552, + 0.7821072339266707, + 0.7753403490721149, + 0.7682422073314642, + 0.7604971517068638, + 0.7549863467424895, + 0.7508596466770359, + 0.7467718018887739 + ], + "population_cv": 0.2714186609524326 + }, + "activation_output_rms_by_block": [ + 0.0034885562490671873, + 0.004831995815038681, + 0.005952076520770788, + 0.006925336085259914, + 0.00385080324485898, + 0.005542940925806761, + 0.006729288958013058, + 0.007927131839096546, + 0.004081308841705322, + 0.005811207927763462, + 0.007534808944910765, + 0.008581922389566898, + 0.00499920267611742, + 0.007102519739419222, + 0.008542952127754688, + 0.010447698645293713, + 0.005041283555328846, + 0.007636935915797949, + 0.009387043304741383, + 0.011020516976714134, + 0.005807018838822842, + 0.008367395959794521, + 0.01024632342159748, + 0.012378672137856483, + 0.006453148555010557, + 0.009127611294388771, + 0.011037957854568958, + 0.012611051090061665, + 0.007010402157902718, + 0.009794782847166061, + 0.012445803731679916, + 0.014087481424212456 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.0056560162047389895, + "first_to_last_ratio": 0.5480088979804595, + "imbalance_abs_log_ratio": 0.6014637549753238, + "last_quartile_mean": 0.010321029869373888, + "mean": 0.007962599374877755, + "normalized": [ + 0.4381177659237371, + 0.606836484864952, + 0.747504205668035, + 0.8697330807712821, + 0.4836113263475722, + 0.6961220406611074, + 0.845112084785299, + 0.9955457339856744, + 0.512559862622497, + 0.7298129234151854, + 0.9462750278110585, + 1.077778999737589, + 0.6278355146046978, + 0.8919850673170736, + 1.072884836415099, + 1.3120964842531853, + 0.6331203314377778, + 0.9591008609440675, + 1.1788918244911069, + 1.384035094304029, + 0.7292868277593074, + 1.0508372411895934, + 1.2868063479276561, + 1.554601902603762, + 0.8104324041933391, + 1.1463105029730196, + 1.3862254440922965, + 1.5837857082010076, + 0.8804162846646224, + 1.2300986632667847, + 1.5630327667805075, + 1.7692063559870779 + ], + "population_cv": 0.3451071044649734 + }, + "bits_per_byte": 7.959341947457508, + "branch_output_rms_by_sublayer": [ + 0.0022265110164880753, + 0.0027073738165199757, + 0.0021613112185150385, + 0.002722575329244137, + 0.0025601957459002733, + 0.0026850115973502398, + 0.0025323182344436646, + 0.0026861224323511124, + 0.0027253651060163975, + 0.00271838391199708, + 0.002904455177485943, + 0.0026645169127732515, + 0.0029208383057266474, + 0.0026859452482312918, + 0.00299624796025455, + 0.0027359756641089916, + 0.0030937574338167906, + 0.0026767742820084095, + 0.0033804215490818024, + 0.0027451682835817337, + 0.0034370317589491606, + 0.0026999011170119047, + 0.0038969130255281925, + 0.002663193503394723, + 0.004255065228790045, + 0.002626648638397455, + 0.004119740333408117, + 0.0026531771291047335, + 0.003970506135374308, + 0.0026517712976783514, + 0.004385598469525576, + 0.002676903735846281, + 0.0041999719105660915, + 0.002670086920261383, + 0.005232699681073427, + 0.0026355870068073273, + 0.004664790350943804, + 0.0026610493659973145, + 0.004898969549685717, + 0.0026388936676084995, + 0.005131533369421959, + 0.0026469926815479994, + 0.005069608334451914, + 0.00264862016774714, + 0.0051318020559847355, + 0.002641494618728757, + 0.0058372304774820805, + 0.002674198243767023, + 0.0059690922498703, + 0.002557571977376938, + 0.00554921617731452, + 0.00252733426168561, + 0.0058550527319312096, + 0.0025468147359788418, + 0.006255472078919411, + 0.0026856842450797558, + 0.006502739619463682, + 0.002551089273765683, + 0.006122888531535864, + 0.0025778315030038357, + 0.007071305997669697, + 0.0026001501828432083, + 0.006241494789719582, + 0.0025662274565547705 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0056993408498569815, + 0.005586029321860134, + 0.005830344684528519, + 0.00540288712216694, + 0.00538025811909819, + 0.0052226633069546126, + 0.005297605878918106, + 0.0052023753769565, + 0.0048107019260181325, + 0.005248672753719287, + 0.004962353375385338, + 0.005501319898161136, + 0.005588445553926757, + 0.005229606380765812, + 0.004884302670286647, + 0.005031817046498285, + 0.005000470961685071, + 0.0053937604587735305, + 0.004980398173225761, + 0.004658922297714218, + 0.00496598185096497, + 0.004578385934052906, + 0.004596161734371138, + 0.004990442293511754, + 0.005201771295872185, + 0.00459813521112035, + 0.004706502430387731, + 0.004781878207244352, + 0.0048378722082161795, + 0.004473110720597102, + 0.005084432926394394, + 0.004685705425265166 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.005452688082542498, + "first_to_last_ratio": 1.1368823875795573, + "imbalance_abs_log_ratio": 0.12828976841498158, + "last_quartile_mean": 0.004796176053137182, + "mean": 0.005075395512328069, + "normalized": [ + 1.122935313319594, + 1.1006096585560166, + 1.1487468652180286, + 1.064525337787646, + 1.060066768398564, + 1.0290160233362764, + 1.0437818818356701, + 1.025018713185997, + 0.947847692723257, + 1.0341406380981206, + 0.9777274230809889, + 1.0839194472230789, + 1.1010857262951226, + 1.0303840100861434, + 0.9623491722808084, + 0.9914137793352396, + 0.9852376922229987, + 1.0627271206100406, + 0.9812827711906272, + 0.9179427074003902, + 0.9784423379227619, + 0.9020747098294245, + 0.9055770576316904, + 0.9832617539638114, + 1.024899691706223, + 0.905965889742295, + 0.9273173724009683, + 0.9421685848185097, + 0.9532010257062827, + 0.8813324419214177, + 1.00178063247374, + 0.9232197596982638 + ], + "population_cv": 0.06921897980134169 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.02802751213312149, + 0.014061377383768559, + 0.01413465291261673, + 0.014165290631353855, + 0.014225807972252369, + 0.014281988143920898, + 0.01433455292135477, + 0.014350161887705326, + 0.01440478302538395, + 0.00966146495193243, + 0.009703479707241058, + 0.009741888381540775, + 0.00977881159633398, + 0.009820535778999329, + 0.009858808480203152, + 0.00991273857653141, + 0.009950309991836548, + 0.0074927168898284435, + 0.00752287870272994, + 0.007546746172010899, + 0.007571155205368996, + 0.00766373984515667, + 0.007693085819482803, + 0.007762390188872814, + 0.007788818795233965, + 0.006325984839349985, + 0.006353048607707024, + 0.006375366821885109, + 0.006389216054230928, + 0.006445344537496567, + 0.0064707002602517605, + 0.006555322092026472, + 0.00656962301582098, + 0.005531130358576775, + 0.005553483963012695, + 0.005611324217170477, + 0.0056249531917274, + 0.0056777093559503555, + 0.005688921548426151, + 0.00574630219489336, + 0.005768724251538515, + 0.004946626722812653, + 0.004966245498508215, + 0.0050392793491482735, + 0.005057469941675663, + 0.005084683652967215, + 0.005098348017781973, + 0.005203457549214363, + 0.005216081161051989, + 0.004667261149734259, + 0.0046758754178881645, + 0.004717131145298481, + 0.004727107938379049, + 0.004768412560224533, + 0.004777040798217058, + 0.004826710093766451, + 0.004837613552808762, + 0.0044052014127373695, + 0.0044186655431985855, + 0.004504534415900707, + 0.004510571248829365, + 0.004574235528707504, + 0.0045849340967834, + 0.004644096828997135 + ], + "loss_nats": 5.516995429992676, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0022265110164880753, + 0.0034885562490671873, + 0.0040087271481752396, + 0.004831995815038681, + 0.005379301495850086, + 0.005952076520770788, + 0.006410556845366955, + 0.006925336085259914, + 0.0027253651060163975, + 0.00385080324485898, + 0.004838948138058186, + 0.005542940925806761, + 0.006175771821290255, + 0.006729288958013058, + 0.00742313964292407, + 0.007927131839096546, + 0.0030937574338167906, + 0.004081308841705322, + 0.005150026176124811, + 0.005811207927763462, + 0.007019025273621082, + 0.007534808944910765, + 0.008143599145114422, + 0.008581922389566898, + 0.004255065228790045, + 0.00499920267611742, + 0.0066224560141563416, + 0.007102519739419222, + 0.008129785768687725, + 0.008542952127754688, + 0.010099578648805618, + 0.010447698645293713, + 0.0041999719105660915, + 0.005041283555328846, + 0.00719039561226964, + 0.007636935915797949, + 0.009030314162373543, + 0.009387043304741383, + 0.010736005380749702, + 0.011020516976714134, + 0.005131533369421959, + 0.005807018838822842, + 0.00787449348717928, + 0.008367395959794521, + 0.009912566281855106, + 0.01024632342159748, + 0.012113703414797783, + 0.012378672137856483, + 0.0059690922498703, + 0.006453148555010557, + 0.00876442901790142, + 0.009127611294388771, + 0.010806876234710217, + 0.011037957854568958, + 0.012353328987956047, + 0.012611051090061665, + 0.006502739619463682, + 0.007010402157902718, + 0.009470963850617409, + 0.009794782847166061, + 0.012123330496251583, + 0.012445803731679916, + 0.01384514570236206, + 0.014087481424212456 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.974741700105369e-05, + 6.653120362898335e-05, + 6.306816067080945e-05, + 6.021809531375766e-05, + 5.5740663810865954e-05, + 5.2713170589413494e-05, + 4.972824172000401e-05, + 4.646891829906963e-05, + 4.2185489292023703e-05, + 3.9496502722613513e-05, + 3.673950050142594e-05, + 3.431064760661684e-05, + 3.162782741128467e-05, + 2.9497121431631967e-05, + 2.7693702577380463e-05, + 2.6083735065185465e-05, + 2.4782084437902085e-05, + 2.359730024181772e-05, + 2.26194788410794e-05, + 2.171658525185194e-05, + 2.1000301785534248e-05, + 2.0307239537942223e-05, + 1.9717394025065005e-05, + 1.9229166355216876e-05, + 1.8958127839141525e-05, + 1.8537548385211267e-05, + 1.8187918612966314e-05, + 1.787612200132571e-05, + 1.748248905641958e-05, + 1.7280985048273578e-05, + 1.7088412278098986e-05, + 1.6941357898758724e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.8026983879244654e-05, + "first_to_last_ratio": 3.261020124772794, + "imbalance_abs_log_ratio": 1.1820400681382757, + "last_quartile_mean": 1.779412014002446e-05, + "mean": 3.2724153413710155e-05, + "normalized": [ + 2.1313742213368374, + 2.033091667425974, + 1.9272663794683937, + 1.8401727480145782, + 1.7033493000162006, + 1.6108337448182453, + 1.5196188910167434, + 1.4200189600504973, + 1.2891239311434597, + 1.2069526206922745, + 1.1227028561121923, + 1.0484808322724097, + 0.9664979567671209, + 0.9013868459397276, + 0.8462771283115263, + 0.7970789873591471, + 0.7573025381160617, + 0.7210973479891878, + 0.6912166238532151, + 0.6636255788592389, + 0.6417370533636458, + 0.620558132741007, + 0.6025333574192382, + 0.5876138677176779, + 0.5793313458553462, + 0.5664790820056708, + 0.5557949317444001, + 0.5462669049166695, + 0.5342380851049028, + 0.5280804312888202, + 0.5221957024238739, + 0.5177019458557162 + ], + "population_cv": 0.5019001147827995 + }, + "activation_output_rms_by_block": [ + 0.005867328029125929, + 0.010147687047719955, + 0.014831257984042168, + 0.019580213353037834, + 0.006727410480380058, + 0.011935599148273468, + 0.0171393770724535, + 0.02301580272614956, + 0.007510268595069647, + 0.01379336416721344, + 0.020429225638508797, + 0.026878684759140015, + 0.00918995589017868, + 0.016780642792582512, + 0.024294495582580566, + 0.031401704996824265, + 0.009854132309556007, + 0.01866966113448143, + 0.027872132137417793, + 0.037801310420036316, + 0.012546072714030743, + 0.02335970476269722, + 0.03464336320757866, + 0.046944838017225266, + 0.017289092764258385, + 0.02922297827899456, + 0.04434135556221008, + 0.05992048978805542, + 0.01907295174896717, + 0.03784540295600891, + 0.05689995363354683, + 0.07636377215385437 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013655584480147809, + "first_to_last_ratio": 0.3204069640627036, + "imbalance_abs_log_ratio": 1.1381633284976056, + "last_quartile_mean": 0.042619499610736966, + "mean": 0.025380319682881236, + "normalized": [ + 0.23117628550137534, + 0.39982502878261483, + 0.5843605663503797, + 0.7714722902503267, + 0.26506405610476325, + 0.4702698507112937, + 0.6753018593384319, + 0.9068365967696412, + 0.29590914097647264, + 0.5434669200213786, + 0.8049238896028602, + 1.059036493431933, + 0.3620898398839795, + 0.6611675109790237, + 0.9572178714110899, + 1.2372462360276884, + 0.3882587939269543, + 0.735595980182784, + 1.0981789231054193, + 1.4893945739199224, + 0.494322879726883, + 0.9203865457397331, + 1.364969537044297, + 1.8496551108806196, + 0.6812007484649494, + 1.15140308097479, + 1.747076321978634, + 2.3609036661768754, + 0.7514858751693219, + 1.49113184659905, + 2.2418927083855946, + 3.0087789715809192 + ], + "population_cv": 0.6534126396160335 + }, + "bits_per_byte": 5.151187898320861, + "branch_output_rms_by_sublayer": [ + 0.0029144790023565292, + 0.005083088763058186, + 0.0026895629707723856, + 0.005225023254752159, + 0.0028422826435416937, + 0.005695471540093422, + 0.0029082950204610825, + 0.005877476651221514, + 0.003075477434322238, + 0.00591037655249238, + 0.003283967263996601, + 0.005918757990002632, + 0.003857307368889451, + 0.005890544503927231, + 0.003398116445168853, + 0.00667755538597703, + 0.0037501712795346975, + 0.006289918441325426, + 0.004310416057705879, + 0.006426617503166199, + 0.004353705327957869, + 0.006869504228234291, + 0.004851045086979866, + 0.006513030733913183, + 0.005521548446267843, + 0.0067321620881557465, + 0.005483947694301605, + 0.007041785400360823, + 0.005392563529312611, + 0.0069812205620110035, + 0.0057341353967785835, + 0.006852445658296347, + 0.00562616903334856, + 0.0070425765588879585, + 0.0071852318942546844, + 0.007167586125433445, + 0.006887933239340782, + 0.0075241681188344955, + 0.007283597253262997, + 0.007719857152551413, + 0.007806537207216024, + 0.007892590947449207, + 0.008220287039875984, + 0.008214903064072132, + 0.008879480883479118, + 0.008157636970281601, + 0.00926071498543024, + 0.008292007260024548, + 0.012701748870313168, + 0.007939447648823261, + 0.00895837415009737, + 0.008016198873519897, + 0.012192811816930771, + 0.008416985161602497, + 0.012262859381735325, + 0.009026641957461834, + 0.01350643765181303, + 0.008439409546554089, + 0.014926462434232235, + 0.008805729448795319, + 0.015486822463572025, + 0.008856129832565784, + 0.014737230725586414, + 0.009201034903526306 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0003971361420877675, + 0.0003347177852838067, + 0.00031677715144169444, + 0.00031050553708158156, + 0.0002847314677585494, + 0.0002830997136967945, + 0.0002949253254559385, + 0.0002627758903653516, + 0.00024084111026885302, + 0.000260480450319841, + 0.0002378020967135893, + 0.00024377261778408456, + 0.0002501041851799639, + 0.00023593078790904362, + 0.00021083509767175847, + 0.0002109075302943071, + 0.00019896719470539743, + 0.00020563042073403574, + 0.0001981601474762962, + 0.00018790015718287022, + 0.0001884936723186004, + 0.00017817902827678802, + 0.0001761006050429699, + 0.00017200844302156607, + 0.00018851525845692076, + 0.00015141414169174544, + 0.00017428421344058988, + 0.0001597871095092927, + 0.00016189272017998549, + 0.00016975787890948776, + 0.0001704808367009586, + 0.0001627107201923559 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0003105836266464355, + "first_to_last_ratio": 1.8558331616002395, + "imbalance_abs_log_ratio": 0.6183337389443554, + "last_quartile_mean": 0.00016735535988516708, + "mean": 0.00022561329491102455, + "normalized": [ + 1.7602515057651487, + 1.4835906962655274, + 1.4040712962710036, + 1.3762732254056043, + 1.2620331965402984, + 1.2548006703613852, + 1.3072160732914637, + 1.1647181096707218, + 1.067495203967629, + 1.154543886354601, + 1.054025193007228, + 1.0804887091437654, + 1.1085525136211403, + 1.0457308732718433, + 0.9344976667040203, + 0.9348187143735612, + 0.8818948137957225, + 0.9114286496951809, + 0.878317687592147, + 0.8328416871752734, + 0.8354723616484433, + 0.7897541159844181, + 0.780541789934937, + 0.7624038427762039, + 0.8355680392472695, + 0.6711224254379994, + 0.772490883169463, + 0.7082344574255174, + 0.7175672880617756, + 0.752428525882907, + 0.7556329371821434, + 0.721192960975657 + ], + "population_cv": 0.26357338749356235 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.693089485168457, + "mean_weights": [ + 0.5025052428245544, + 0.49749475717544556 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929048299789429, + "mean_weights": [ + 0.5103185176849365, + 0.4896814823150635 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928635835647583, + "mean_weights": [ + 0.5053519606590271, + 0.4946480393409729 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927831172943115, + "mean_weights": [ + 0.5126674771308899, + 0.4873324930667877 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928117275238037, + "mean_weights": [ + 0.5030385255813599, + 0.49696147441864014 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927525401115417, + "mean_weights": [ + 0.5128871202468872, + 0.487112820148468 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6925652623176575, + "mean_weights": [ + 0.5113205909729004, + 0.4886793792247772 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929247975349426, + "mean_weights": [ + 0.5095095038414001, + 0.49049049615859985 + ], + "sources": 2 + }, + { + "entropy_mean": 1.098347783088684, + "mean_weights": [ + 0.3398834466934204, + 0.32835865020751953, + 0.33175787329673767 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098447561264038, + "mean_weights": [ + 0.3407951593399048, + 0.3304348587989807, + 0.3287699520587921 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0980420112609863, + "mean_weights": [ + 0.3472563922405243, + 0.33136701583862305, + 0.3213765621185303 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985233783721924, + "mean_weights": [ + 0.33736783266067505, + 0.3342247009277344, + 0.32840749621391296 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982682704925537, + "mean_weights": [ + 0.3445430099964142, + 0.3306812047958374, + 0.3247758150100708 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985300540924072, + "mean_weights": [ + 0.33717384934425354, + 0.33328232169151306, + 0.3295438289642334 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098315715789795, + "mean_weights": [ + 0.34228670597076416, + 0.33276987075805664, + 0.3249434232711792 + ], + "sources": 3 + }, + { + "entropy_mean": 1.09848952293396, + "mean_weights": [ + 0.33792412281036377, + 0.33447134494781494, + 0.3276045322418213 + ], + "sources": 3 + }, + { + "entropy_mean": 1.386106252670288, + "mean_weights": [ + 0.2561790645122528, + 0.2513994574546814, + 0.24582090973854065, + 0.24660053849220276 + ], + "sources": 4 + }, + { + "entropy_mean": 1.386183261871338, + "mean_weights": [ + 0.2536899447441101, + 0.25216740369796753, + 0.2471930980682373, + 0.24694956839084625 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3860970735549927, + "mean_weights": [ + 0.257076233625412, + 0.24995796382427216, + 0.2462586909532547, + 0.24670711159706116 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862221240997314, + "mean_weights": [ + 0.25144457817077637, + 0.25284701585769653, + 0.24949511885643005, + 0.24621328711509705 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3860111236572266, + "mean_weights": [ + 0.2548638880252838, + 0.2537461817264557, + 0.2485770881175995, + 0.242812842130661 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862276077270508, + "mean_weights": [ + 0.2517159581184387, + 0.2530348300933838, + 0.24868342280387878, + 0.2465657889842987 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859820365905762, + "mean_weights": [ + 0.2546064257621765, + 0.25476449728012085, + 0.24875503778457642, + 0.2418740689754486 + ], + "sources": 4 + }, + { + "entropy_mean": 1.386254072189331, + "mean_weights": [ + 0.25021401047706604, + 0.25216448307037354, + 0.24994511902332306, + 0.24767637252807617 + ], + "sources": 4 + }, + { + "entropy_mean": 1.609065055847168, + "mean_weights": [ + 0.20364554226398468, + 0.20639754831790924, + 0.19984170794487, + 0.19552560150623322, + 0.19458958506584167 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093063354492188, + "mean_weights": [ + 0.20175650715827942, + 0.20363816618919373, + 0.20010897517204285, + 0.19684356451034546, + 0.19765278697013855 + ], + "sources": 5 + }, + { + "entropy_mean": 1.608630895614624, + "mean_weights": [ + 0.2043527066707611, + 0.2084214687347412, + 0.20422585308551788, + 0.19542182981967926, + 0.18757814168930054 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609345555305481, + "mean_weights": [ + 0.2002810686826706, + 0.20385204255580902, + 0.2001519799232483, + 0.1976388394832611, + 0.19807609915733337 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6086039543151855, + "mean_weights": [ + 0.20521339774131775, + 0.21034619212150574, + 0.20178894698619843, + 0.19381338357925415, + 0.18883806467056274 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093709468841553, + "mean_weights": [ + 0.20029816031455994, + 0.2033783495426178, + 0.2000802755355835, + 0.1980302631855011, + 0.19821295142173767 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6083776950836182, + "mean_weights": [ + 0.20369765162467957, + 0.21106396615505219, + 0.20391109585762024, + 0.19408971071243286, + 0.18723760545253754 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609338402748108, + "mean_weights": [ + 0.200865238904953, + 0.203323632478714, + 0.20118102431297302, + 0.199003204703331, + 0.1956268846988678 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7908425331115723, + "mean_weights": [ + 0.1716410219669342, + 0.1773453801870346, + 0.1690019965171814, + 0.16098390519618988, + 0.1585077941417694, + 0.1625199019908905 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7916772365570068, + "mean_weights": [ + 0.16707879304885864, + 0.16981184482574463, + 0.16858185827732086, + 0.16526785492897034, + 0.16461201012134552, + 0.16464760899543762 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7898263931274414, + "mean_weights": [ + 0.1728883534669876, + 0.18102073669433594, + 0.17351166903972626, + 0.1623196005821228, + 0.15690836310386658, + 0.15335126221179962 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917211055755615, + "mean_weights": [ + 0.1661764681339264, + 0.16885922849178314, + 0.1676425337791443, + 0.16660507023334503, + 0.1660776436328888, + 0.16463905572891235 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7891093492507935, + "mean_weights": [ + 0.17247596383094788, + 0.18407422304153442, + 0.17474780976772308, + 0.1621180772781372, + 0.15558375418186188, + 0.15100015699863434 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917201519012451, + "mean_weights": [ + 0.16665595769882202, + 0.1689598709344864, + 0.16597317159175873, + 0.16514113545417786, + 0.16659271717071533, + 0.16667716205120087 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7891130447387695, + "mean_weights": [ + 0.1723570078611374, + 0.18360206484794617, + 0.1746024489402771, + 0.1640183925628662, + 0.15544453263282776, + 0.14997556805610657 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917444705963135, + "mean_weights": [ + 0.1658288836479187, + 0.16771969199180603, + 0.1663144826889038, + 0.16698287427425385, + 0.1669963002204895, + 0.16615775227546692 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9437212944030762, + "mean_weights": [ + 0.14780430495738983, + 0.15844447910785675, + 0.1498783379793167, + 0.14075885713100433, + 0.1353672742843628, + 0.1310281902551651, + 0.13671858608722687 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458949565887451, + "mean_weights": [ + 0.14285430312156677, + 0.14305134117603302, + 0.1416061967611313, + 0.14283838868141174, + 0.14389833807945251, + 0.14331653714179993, + 0.14243489503860474 + ], + "sources": 7 + }, + { + "entropy_mean": 1.942615270614624, + "mean_weights": [ + 0.1483418047428131, + 0.16144202649593353, + 0.15263789892196655, + 0.1423717439174652, + 0.13469448685646057, + 0.1308881789445877, + 0.1296238899230957 + ], + "sources": 7 + }, + { + "entropy_mean": 1.945899486541748, + "mean_weights": [ + 0.14266017079353333, + 0.14297112822532654, + 0.14172877371311188, + 0.14274320006370544, + 0.14322899281978607, + 0.1432417929172516, + 0.14342594146728516 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9411609172821045, + "mean_weights": [ + 0.1491367369890213, + 0.16433076560497284, + 0.15439118444919586, + 0.14253677427768707, + 0.13549132645130157, + 0.12850789725780487, + 0.1256052851676941 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458861351013184, + "mean_weights": [ + 0.14266805350780487, + 0.14308907091617584, + 0.14319077134132385, + 0.1422133892774582, + 0.14128680527210236, + 0.14307695627212524, + 0.14447495341300964 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9416497945785522, + "mean_weights": [ + 0.14896368980407715, + 0.1632051169872284, + 0.1538591831922531, + 0.14291420578956604, + 0.1364806443452835, + 0.12957154214382172, + 0.12500563263893127 + ], + "sources": 7 + }, + { + "entropy_mean": 1.945875644683838, + "mean_weights": [ + 0.14194919168949127, + 0.14195851981639862, + 0.1417105346918106, + 0.14289532601833344, + 0.14394041895866394, + 0.1432405710220337, + 0.14430546760559082 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0794143676757812, + "mean_weights": [ + 0.12475782632827759, + 0.12462858855724335, + 0.1242634654045105, + 0.12421638518571854, + 0.12483115494251251, + 0.12466961145401001, + 0.12629851698875427, + 0.12633445858955383 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0793282985687256, + "mean_weights": [ + 0.12421073764562607, + 0.12278284132480621, + 0.1236410066485405, + 0.12383973598480225, + 0.124093197286129, + 0.1260346919298172, + 0.12756547331809998, + 0.1278323084115982 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0794224739074707, + "mean_weights": [ + 0.12477025389671326, + 0.12492755055427551, + 0.12423760443925858, + 0.12470321357250214, + 0.12560197710990906, + 0.12460009753704071, + 0.12593437731266022, + 0.12522491812705994 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079335927963257, + "mean_weights": [ + 0.12416492402553558, + 0.1224912628531456, + 0.12276282906532288, + 0.12438546866178513, + 0.12566527724266052, + 0.12613417208194733, + 0.12749025225639343, + 0.12690579891204834 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1906607151031494, + "mean_weights": [ + 0.11858130246400833, + 0.13274145126342773, + 0.12361472845077515, + 0.1174783855676651, + 0.11157400906085968, + 0.10437168180942535, + 0.09862296283245087, + 0.09466812014579773, + 0.09834735095500946 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197183609008789, + "mean_weights": [ + 0.11082965135574341, + 0.11017745733261108, + 0.1101604551076889, + 0.11044102907180786, + 0.11130303889513016, + 0.11165212094783783, + 0.11251389235258102, + 0.11188136041164398, + 0.11104100942611694 + ], + "sources": 9 + }, + { + "entropy_mean": 2.189944267272949, + "mean_weights": [ + 0.11908549815416336, + 0.1333266645669937, + 0.12376245856285095, + 0.11780785024166107, + 0.11251536011695862, + 0.1054278016090393, + 0.09970364719629288, + 0.09443314373493195, + 0.09393756091594696 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197188377380371, + "mean_weights": [ + 0.11065521836280823, + 0.1101783812046051, + 0.11036231368780136, + 0.11101861298084259, + 0.11014353483915329, + 0.11143915355205536, + 0.11245327442884445, + 0.11143463850021362, + 0.1123148649930954 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1897130012512207, + "mean_weights": [ + 0.11971777677536011, + 0.13409766554832458, + 0.12355241179466248, + 0.11792266368865967, + 0.111484095454216, + 0.10532692074775696, + 0.1002035140991211, + 0.09460292756557465, + 0.09309203177690506 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197194814682007, + "mean_weights": [ + 0.11115394532680511, + 0.11169658601284027, + 0.1113947182893753, + 0.1111471951007843, + 0.11216932535171509, + 0.11137214303016663, + 0.11164958775043488, + 0.1097034439444542, + 0.10971304029226303 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1886563301086426, + "mean_weights": [ + 0.11989110708236694, + 0.13586243987083435, + 0.12442915141582489, + 0.11758226156234741, + 0.11255474388599396, + 0.10467666387557983, + 0.09948796033859253, + 0.0941111147403717, + 0.09140457957983017 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.02857121266424656, + 0.014366661198437214, + 0.01502841804176569, + 0.01491964515298605, + 0.015756145119667053, + 0.015600254759192467, + 0.01672360487282276, + 0.01673336699604988, + 0.0178521815687418, + 0.012046977877616882, + 0.012998392805457115, + 0.013210508972406387, + 0.014046002179384232, + 0.01421300694346428, + 0.015207725577056408, + 0.01538239885121584, + 0.016687525436282158, + 0.012679810635745525, + 0.013633915223181248, + 0.013726244680583477, + 0.014753643423318863, + 0.0149829788133502, + 0.01603475585579872, + 0.016217511147260666, + 0.017353398725390434, + 0.014127095229923725, + 0.014999712817370892, + 0.015245530754327774, + 0.016187410801649094, + 0.016406599432229996, + 0.017409054562449455, + 0.0175496656447649, + 0.018515538424253464, + 0.015638958662748337, + 0.016505438834428787, + 0.016804546117782593, + 0.017694544047117233, + 0.017854822799563408, + 0.018941225484013557, + 0.019057651981711388, + 0.020315859466791153, + 0.017555315047502518, + 0.018615294247865677, + 0.01868908852338791, + 0.0198390893638134, + 0.01984322816133499, + 0.021209733560681343, + 0.02109113708138466, + 0.02274204045534134, + 0.020587537437677383, + 0.02132096141576767, + 0.021864473819732666, + 0.02269725315272808, + 0.023383859544992447, + 0.024180114269256592, + 0.024959729984402657, + 0.025995153933763504, + 0.022719956934452057, + 0.02463533543050289, + 0.02416539378464222, + 0.026327747851610184, + 0.025569451972842216, + 0.027951911091804504, + 0.02688952535390854 + ], + "loss_nats": 3.5705313682556152, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.193145513534546, + "mean_weights": [ + 0.1007326990365982, + 0.09430979192256927, + 0.10201791673898697, + 0.10992352664470673, + 0.1144738644361496, + 0.11844484508037567, + 0.11965207010507584, + 0.12089744210243225, + 0.11954784393310547 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.0029144790023565292, + 0.005867328029125929, + 0.00653798459097743, + 0.010147687047719955, + 0.010621722787618637, + 0.014831257984042168, + 0.015127260237932205, + 0.019580213353037834, + 0.003075477434322238, + 0.006727410480380058, + 0.007619299925863743, + 0.011935599148273468, + 0.01270378939807415, + 0.0171393770724535, + 0.01777406595647335, + 0.02301580272614956, + 0.0037501712795346975, + 0.007510268595069647, + 0.009127544239163399, + 0.01379336416721344, + 0.015184290707111359, + 0.020429225638508797, + 0.021826842799782753, + 0.026878684759140015, + 0.005521548446267843, + 0.00918995589017868, + 0.011899306438863277, + 0.016780642792582512, + 0.019033949822187424, + 0.024294495582580566, + 0.026245392858982086, + 0.031401704996824265, + 0.00562616903334856, + 0.009854132309556007, + 0.01374119520187378, + 0.01866966113448143, + 0.02210313454270363, + 0.027872132137417793, + 0.03175624832510948, + 0.037801310420036316, + 0.007806537207216024, + 0.012546072714030743, + 0.017673099413514137, + 0.02335970476269722, + 0.028496280312538147, + 0.03464336320757866, + 0.040420446544885635, + 0.046944838017225266, + 0.012701748870313168, + 0.017289092764258385, + 0.023572605103254318, + 0.02922297827899456, + 0.03792540729045868, + 0.04434135556221008, + 0.052800603210926056, + 0.05992048978805542, + 0.01350643765181303, + 0.01907295174896717, + 0.03146032243967056, + 0.03784540295600891, + 0.05003572627902031, + 0.05689995363354683, + 0.06909419596195221, + 0.07636377215385437 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00013968476559966803, + 0.00014419027138501406, + 0.00014389782154466957, + 0.0001367047370877117, + 0.0001354013365926221, + 0.00012278779468033463, + 0.00011162846931256354, + 0.0001027525431709364, + 9.788239549379796e-05, + 8.922146662371233e-05, + 8.216388232540339e-05, + 7.673823711229488e-05, + 7.37546433811076e-05, + 6.787071470171213e-05, + 6.234207830857486e-05, + 5.7451670727459714e-05, + 5.7620680308900774e-05, + 5.1877690566470847e-05, + 4.701966099673882e-05, + 4.3048319639638066e-05, + 4.742821693071164e-05, + 4.329146395321004e-05, + 3.963017661590129e-05, + 3.652414306998253e-05, + 2.8894202841911465e-05, + 2.6984631404047832e-05, + 2.5778352210181765e-05, + 2.502644019841682e-05, + 2.6952677217195742e-05, + 2.67578216153197e-05, + 2.666404179763049e-05, + 2.6654974135453813e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00012963096742169, + "first_to_last_ratio": 4.852522088637946, + "imbalance_abs_log_ratio": 1.5794985880626093, + "last_quartile_mean": 2.6714142677519703e-05, + "mean": 6.951957254841545e-05, + "normalized": [ + 2.0092868882700263, + 2.074096058122276, + 2.069889331446262, + 1.9664208520917836, + 1.9476721681268203, + 1.7662334531015942, + 1.6057128261946982, + 1.4780376145059946, + 1.407983275869969, + 1.2834006791623453, + 1.1818812934757628, + 1.1038364348234757, + 1.0609191149692803, + 0.9762821060852321, + 0.8967557771612882, + 0.8264100111871185, + 0.8288411190786891, + 0.7462314376335055, + 0.6763514111654377, + 0.6192258965582386, + 0.6822282587782204, + 0.6227233909279377, + 0.5700578292293395, + 0.5253792814181366, + 0.41562687719043007, + 0.38815876471701477, + 0.37080711611437156, + 0.3599912841953635, + 0.3876991216887174, + 0.38489623331163, + 0.38354726331294503, + 0.38341683008609573 + ], + "population_cv": 0.578222701095633 + }, + "activation_output_rms_by_block": [ + 0.02206760086119175, + 0.03904516622424126, + 0.05045926570892334, + 0.058496471494436264, + 0.018059836700558662, + 0.026500778272747993, + 0.03583977371454239, + 0.039596036076545715, + 0.019077016040682793, + 0.02743501216173172, + 0.033393505960702896, + 0.04118926450610161, + 0.018032971769571304, + 0.030527887865900993, + 0.038561977446079254, + 0.045049604028463364, + 0.023235268890857697, + 0.03527042642235756, + 0.04559057578444481, + 0.05344199389219284, + 0.025266120210289955, + 0.040697868913412094, + 0.05694866180419922, + 0.0770706906914711, + 0.02325853891670704, + 0.04341072589159012, + 0.06275466829538345, + 0.08457393944263458, + 0.030810445547103882, + 0.058590520173311234, + 0.09210194647312164, + 0.12781774997711182 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.03625811613164842, + "first_to_last_ratio": 0.5542798693535069, + "imbalance_abs_log_ratio": 0.5900855404565164, + "last_quartile_mean": 0.06541481683962047, + "mean": 0.04450538469245657, + "normalized": [ + 0.49584114402525525, + 0.8773133070088753, + 1.1337788911973143, + 1.3143684050516948, + 0.40578992464297714, + 0.5954510550998503, + 0.8052907296994344, + 0.889690907070331, + 0.42864512176469843, + 0.6164425350171574, + 0.7503250717067258, + 0.9254894613478752, + 0.40518629137089107, + 0.6859369507050976, + 0.8664564459458611, + 1.0122281683388983, + 0.5220777002922065, + 0.7924979565076248, + 1.0243833661811301, + 1.2007983811732106, + 0.5677092869747159, + 0.9144481997997462, + 1.2795903731139238, + 1.731716088380069, + 0.5226005589532459, + 0.9754039020574516, + 1.4100466433226908, + 1.900308019514084, + 0.6922857932812361, + 1.3164816028034922, + 2.0694562491610693, + 2.8719614684911656 + ], + "population_cv": 0.5342880822877267 + }, + "bits_per_byte": 3.0698287481121254, + "branch_output_rms_by_sublayer": [ + 0.011440085247159004, + 0.017743771895766258, + 0.014298862777650356, + 0.01526967715471983, + 0.014941251836717129, + 0.011439034715294838, + 0.012462953105568886, + 0.010528148151934147, + 0.015554625540971756, + 0.008605039678514004, + 0.018375173211097717, + 0.007548195309937, + 0.023418370634317398, + 0.008042574860155582, + 0.021795328706502914, + 0.008146703243255615, + 0.01653105393052101, + 0.008701125159859657, + 0.012565970420837402, + 0.00839450117200613, + 0.013558699749410152, + 0.008161830715835094, + 0.012874778360128403, + 0.00850198045372963, + 0.014525899663567543, + 0.009213869459927082, + 0.018652023747563362, + 0.01001427136361599, + 0.02060537412762642, + 0.009777705185115337, + 0.018269913271069527, + 0.01001414842903614, + 0.019669709727168083, + 0.010826915502548218, + 0.02017567679286003, + 0.011225171387195587, + 0.021515851840376854, + 0.012702127918601036, + 0.02303197793662548, + 0.014425408095121384, + 0.018934987485408783, + 0.015180723741650581, + 0.02722637727856636, + 0.01707947812974453, + 0.029236918315291405, + 0.019315408542752266, + 0.013673579320311546, + 0.025045383721590042, + 0.013713943772017956, + 0.0175370704382658, + 0.012548972852528095, + 0.01887757144868374, + 0.009189880453050137, + 0.02027132548391819, + 0.007580145727843046, + 0.022869953885674477, + 0.008130092173814774, + 0.027308499440550804, + 0.007614650763571262, + 0.026508644223213196, + 0.008368851616978645, + 0.03155560791492462, + 0.010984547436237335, + 0.031208017840981483 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0004480040288413438, + 0.00034905393563009515, + 0.0003186876554230452, + 0.00029722272139999473, + 0.0003836655617173932, + 0.00035027461090326024, + 0.0003203419420245184, + 0.0002421217796566622, + 0.00021224789594945212, + 0.00018749022201730562, + 0.00017000555468925976, + 0.00015348661157856545, + 0.00016588397861071855, + 0.0001504108235380311, + 0.00014559812045772326, + 0.00013398186820032227, + 0.00014478352065544023, + 0.0001292278785505549, + 0.00012222577049633693, + 0.00012344687657585363, + 0.00013778052733590365, + 0.00013789910209864215, + 0.00014048182002837525, + 0.00011674426345046287, + 9.508220691845448e-05, + 8.49146518544258e-05, + 7.621664984318456e-05, + 7.673780865014462e-05, + 8.676035524948501e-05, + 8.286897682235364e-05, + 9.231935132123829e-05, + 9.233285505256992e-05 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00033867152944953914, + "first_to_last_ratio": 3.9424369965400805, + "imbalance_abs_log_ratio": 1.3717990591414633, + "last_quartile_mean": 8.590410696398203e-05, + "mean": 0.0001802593726731599, + "normalized": [ + 2.4853300119580983, + 1.9363982602057965, + 1.7679394457944704, + 1.6488613989515473, + 2.1284083930162256, + 1.9431700316541438, + 1.7771167028598918, + 1.3431855224286677, + 1.1774583080031722, + 1.0401135832046662, + 0.9431163116134215, + 0.8514764547464729, + 0.9202516138314409, + 0.8344133306774059, + 0.8077145631795621, + 0.7432726865373795, + 0.8031955204790195, + 0.7168996354207138, + 0.6780550086455281, + 0.6848291700187111, + 0.7643459826398185, + 0.7650037834574963, + 0.7793315706423325, + 0.6476460098534768, + 0.5274744137208015, + 0.4710692742085047, + 0.422816570993942, + 0.4257077316544476, + 0.481308427755354, + 0.4597207656580988, + 0.5121473017030219, + 0.5122222144863706 + ], + "population_cv": 0.5585066947395415 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6488328576087952, + "mean_weights": [ + 0.3713088631629944, + 0.6286911964416504 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6677432656288147, + "mean_weights": [ + 0.42424651980400085, + 0.5757534503936768 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6897715330123901, + "mean_weights": [ + 0.5088717937469482, + 0.49112820625305176 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6753112077713013, + "mean_weights": [ + 0.45665261149406433, + 0.5433473587036133 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6805926561355591, + "mean_weights": [ + 0.45422112941741943, + 0.5457788705825806 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6749809980392456, + "mean_weights": [ + 0.44650328159332275, + 0.5534967184066772 + ], + "sources": 2 + }, + { + "entropy_mean": 0.68096923828125, + "mean_weights": [ + 0.44845131039619446, + 0.5515486598014832 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6780940294265747, + "mean_weights": [ + 0.4500512480735779, + 0.5499487519264221 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0932620763778687, + "mean_weights": [ + 0.3082190752029419, + 0.34458988904953003, + 0.3471910357475281 + ], + "sources": 3 + }, + { + "entropy_mean": 1.083313226699829, + "mean_weights": [ + 0.29535767436027527, + 0.32296058535575867, + 0.38168174028396606 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0925581455230713, + "mean_weights": [ + 0.3045356869697571, + 0.3392125964164734, + 0.35625171661376953 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0686767101287842, + "mean_weights": [ + 0.2977526783943176, + 0.2887532413005829, + 0.4134941101074219 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0869477987289429, + "mean_weights": [ + 0.28761792182922363, + 0.325837641954422, + 0.38654446601867676 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0719780921936035, + "mean_weights": [ + 0.28977927565574646, + 0.30558985471725464, + 0.4046308398246765 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0915818214416504, + "mean_weights": [ + 0.30292487144470215, + 0.3265923261642456, + 0.37048280239105225 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0644346475601196, + "mean_weights": [ + 0.2928414046764374, + 0.2952367961406708, + 0.41192179918289185 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3766294717788696, + "mean_weights": [ + 0.22489973902702332, + 0.2307153046131134, + 0.26034051179885864, + 0.28404444456100464 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3641514778137207, + "mean_weights": [ + 0.23395879566669464, + 0.22087553143501282, + 0.2556862533092499, + 0.28947943449020386 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3795398473739624, + "mean_weights": [ + 0.2378663569688797, + 0.2324407696723938, + 0.25318077206611633, + 0.27651211619377136 + ], + "sources": 4 + }, + { + "entropy_mean": 1.355592966079712, + "mean_weights": [ + 0.24115759134292603, + 0.21447840332984924, + 0.2324349284172058, + 0.3119291067123413 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3810280561447144, + "mean_weights": [ + 0.2541203200817108, + 0.2300688922405243, + 0.24113205075263977, + 0.27467870712280273 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3511195182800293, + "mean_weights": [ + 0.24730737507343292, + 0.21268269419670105, + 0.22346141934394836, + 0.31654852628707886 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3791983127593994, + "mean_weights": [ + 0.2683478891849518, + 0.2200746387243271, + 0.2366974651813507, + 0.27487999200820923 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3404144048690796, + "mean_weights": [ + 0.2448512315750122, + 0.20816002786159515, + 0.21612173318862915, + 0.3308670222759247 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6046935319900513, + "mean_weights": [ + 0.20989567041397095, + 0.1830345094203949, + 0.18817012012004852, + 0.20194348692893982, + 0.216956228017807 + ], + "sources": 5 + }, + { + "entropy_mean": 1.578981637954712, + "mean_weights": [ + 0.1952551007270813, + 0.17473794519901276, + 0.16826944053173065, + 0.21871528029441833, + 0.24302224814891815 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6067560911178589, + "mean_weights": [ + 0.1974964141845703, + 0.19614408910274506, + 0.18701592087745667, + 0.20260316133499146, + 0.21674039959907532 + ], + "sources": 5 + }, + { + "entropy_mean": 1.582057237625122, + "mean_weights": [ + 0.20122107863426208, + 0.18211950361728668, + 0.1691378653049469, + 0.19685104489326477, + 0.25067052245140076 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6024136543273926, + "mean_weights": [ + 0.20406027138233185, + 0.1882825493812561, + 0.17937149107456207, + 0.19724251329898834, + 0.23104315996170044 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5875946283340454, + "mean_weights": [ + 0.19846211373806, + 0.19182303547859192, + 0.1741437017917633, + 0.19055870175361633, + 0.24501246213912964 + ], + "sources": 5 + }, + { + "entropy_mean": 1.598887324333191, + "mean_weights": [ + 0.21854522824287415, + 0.1861659586429596, + 0.17433759570121765, + 0.1919453889131546, + 0.2290058135986328 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5872113704681396, + "mean_weights": [ + 0.20410007238388062, + 0.18717682361602783, + 0.16822315752506256, + 0.19171306490898132, + 0.2487868368625641 + ], + "sources": 5 + }, + { + "entropy_mean": 1.786472201347351, + "mean_weights": [ + 0.16070088744163513, + 0.16823704540729523, + 0.1575208306312561, + 0.15736232697963715, + 0.17022466659545898, + 0.1859542429447174 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7729823589324951, + "mean_weights": [ + 0.15845254063606262, + 0.1613212376832962, + 0.1480470448732376, + 0.14625367522239685, + 0.17868295311927795, + 0.20724254846572876 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7837891578674316, + "mean_weights": [ + 0.16014620661735535, + 0.1647244095802307, + 0.15179918706417084, + 0.1573302447795868, + 0.16792604327201843, + 0.19807390868663788 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7699990272521973, + "mean_weights": [ + 0.15215054154396057, + 0.16884371638298035, + 0.1550704687833786, + 0.14430604875087738, + 0.1683318316936493, + 0.2112973928451538 + ], + "sources": 6 + }, + { + "entropy_mean": 1.777813196182251, + "mean_weights": [ + 0.14987824857234955, + 0.1710509955883026, + 0.16105830669403076, + 0.15297472476959229, + 0.15995559096336365, + 0.20508211851119995 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7798333168029785, + "mean_weights": [ + 0.16338403522968292, + 0.16634798049926758, + 0.15853464603424072, + 0.1483948677778244, + 0.15729545056819916, + 0.2060430347919464 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7787220478057861, + "mean_weights": [ + 0.14968574047088623, + 0.17835155129432678, + 0.16151277720928192, + 0.15410974621772766, + 0.16275250911712646, + 0.19358766078948975 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7812831401824951, + "mean_weights": [ + 0.16528452932834625, + 0.17285624146461487, + 0.15596893429756165, + 0.1506594866514206, + 0.15738031268119812, + 0.19785049557685852 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9317415952682495, + "mean_weights": [ + 0.1265774667263031, + 0.15059247612953186, + 0.13667511940002441, + 0.13014787435531616, + 0.12994559109210968, + 0.14075390994548798, + 0.1853075623512268 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9381402730941772, + "mean_weights": [ + 0.1495429277420044, + 0.1444854587316513, + 0.13528448343276978, + 0.1298307478427887, + 0.12604768574237823, + 0.14492058753967285, + 0.16988810896873474 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9237060546875, + "mean_weights": [ + 0.11751418560743332, + 0.16999870538711548, + 0.1501305103302002, + 0.1269984394311905, + 0.12584897875785828, + 0.13028855621814728, + 0.17922061681747437 + ], + "sources": 7 + }, + { + "entropy_mean": 1.935704231262207, + "mean_weights": [ + 0.14528027176856995, + 0.1476873755455017, + 0.13251766562461853, + 0.1330593079328537, + 0.12958043813705444, + 0.1360592395067215, + 0.17581568658351898 + ], + "sources": 7 + }, + { + "entropy_mean": 1.916542887687683, + "mean_weights": [ + 0.11694975942373276, + 0.1744362711906433, + 0.15474165976047516, + 0.131773442029953, + 0.12231090664863586, + 0.1261064112186432, + 0.1736815720796585 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9355347156524658, + "mean_weights": [ + 0.14214879274368286, + 0.1516517698764801, + 0.13308337330818176, + 0.13386298716068268, + 0.1280868500471115, + 0.13704150915145874, + 0.17412473261356354 + ], + "sources": 7 + }, + { + "entropy_mean": 1.914894938468933, + "mean_weights": [ + 0.11490190029144287, + 0.18283718824386597, + 0.1658046841621399, + 0.13227716088294983, + 0.12100826948881149, + 0.12428820133209229, + 0.15888255834579468 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9270142316818237, + "mean_weights": [ + 0.1406678557395935, + 0.16653791069984436, + 0.14315444231033325, + 0.13022293150424957, + 0.12133066356182098, + 0.13483861088752747, + 0.16324755549430847 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.063281774520874, + "mean_weights": [ + 0.1224159449338913, + 0.14470532536506653, + 0.13147830963134766, + 0.12095212936401367, + 0.11100582033395767, + 0.12170186638832092, + 0.12995365262031555, + 0.11778692901134491 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0642433166503906, + "mean_weights": [ + 0.12361180037260056, + 0.14234347641468048, + 0.13482880592346191, + 0.11730901896953583, + 0.10965236276388168, + 0.11442940682172775, + 0.13640299439430237, + 0.12142214179039001 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0607056617736816, + "mean_weights": [ + 0.1239023357629776, + 0.1408671736717224, + 0.13723976910114288, + 0.11641045659780502, + 0.10561946034431458, + 0.11147914081811905, + 0.13947603106498718, + 0.12500561773777008 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.051264762878418, + "mean_weights": [ + 0.11618876457214355, + 0.14723065495491028, + 0.1393716037273407, + 0.12638577818870544, + 0.11495404690504074, + 0.1242920309305191, + 0.1266207993030548, + 0.10495632886886597 + ], + "sources": 8 + }, + { + "entropy_mean": 2.11655330657959, + "mean_weights": [ + 0.08525651693344116, + 0.17644855380058289, + 0.1599549949169159, + 0.12423337996006012, + 0.10792230069637299, + 0.09925681352615356, + 0.10649006068706512, + 0.07265006005764008, + 0.06778731942176819 + ], + "sources": 9 + }, + { + "entropy_mean": 2.191338539123535, + "mean_weights": [ + 0.11083044111728668, + 0.11674025654792786, + 0.11638931930065155, + 0.11020419001579285, + 0.10246333479881287, + 0.1087266281247139, + 0.11676684021949768, + 0.10744082182645798, + 0.11043815314769745 + ], + "sources": 9 + }, + { + "entropy_mean": 2.10215425491333, + "mean_weights": [ + 0.08375819027423859, + 0.18608424067497253, + 0.16340342164039612, + 0.12258696556091309, + 0.10664238035678864, + 0.09718240797519684, + 0.11128680408000946, + 0.07153131067752838, + 0.05752427503466606 + ], + "sources": 9 + }, + { + "entropy_mean": 2.181058883666992, + "mean_weights": [ + 0.11155849695205688, + 0.1251899152994156, + 0.11970280110836029, + 0.10934481024742126, + 0.10090267658233643, + 0.1079772412776947, + 0.1149747371673584, + 0.10497056692838669, + 0.10537874698638916 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0824403762817383, + "mean_weights": [ + 0.08007125556468964, + 0.18928390741348267, + 0.17611780762672424, + 0.1278923749923706, + 0.11030403524637222, + 0.09920881688594818, + 0.0996551513671875, + 0.06712093949317932, + 0.05034572258591652 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1627016067504883, + "mean_weights": [ + 0.10047890990972519, + 0.1431562453508377, + 0.12687095999717712, + 0.12467103451490402, + 0.11786037683486938, + 0.11405743658542633, + 0.10789870470762253, + 0.08546975255012512, + 0.0795365646481514 + ], + "sources": 9 + }, + { + "entropy_mean": 2.09004807472229, + "mean_weights": [ + 0.08143030107021332, + 0.188899427652359, + 0.16913121938705444, + 0.12752610445022583, + 0.11106959730386734, + 0.10009709000587463, + 0.10164514929056168, + 0.06898628175258636, + 0.05121484398841858 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.043948832899332047, + 0.017426572740077972, + 0.023905858397483826, + 0.027812963351607323, + 0.03108355402946472, + 0.032266341149806976, + 0.03562644124031067, + 0.03631545603275299, + 0.038708511739969254, + 0.025997379794716835, + 0.026929674670100212, + 0.02785138599574566, + 0.028170479461550713, + 0.029001908376812935, + 0.030345099046826363, + 0.03039054200053215, + 0.03182388097047806, + 0.022660963237285614, + 0.024408243596553802, + 0.023715563118457794, + 0.02506357431411743, + 0.024509765207767487, + 0.02601918764412403, + 0.024998920038342476, + 0.02737889066338539, + 0.02053835801780224, + 0.022057535126805305, + 0.02112138271331787, + 0.022711308673024178, + 0.021496085450053215, + 0.023205196484923363, + 0.022037407383322716, + 0.023875990882515907, + 0.018683385103940964, + 0.019818224012851715, + 0.019258731976151466, + 0.020764097571372986, + 0.020242908969521523, + 0.02126452699303627, + 0.021276988089084625, + 0.02234877087175846, + 0.018207166343927383, + 0.01914789527654648, + 0.01944340392947197, + 0.020505165681242943, + 0.020716268569231033, + 0.022401606664061546, + 0.021901661530137062, + 0.024236146360635757, + 0.02131612040102482, + 0.022272150963544846, + 0.02255966328084469, + 0.023864896968007088, + 0.02386752888560295, + 0.025570377707481384, + 0.02549011819064617, + 0.026147278025746346, + 0.022851455956697464, + 0.02670980989933014, + 0.024300290271639824, + 0.028722746297717094, + 0.024736108258366585, + 0.028534099459648132, + 0.02614494413137436 + ], + "loss_nats": 2.127843141555786, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1680848598480225, + "mean_weights": [ + 0.12267276644706726, + 0.09314168244600296, + 0.0893743485212326, + 0.0950310230255127, + 0.09316371381282806, + 0.10472740232944489, + 0.14295271039009094, + 0.12088243663311005, + 0.13805390894412994 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.011440085247159004, + 0.02206760086119175, + 0.02898360975086689, + 0.03904516622424126, + 0.042851101607084274, + 0.05045926570892334, + 0.052241306751966476, + 0.058496471494436264, + 0.015554625540971756, + 0.018059836700558662, + 0.023158090189099312, + 0.026500778272747993, + 0.03209060803055763, + 0.03583977371454239, + 0.03561529517173767, + 0.039596036076545715, + 0.01653105393052101, + 0.019077016040682793, + 0.02376493252813816, + 0.02743501216173172, + 0.029219049960374832, + 0.033393505960702896, + 0.03676135092973709, + 0.04118926450610161, + 0.014525899663567543, + 0.018032971769571304, + 0.025643398985266685, + 0.030527887865900993, + 0.033384568989276886, + 0.038561977446079254, + 0.039308302104473114, + 0.045049604028463364, + 0.019669709727168083, + 0.023235268890857697, + 0.030136700719594955, + 0.03527042642235756, + 0.03876589611172676, + 0.04559057578444481, + 0.044606152921915054, + 0.05344199389219284, + 0.018934987485408783, + 0.025266120210289955, + 0.03092169016599655, + 0.040697868913412094, + 0.04323670268058777, + 0.05694866180419922, + 0.057210735976696014, + 0.0770706906914711, + 0.013713943772017956, + 0.02325853891670704, + 0.02911773882806301, + 0.04341072589159012, + 0.04606543108820915, + 0.06275466829538345, + 0.06535276770591736, + 0.08457393944263458, + 0.008130092173814774, + 0.030810445547103882, + 0.034127119928598404, + 0.058590520173311234, + 0.06267685443162918, + 0.09210194647312164, + 0.09862504154443741, + 0.12781774997711182 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.517225458286703e-05, + 8.485139551339671e-05, + 7.773709512548521e-05, + 7.268197805387899e-05, + 7.360271411016583e-05, + 6.907495844643563e-05, + 6.462758028646931e-05, + 5.963841249467805e-05, + 6.062182001187466e-05, + 5.6874352594604716e-05, + 5.399485235102475e-05, + 5.128573320689611e-05, + 5.339847120922059e-05, + 5.103225339553319e-05, + 4.874135265708901e-05, + 4.698809061665088e-05, + 5.213888653088361e-05, + 5.099077316117473e-05, + 5.510512710316107e-05, + 6.177843170007691e-05, + 0.0001168480230262503, + 9.295561903854832e-05, + 7.254831871250644e-05, + 5.577508272835985e-05, + 3.9492573705501854e-05, + 3.742556509678252e-05, + 3.597760951379314e-05, + 3.5211978683946654e-05, + 4.5540353312389925e-05, + 3.766817826544866e-05, + 3.6782610550289974e-05, + 3.538792589097284e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 7.21732985766721e-05, + "first_to_last_ratio": 1.9025090978900423, + "imbalance_abs_log_ratio": 0.6431735928132513, + "last_quartile_mean": 3.7935849377390696e-05, + "mean": 5.818594911488617e-05, + "normalized": [ + 1.2919313979813576, + 1.4582798219181838, + 1.3360114651046762, + 1.2491328088568412, + 1.2649568363118007, + 1.1871415607580704, + 1.1107076754710035, + 1.024962441996504, + 1.0418635587120688, + 0.9774585352609484, + 0.927970638485483, + 0.8814109589522066, + 0.9177210653346417, + 0.8770545839988233, + 0.8376825229893711, + 0.8075504710574455, + 0.8960734906624477, + 0.8763416930863358, + 0.9470521309939943, + 1.0617414107673573, + 2.008182813956302, + 1.5975612747161798, + 1.2468357020225322, + 0.9585661758001721, + 0.6787304204237541, + 0.6432062321933947, + 0.6183212624538714, + 0.605162916813848, + 0.7826692527172162, + 0.6473758499852623, + 0.6321562354798744, + 0.6081867947380318 + ], + "population_cv": 0.31406897605312034 + }, + "activation_output_rms_by_block": [ + 0.08465143293142319, + 0.18075311183929443, + 0.20576956868171692, + 0.21735580265522003, + 0.09275905787944794, + 0.10715322941541672, + 0.11858425289392471, + 0.13457369804382324, + 0.04503215476870537, + 0.06750404834747314, + 0.0899076983332634, + 0.11102772504091263, + 0.04522238299250603, + 0.07392862439155579, + 0.099777951836586, + 0.124529629945755, + 0.06231136992573738, + 0.0930875614285469, + 0.1175178736448288, + 0.14541363716125488, + 0.048506781458854675, + 0.07990526407957077, + 0.11170225590467453, + 0.15582118928432465, + 0.07912440598011017, + 0.11874818056821823, + 0.15173569321632385, + 0.1927577704191208, + 0.059721093624830246, + 0.11584942042827606, + 0.18759150803089142, + 0.2713634669780731 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.1427000192925334, + "first_to_last_ratio": 0.9700130524107672, + "imbalance_abs_log_ratio": 0.030445751481666544, + "last_quartile_mean": 0.14711144240573049, + "mean": 0.11842774506658316, + "normalized": [ + 0.714793926742673, + 1.5262733554343226, + 1.7375114975466932, + 1.8353452776882389, + 0.7832544462220098, + 0.9047983591612961, + 1.001321546967339, + 1.1363358980462088, + 0.3802500397469117, + 0.5700019730133392, + 0.7591776596161225, + 0.9375144733059804, + 0.3818563206373711, + 0.6242508826794869, + 0.8425217568779025, + 1.051524116040048, + 0.5261551665169704, + 0.786028319429798, + 0.9923170755193995, + 1.2278679888694963, + 0.40958967370005145, + 0.6747174324280679, + 0.9432101898239523, + 1.3157490180761098, + 0.6681238922148215, + 1.0027057468792884, + 1.281251223106664, + 1.6276402992453107, + 0.5042829582818916, + 0.9782287112124148, + 1.5840165488705589, + 2.29138422609926 + ], + "population_cv": 0.45325373930464574 + }, + "bits_per_byte": 2.0817333844570136, + "branch_output_rms_by_sublayer": [ + 0.04390624910593033, + 0.06928635388612747, + 0.11228280514478683, + 0.028686286881566048, + 0.04998920112848282, + 0.010055449791252613, + 0.033025383949279785, + 0.018092481419444084, + 0.09138742089271545, + 0.010032282210886478, + 0.028335820883512497, + 0.010672114789485931, + 0.03341131657361984, + 0.012925518676638603, + 0.037388354539871216, + 0.025385383516550064, + 0.0395183190703392, + 0.01774551346898079, + 0.02706139162182808, + 0.019615843892097473, + 0.030180802568793297, + 0.023827431723475456, + 0.026625270023941994, + 0.026467876508831978, + 0.03392622619867325, + 0.02486017346382141, + 0.023459358140826225, + 0.029297394677996635, + 0.022350549697875977, + 0.03025609254837036, + 0.023843606933951378, + 0.03220728039741516, + 0.05231805890798569, + 0.027116769924759865, + 0.034788578748703, + 0.02850901335477829, + 0.03384474292397499, + 0.02328450046479702, + 0.038353417068719864, + 0.02996371127665043, + 0.038595229387283325, + 0.02724728174507618, + 0.036492832005023956, + 0.03831745311617851, + 0.037211425602436066, + 0.049612995237112045, + 0.039970748126506805, + 0.07030529528856277, + 0.0609821118414402, + 0.044548459351062775, + 0.04000308737158775, + 0.04832696169614792, + 0.028224391862750053, + 0.054304689168930054, + 0.015577503480017185, + 0.0742250457406044, + 0.015223544090986252, + 0.05709763616323471, + 0.009802255779504776, + 0.06710739433765411, + 0.013497802428901196, + 0.08462034165859222, + 0.01913478970527649, + 0.09220738708972931 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00037687773921043333, + 0.00026279280020208946, + 0.00016239452254986574, + 0.0001543457228984768, + 0.000193959576339879, + 0.0001106127485487289, + 0.00012049326147307754, + 0.00015979187292301756, + 0.0001471617255255906, + 0.00011917325219023687, + 0.00013004863800114487, + 0.00013021151769623154, + 0.00012804667363973534, + 0.00013264725187459112, + 0.0001317253306671419, + 0.00013545879620193265, + 0.00015634571029041953, + 0.00015444803058446918, + 0.00017147018797781908, + 0.00021266064648357377, + 0.0004684058636397824, + 0.0004994826280675798, + 0.0004365981142841108, + 0.00037584478572642025, + 0.00020019134799683643, + 0.00017010108071652648, + 0.00015733873594452418, + 0.00015531906314983726, + 0.00021441198712950573, + 0.00018956133022331713, + 0.0001885268756838635, + 0.00018789008488053627 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00019265853051819604, + "first_to_last_ratio": 1.0532533187701352, + "imbalance_abs_log_ratio": 0.051883772852451944, + "last_quartile_mean": 0.00018291756321561837, + "mean": 0.00020419805946004046, + "normalized": [ + 1.8456479959065653, + 1.2869505268413945, + 0.7952794604379908, + 0.7558628289936355, + 0.949860037126528, + 0.5416934364666415, + 0.5900803454826997, + 0.78253374858485, + 0.7206813126174134, + 0.5836159878569164, + 0.6368749945275268, + 0.6376726499779134, + 0.6270709622722581, + 0.6496009424641417, + 0.6450861042238222, + 0.6633696547368052, + 0.7656571796218017, + 0.7563638508263745, + 0.8397248653157456, + 1.0414430335229967, + 2.293880093074267, + 2.4460694160775, + 2.138110986160222, + 1.8405894097145883, + 0.9803783078360345, + 0.833020064766157, + 0.7705202309981493, + 0.7606294768938863, + 1.0500197097684174, + 0.9283209190360225, + 0.9232549818660561, + 0.9201364860046797 + ], + "population_cv": 0.5126365322353433 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.4651530385017395, + "mean_weights": [ + 0.8127976655960083, + 0.1872023493051529 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6420744061470032, + "mean_weights": [ + 0.5006070137023926, + 0.4993929862976074 + ], + "sources": 2 + }, + { + "entropy_mean": 0.4848511219024658, + "mean_weights": [ + 0.8054038882255554, + 0.1945960968732834 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6888383626937866, + "mean_weights": [ + 0.48984581232070923, + 0.5101541876792908 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6307179927825928, + "mean_weights": [ + 0.6166225075721741, + 0.3833775222301483 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6908442378044128, + "mean_weights": [ + 0.5011590719223022, + 0.49884092807769775 + ], + "sources": 2 + }, + { + "entropy_mean": 0.610953688621521, + "mean_weights": [ + 0.6748899221420288, + 0.3251100480556488 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6780859231948853, + "mean_weights": [ + 0.522329568862915, + 0.47767046093940735 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0578476190567017, + "mean_weights": [ + 0.36196666955947876, + 0.24975785613059998, + 0.38827547430992126 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0931516885757446, + "mean_weights": [ + 0.35610058903694153, + 0.30103904008865356, + 0.3428604006767273 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0491304397583008, + "mean_weights": [ + 0.383840948343277, + 0.2327791005373001, + 0.3833799362182617 + ], + "sources": 3 + }, + { + "entropy_mean": 1.082998275756836, + "mean_weights": [ + 0.3329922556877136, + 0.2672775983810425, + 0.3997301161289215 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0236760377883911, + "mean_weights": [ + 0.30450838804244995, + 0.22245538234710693, + 0.4730361998081207 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0747684240341187, + "mean_weights": [ + 0.3861541748046875, + 0.23807209730148315, + 0.37577372789382935 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0452728271484375, + "mean_weights": [ + 0.34277939796447754, + 0.22071735560894012, + 0.43650326132774353 + ], + "sources": 3 + }, + { + "entropy_mean": 1.090815544128418, + "mean_weights": [ + 0.3553965091705322, + 0.29299598932266235, + 0.3516075015068054 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3526535034179688, + "mean_weights": [ + 0.2596277594566345, + 0.17169977724552155, + 0.2613668143749237, + 0.30730560421943665 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3789772987365723, + "mean_weights": [ + 0.2838491201400757, + 0.22714993357658386, + 0.2223297357559204, + 0.26667118072509766 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3559783697128296, + "mean_weights": [ + 0.27299031615257263, + 0.2011382281780243, + 0.22656817734241486, + 0.299303263425827 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3727587461471558, + "mean_weights": [ + 0.27545708417892456, + 0.22125399112701416, + 0.20615285634994507, + 0.2971360385417938 + ], + "sources": 4 + }, + { + "entropy_mean": 1.347727656364441, + "mean_weights": [ + 0.3159601092338562, + 0.208219975233078, + 0.19020767509937286, + 0.2856122553348541 + ], + "sources": 4 + }, + { + "entropy_mean": 1.367835521697998, + "mean_weights": [ + 0.2889861464500427, + 0.20999683439731598, + 0.2041228860616684, + 0.2968941330909729 + ], + "sources": 4 + }, + { + "entropy_mean": 1.349614143371582, + "mean_weights": [ + 0.3384607434272766, + 0.19480931758880615, + 0.19655916094779968, + 0.27017080783843994 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3714534044265747, + "mean_weights": [ + 0.2676551640033722, + 0.2316403090953827, + 0.20704731345176697, + 0.29365724325180054 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5877301692962646, + "mean_weights": [ + 0.23911790549755096, + 0.165030837059021, + 0.16361436247825623, + 0.19953684508800507, + 0.23270004987716675 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6011887788772583, + "mean_weights": [ + 0.2090151607990265, + 0.19815683364868164, + 0.16081154346466064, + 0.20760394632816315, + 0.22441251575946808 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5997718572616577, + "mean_weights": [ + 0.19353170692920685, + 0.18450140953063965, + 0.18869927525520325, + 0.2140544056892395, + 0.21921320259571075 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5949475765228271, + "mean_weights": [ + 0.20263779163360596, + 0.19673538208007812, + 0.15482646226882935, + 0.1956186443567276, + 0.2501817047595978 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5756914615631104, + "mean_weights": [ + 0.1924477517604828, + 0.16689088940620422, + 0.16826164722442627, + 0.20688454806804657, + 0.26551520824432373 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5845041275024414, + "mean_weights": [ + 0.20030935108661652, + 0.20027855038642883, + 0.14635680615901947, + 0.1845313161611557, + 0.2685239613056183 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5699039697647095, + "mean_weights": [ + 0.23370817303657532, + 0.1585724949836731, + 0.15010330080986023, + 0.17950938642024994, + 0.2781066298484802 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5590252876281738, + "mean_weights": [ + 0.2071877419948578, + 0.20685800909996033, + 0.11514061689376831, + 0.18881018459796906, + 0.2820034623146057 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7723299264907837, + "mean_weights": [ + 0.15703797340393066, + 0.14098086953163147, + 0.1455405056476593, + 0.16996634006500244, + 0.21217042207717896, + 0.17430385947227478 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7444664239883423, + "mean_weights": [ + 0.17010627686977386, + 0.17276832461357117, + 0.09049077332019806, + 0.13107657432556152, + 0.1936773955821991, + 0.2418806552886963 + ], + "sources": 6 + }, + { + "entropy_mean": 1.781677007675171, + "mean_weights": [ + 0.16565977036952972, + 0.14947660267353058, + 0.14714768528938293, + 0.1545170545578003, + 0.1947079300880432, + 0.18849092721939087 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7424967288970947, + "mean_weights": [ + 0.1646571159362793, + 0.172885462641716, + 0.09366955608129501, + 0.13456930220127106, + 0.1805109828710556, + 0.2537075877189636 + ], + "sources": 6 + }, + { + "entropy_mean": 1.763336181640625, + "mean_weights": [ + 0.1412973254919052, + 0.14228719472885132, + 0.16781359910964966, + 0.14783284068107605, + 0.1722586452960968, + 0.22851040959358215 + ], + "sources": 6 + }, + { + "entropy_mean": 1.747908592224121, + "mean_weights": [ + 0.17352715134620667, + 0.19590088725090027, + 0.09425890445709229, + 0.13077187538146973, + 0.16420339047908783, + 0.24133780598640442 + ], + "sources": 6 + }, + { + "entropy_mean": 1.751960039138794, + "mean_weights": [ + 0.1551324725151062, + 0.1493334174156189, + 0.13342270255088806, + 0.14193940162658691, + 0.164256751537323, + 0.25591522455215454 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7609000205993652, + "mean_weights": [ + 0.1752166450023651, + 0.19103571772575378, + 0.10182008147239685, + 0.14190642535686493, + 0.17088012397289276, + 0.21914100646972656 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7959256172180176, + "mean_weights": [ + 0.09988345205783844, + 0.10111536085605621, + 0.11175689101219177, + 0.10716581344604492, + 0.10944622755050659, + 0.11854209005832672, + 0.35209017992019653 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8758430480957031, + "mean_weights": [ + 0.138294517993927, + 0.14908835291862488, + 0.07905811071395874, + 0.10475936532020569, + 0.11814114451408386, + 0.14923861622810364, + 0.2614198923110962 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7295689582824707, + "mean_weights": [ + 0.09518752992153168, + 0.09867724776268005, + 0.1073126494884491, + 0.10504774749279022, + 0.09614013135433197, + 0.09949200600385666, + 0.3981426954269409 + ], + "sources": 7 + }, + { + "entropy_mean": 1.894284725189209, + "mean_weights": [ + 0.14533184468746185, + 0.15630647540092468, + 0.07761301845312119, + 0.10974875092506409, + 0.1278315782546997, + 0.15352195501327515, + 0.22964635491371155 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7287417650222778, + "mean_weights": [ + 0.10178729891777039, + 0.10765312612056732, + 0.10282261669635773, + 0.09695686399936676, + 0.09294428676366806, + 0.1002386063337326, + 0.39759719371795654 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9077026844024658, + "mean_weights": [ + 0.1482391059398651, + 0.1696133315563202, + 0.08589586615562439, + 0.10970748960971832, + 0.12991493940353394, + 0.15111537277698517, + 0.20551389455795288 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7764999866485596, + "mean_weights": [ + 0.108901746571064, + 0.1258884072303772, + 0.10638659447431564, + 0.09906120598316193, + 0.09665359556674957, + 0.10193656384944916, + 0.3611718714237213 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9226438999176025, + "mean_weights": [ + 0.15282699465751648, + 0.18017172813415527, + 0.09562167525291443, + 0.12072794884443283, + 0.13344143331050873, + 0.155026376247406, + 0.16218385100364685 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0483555793762207, + "mean_weights": [ + 0.12366196513175964, + 0.1489267647266388, + 0.07818387448787689, + 0.10335356742143631, + 0.11486630141735077, + 0.138092041015625, + 0.12528608739376068, + 0.1676294207572937 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0514719486236572, + "mean_weights": [ + 0.12247532606124878, + 0.14577919244766235, + 0.09528691321611404, + 0.1065284013748169, + 0.1054653450846672, + 0.1176929622888565, + 0.133324533700943, + 0.17344734072685242 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.032230854034424, + "mean_weights": [ + 0.11086852103471756, + 0.1275845170021057, + 0.10895976424217224, + 0.11432527750730515, + 0.09906959533691406, + 0.09175725281238556, + 0.15711331367492676, + 0.19032172858715057 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0012054443359375, + "mean_weights": [ + 0.10136356949806213, + 0.12203269451856613, + 0.10979677736759186, + 0.12125705927610397, + 0.1056654155254364, + 0.09163117408752441, + 0.19582507014274597, + 0.1524282544851303 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0921454429626465, + "mean_weights": [ + 0.08922591060400009, + 0.10335547477006912, + 0.0791240930557251, + 0.08433033525943756, + 0.07891690731048584, + 0.08416393399238586, + 0.21752747893333435, + 0.1574552357196808, + 0.10590065270662308 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1357202529907227, + "mean_weights": [ + 0.09020715206861496, + 0.10346287488937378, + 0.10054519772529602, + 0.09665106236934662, + 0.09092667698860168, + 0.07800492644309998, + 0.16873988509178162, + 0.13561326265335083, + 0.13584895431995392 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0546798706054688, + "mean_weights": [ + 0.0920742005109787, + 0.1087719202041626, + 0.08028727024793625, + 0.07636439800262451, + 0.06722673028707504, + 0.0676979050040245, + 0.19690822064876556, + 0.14703422784805298, + 0.16363510489463806 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1065030097961426, + "mean_weights": [ + 0.10379628837108612, + 0.10537079721689224, + 0.09759557992219925, + 0.09976489096879959, + 0.0832250565290451, + 0.07068105041980743, + 0.15382170677185059, + 0.1477411985397339, + 0.13800343871116638 + ], + "sources": 9 + }, + { + "entropy_mean": 2.040592670440674, + "mean_weights": [ + 0.07865725457668304, + 0.13316026329994202, + 0.09557083994150162, + 0.10485769808292389, + 0.08062814921140671, + 0.08316262811422348, + 0.21876806020736694, + 0.13236114382743835, + 0.07283399254083633 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1497480869293213, + "mean_weights": [ + 0.10509763658046722, + 0.13008880615234375, + 0.09965328872203827, + 0.11725490540266037, + 0.10137925297021866, + 0.10018149018287659, + 0.1292155385017395, + 0.11170174926519394, + 0.1054273247718811 + ], + "sources": 9 + }, + { + "entropy_mean": 2.097411870956421, + "mean_weights": [ + 0.08779411017894745, + 0.1283528208732605, + 0.09681326150894165, + 0.10164344310760498, + 0.08284278213977814, + 0.07655259966850281, + 0.195144385099411, + 0.13261651992797852, + 0.09824012219905853 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.056080643087625504, + 0.04724235460162163, + 0.0581083819270134, + 0.06172836944460869, + 0.10229703038930893, + 0.0942225307226181, + 0.1127886027097702, + 0.08721354603767395, + 0.11344216018915176, + 0.0775911808013916, + 0.08579424023628235, + 0.07847616076469421, + 0.08559268712997437, + 0.08388359844684601, + 0.08018209785223007, + 0.08067329972982407, + 0.08543439209461212, + 0.05588238313794136, + 0.06336403638124466, + 0.059382498264312744, + 0.062162820249795914, + 0.05949684977531433, + 0.06218026950955391, + 0.05876598134636879, + 0.06768060475587845, + 0.047221262007951736, + 0.05390105023980141, + 0.052360907196998596, + 0.054280854761600494, + 0.051704611629247665, + 0.05718601122498512, + 0.05217532813549042, + 0.06374549865722656, + 0.04900486767292023, + 0.05237171798944473, + 0.05048948526382446, + 0.056452542543411255, + 0.052785832434892654, + 0.06046391278505325, + 0.05767803639173508, + 0.06224922090768814, + 0.03627884015440941, + 0.0458436980843544, + 0.03974709287285805, + 0.049082495272159576, + 0.046506255865097046, + 0.05105996131896973, + 0.05303143709897995, + 0.05296381562948227, + 0.042254336178302765, + 0.044069062918424606, + 0.043082669377326965, + 0.04604873061180115, + 0.044737860560417175, + 0.052035875618457794, + 0.047029029577970505, + 0.05734840780496597, + 0.05523588880896568, + 0.05229348689317703, + 0.057324543595314026, + 0.06147978454828262, + 0.059001464396715164, + 0.05949966236948967, + 0.06294798105955124 + ], + "loss_nats": 1.4429476261138916, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.9272491931915283, + "mean_weights": [ + 0.11622770875692368, + 0.0854349434375763, + 0.0652630478143692, + 0.057709939777851105, + 0.05988496169447899, + 0.0613686665892601, + 0.09450254589319229, + 0.14397841691970825, + 0.315629780292511 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.04390624910593033, + 0.08465143293142319, + 0.16762669384479523, + 0.18075311183929443, + 0.2027391642332077, + 0.20576956868171692, + 0.21098043024539948, + 0.21735580265522003, + 0.09138742089271545, + 0.09275905787944794, + 0.10557211190462112, + 0.10715322941541672, + 0.11656615883111954, + 0.11858425289392471, + 0.12742063403129578, + 0.13457369804382324, + 0.0395183190703392, + 0.04503215476870537, + 0.05950815603137016, + 0.06750404834747314, + 0.07847516983747482, + 0.0899076983332634, + 0.09618669748306274, + 0.11102772504091263, + 0.03392622619867325, + 0.04522238299250603, + 0.05688576027750969, + 0.07392862439155579, + 0.08203151077032089, + 0.099777951836586, + 0.10509888082742691, + 0.124529629945755, + 0.05231805890798569, + 0.06231136992573738, + 0.07987643033266068, + 0.0930875614285469, + 0.10661157220602036, + 0.1175178736448288, + 0.13530665636062622, + 0.14541363716125488, + 0.038595229387283325, + 0.048506781458854675, + 0.06469561159610748, + 0.07990526407957077, + 0.09203410893678665, + 0.11170225590467453, + 0.12540605664253235, + 0.15582118928432465, + 0.0609821118414402, + 0.07912440598011017, + 0.0962463766336441, + 0.11874818056821823, + 0.12394774705171585, + 0.15173569321632385, + 0.15217727422714233, + 0.1927577704191208, + 0.015223544090986252, + 0.059721093624830246, + 0.06095156818628311, + 0.11584942042827606, + 0.11813853681087494, + 0.18759150803089142, + 0.19183191657066345, + 0.2713634669780731 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.17859948053956e-05, + 7.907274994067848e-05, + 7.224992441479117e-05, + 6.829771882621571e-05, + 7.397234003292397e-05, + 6.929977826075628e-05, + 6.524410127894953e-05, + 5.8604124205885455e-05, + 5.7273817219538614e-05, + 5.415566556621343e-05, + 5.203393448027782e-05, + 4.97427572554443e-05, + 5.469415191328153e-05, + 5.181931555853225e-05, + 4.854418148170225e-05, + 4.6246310375863686e-05, + 5.25364994246047e-05, + 5.030919419368729e-05, + 5.219681770540774e-05, + 5.9492314903764054e-05, + 0.00011549922783160582, + 9.179249900626019e-05, + 7.052128057694063e-05, + 5.393161700339988e-05, + 4.102441744180396e-05, + 3.842847581836395e-05, + 3.6664387152995914e-05, + 3.585071681300178e-05, + 5.729396070819348e-05, + 3.996978921350092e-05, + 3.8460828363895416e-05, + 3.506506618577987e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 6.981584147069952e-05, + "first_to_last_ratio": 1.7304833708290828, + "imbalance_abs_log_ratio": 0.548400774586867, + "last_quartile_mean": 4.034470521219191e-05, + "mean": 5.756481118623924e-05, + "normalized": [ + 1.2470464738109992, + 1.373629971352719, + 1.2551057308438176, + 1.186449106994416, + 1.2850270592150734, + 1.2038566067132739, + 1.1334025064004034, + 1.0180546587095323, + 0.9949449331856718, + 0.9407772639261125, + 0.9039191375427709, + 0.864117439637068, + 0.9501317000125254, + 0.900190836913916, + 0.8432961123532122, + 0.8033781301956008, + 0.9126495569425832, + 0.873957425673162, + 0.9067486999398913, + 1.033484062186685, + 2.006420683947552, + 1.5945939388089752, + 1.2250762075599164, + 0.9368851541771778, + 0.7126648484797041, + 0.6675688676201236, + 0.6369236065827741, + 0.6227887501795859, + 0.9952948603067684, + 0.694344138195586, + 0.6681308871050272, + 0.6091406444873758 + ], + "population_cv": 0.29718622148268464 + }, + "activation_output_rms_by_block": [ + 0.12353726476430893, + 0.2644505798816681, + 0.28546804189682007, + 0.2978324592113495, + 0.12184728682041168, + 0.13008765876293182, + 0.14342248439788818, + 0.17330849170684814, + 0.04824596643447876, + 0.07406596094369888, + 0.10108275711536407, + 0.1346256583929062, + 0.05460091680288315, + 0.0907488763332367, + 0.12431846559047699, + 0.15815801918506622, + 0.06505855917930603, + 0.10316865891218185, + 0.13202176988124847, + 0.16410958766937256, + 0.05883970111608505, + 0.10281188040971756, + 0.14683762192726135, + 0.20361240208148956, + 0.0998431146144867, + 0.1491125077009201, + 0.1955009251832962, + 0.24479439854621887, + 0.07111947983503342, + 0.13959142565727234, + 0.21623636782169342, + 0.3244384825229645 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.1924942834302783, + "first_to_last_ratio": 1.0689400495146373, + "imbalance_abs_log_ratio": 0.06666754956726341, + "last_quartile_mean": 0.1800795877352357, + "mean": 0.14821555535309017, + "normalized": [ + 0.8334972970280294, + 1.7842295921752218, + 1.926032940447828, + 2.009454800488591, + 0.8220951338753748, + 0.8776923478310173, + 0.9676614850325017, + 1.1693002889877497, + 0.32551216584213183, + 0.4997178654241346, + 0.6819983022332384, + 0.9083099143824067, + 0.3683885721226005, + 0.612276330355798, + 0.8387680044399946, + 1.067081109052895, + 0.43894555483274794, + 0.6960717359686422, + 0.8907416604602422, + 1.1072359259351587, + 0.39698735383012074, + 0.6936645763313535, + 0.9907031794163157, + 1.3737586557391286, + 0.6736345208614101, + 1.0060516748440684, + 1.3190310876450133, + 1.651610709571282, + 0.47983816317799627, + 0.9418135992860346, + 1.4589316708800166, + 2.1889637815009557 + ], + "population_cv": 0.48570418956964106 + }, + "bits_per_byte": 1.9472061471950126, + "branch_output_rms_by_sublayer": [ + 0.07071450352668762, + 0.0979333221912384, + 0.17061099410057068, + 0.02671070583164692, + 0.05700859799981117, + 0.010335823521018028, + 0.04031931236386299, + 0.023449961096048355, + 0.12009353935718536, + 0.009274116717278957, + 0.03257675841450691, + 0.010161085985600948, + 0.03697984665632248, + 0.014820744283497334, + 0.05203799158334732, + 0.042134903371334076, + 0.0411447174847126, + 0.023270873352885246, + 0.0289313904941082, + 0.02695363759994507, + 0.03025255724787712, + 0.029521740972995758, + 0.035415731370449066, + 0.03637884929776192, + 0.03765307366847992, + 0.03330545872449875, + 0.030057422816753387, + 0.03862540423870087, + 0.026875250041484833, + 0.04363904893398285, + 0.03014916367828846, + 0.046239323914051056, + 0.04659067839384079, + 0.038853347301483154, + 0.03827345743775368, + 0.04027938097715378, + 0.036509063094854355, + 0.031818244606256485, + 0.04758172854781151, + 0.03596416860818863, + 0.04452156648039818, + 0.035672497004270554, + 0.04627932235598564, + 0.05545049160718918, + 0.04934258386492729, + 0.07325158268213272, + 0.04957364872097969, + 0.1029544249176979, + 0.07435838133096695, + 0.06102846935391426, + 0.05453169718384743, + 0.0670262798666954, + 0.05140743777155876, + 0.07338068634271622, + 0.02511586621403694, + 0.09379521012306213, + 0.01625087857246399, + 0.06857483834028244, + 0.011866848915815353, + 0.08694677799940109, + 0.015735991299152374, + 0.09746073931455612, + 0.028070110827684402, + 0.12606686353683472 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00029404178447722416, + 0.00019304255956536327, + 0.00012326449658437447, + 0.0001433295397495038, + 0.00013833056276232184, + 8.724110244115487e-05, + 0.00011666904014735888, + 0.00017615183213780098, + 0.00013948755671820993, + 0.00012045858309473033, + 0.00012948412273725043, + 0.00015442371668138337, + 0.00014708637596099402, + 0.0001521699070379073, + 0.00016539804287907646, + 0.00016393962451591415, + 0.00017691762620668453, + 0.00018640354636184297, + 0.00018168808710531447, + 0.00023094735548300326, + 0.0005384882591201274, + 0.0005835676186900734, + 0.0005372744771449539, + 0.0004735823778547737, + 0.00025710320690722725, + 0.00022727983838179898, + 0.00020973440248118995, + 0.00019654852993909036, + 0.0003070629516988005, + 0.00025083255554336065, + 0.0002310212718002874, + 0.00023003778645078012 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00015900886473313778, + "first_to_last_ratio": 0.6661380567951849, + "imbalance_abs_log_ratio": 0.40625833743860384, + "last_quartile_mean": 0.0002387025679003169, + "mean": 0.00022696902308312115, + "normalized": [ + 1.295515046427897, + 0.8505238157308663, + 0.543089514639327, + 0.6314938391262904, + 0.6094689085023822, + 0.38437448976993316, + 0.514030675034593, + 0.7761051695291926, + 0.6145664938035463, + 0.5307269752428537, + 0.5704924882627284, + 0.6803735354882765, + 0.6480460371331276, + 0.6704435035708783, + 0.7287251829889755, + 0.7222995556352894, + 0.7794791721066417, + 0.8212730699095347, + 0.800497286534014, + 1.01752808531244, + 2.372518733197001, + 2.571133323671036, + 2.3671709462669357, + 2.0865507170171727, + 1.132767826264425, + 1.0013694172643286, + 0.9240661991322953, + 0.8659707270586872, + 1.3528848453753406, + 1.1051400412976187, + 1.0178537522967714, + 1.0135206264096008 + ], + "population_cv": 0.5593182694321974 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.2543017268180847, + "mean_weights": [ + 0.9256092309951782, + 0.07439076900482178 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6060795187950134, + "mean_weights": [ + 0.588118314743042, + 0.4118817448616028 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5039626359939575, + "mean_weights": [ + 0.7907090187072754, + 0.20929095149040222 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6840331554412842, + "mean_weights": [ + 0.4493589997291565, + 0.5506409406661987 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6448079347610474, + "mean_weights": [ + 0.5822745561599731, + 0.41772544384002686 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6884729862213135, + "mean_weights": [ + 0.49841398000717163, + 0.5015859603881836 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5978472828865051, + "mean_weights": [ + 0.6859915852546692, + 0.31400835514068604 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6744088530540466, + "mean_weights": [ + 0.539104700088501, + 0.4608953893184662 + ], + "sources": 2 + }, + { + "entropy_mean": 0.9910719990730286, + "mean_weights": [ + 0.270734965801239, + 0.1987934410572052, + 0.5304716229438782 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0872347354888916, + "mean_weights": [ + 0.3574308753013611, + 0.2692360281944275, + 0.3733330965042114 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0089956521987915, + "mean_weights": [ + 0.3301823139190674, + 0.195428267121315, + 0.4743894338607788 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0733987092971802, + "mean_weights": [ + 0.36850234866142273, + 0.24504244327545166, + 0.3864552080631256 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9976938366889954, + "mean_weights": [ + 0.2819531559944153, + 0.20378541946411133, + 0.5142614245414734 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0640380382537842, + "mean_weights": [ + 0.4051903486251831, + 0.21782611310482025, + 0.37698355317115784 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0448607206344604, + "mean_weights": [ + 0.3460210859775543, + 0.21934065222740173, + 0.43463826179504395 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0859339237213135, + "mean_weights": [ + 0.3818342089653015, + 0.2904532849788666, + 0.3277125358581543 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3507157564163208, + "mean_weights": [ + 0.2784358263015747, + 0.16773328185081482, + 0.2621825933456421, + 0.29164832830429077 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3713499307632446, + "mean_weights": [ + 0.30462729930877686, + 0.2265601009130478, + 0.20282697677612305, + 0.2659856081008911 + ], + "sources": 4 + }, + { + "entropy_mean": 1.337369441986084, + "mean_weights": [ + 0.2833084464073181, + 0.1836237907409668, + 0.23752525448799133, + 0.29554250836372375 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3644449710845947, + "mean_weights": [ + 0.290496289730072, + 0.2083635926246643, + 0.19614923000335693, + 0.30499088764190674 + ], + "sources": 4 + }, + { + "entropy_mean": 1.333448052406311, + "mean_weights": [ + 0.32733020186424255, + 0.20071643590927124, + 0.20320233702659607, + 0.26875102519989014 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3683173656463623, + "mean_weights": [ + 0.2969692349433899, + 0.20804961025714874, + 0.2077639251947403, + 0.2872171998023987 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3367315530776978, + "mean_weights": [ + 0.36340227723121643, + 0.19346699118614197, + 0.19746294617652893, + 0.24566778540611267 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3721983432769775, + "mean_weights": [ + 0.270467072725296, + 0.23268075287342072, + 0.19789132475852966, + 0.2989608347415924 + ], + "sources": 4 + }, + { + "entropy_mean": 1.578795075416565, + "mean_weights": [ + 0.24240365624427795, + 0.1535990685224533, + 0.17186322808265686, + 0.18831008672714233, + 0.24382396042346954 + ], + "sources": 5 + }, + { + "entropy_mean": 1.592212438583374, + "mean_weights": [ + 0.20627960562705994, + 0.1938011795282364, + 0.15111781656742096, + 0.19027581810951233, + 0.2585255801677704 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5892239809036255, + "mean_weights": [ + 0.1897662878036499, + 0.18326935172080994, + 0.200483500957489, + 0.19280505180358887, + 0.23367583751678467 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5840388536453247, + "mean_weights": [ + 0.2059454321861267, + 0.1968708038330078, + 0.15055322647094727, + 0.17195186018943787, + 0.27467870712280273 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5437142848968506, + "mean_weights": [ + 0.18356269598007202, + 0.14464262127876282, + 0.17406654357910156, + 0.19471389055252075, + 0.30301424860954285 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5758908987045288, + "mean_weights": [ + 0.20305782556533813, + 0.18869087100028992, + 0.14675945043563843, + 0.17530936002731323, + 0.2861824631690979 + ], + "sources": 5 + }, + { + "entropy_mean": 1.538482427597046, + "mean_weights": [ + 0.2407410740852356, + 0.1416442096233368, + 0.14945383369922638, + 0.15778927505016327, + 0.31037160754203796 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5542782545089722, + "mean_weights": [ + 0.212637796998024, + 0.21009932458400726, + 0.10470656305551529, + 0.18270303308963776, + 0.2898532748222351 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7608966827392578, + "mean_weights": [ + 0.15334206819534302, + 0.12689590454101562, + 0.15951073169708252, + 0.16330736875534058, + 0.22038079798221588, + 0.1765631139278412 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7265489101409912, + "mean_weights": [ + 0.16850659251213074, + 0.17107035219669342, + 0.07948238402605057, + 0.12312246859073639, + 0.1926162838935852, + 0.2652018964290619 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7728970050811768, + "mean_weights": [ + 0.1661251187324524, + 0.13225069642066956, + 0.14822669327259064, + 0.14951856434345245, + 0.19571126997470856, + 0.2081676423549652 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7339119911193848, + "mean_weights": [ + 0.1636144518852234, + 0.1873069405555725, + 0.08960031718015671, + 0.12653839588165283, + 0.16886180639266968, + 0.2640780806541443 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7245814800262451, + "mean_weights": [ + 0.13370932638645172, + 0.11571834236383438, + 0.16499628126621246, + 0.13595008850097656, + 0.1706637293100357, + 0.27896222472190857 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7347569465637207, + "mean_weights": [ + 0.1758565902709961, + 0.213505819439888, + 0.08526134490966797, + 0.1235002875328064, + 0.15748678147792816, + 0.24438917636871338 + ], + "sources": 6 + }, + { + "entropy_mean": 1.723979115486145, + "mean_weights": [ + 0.153537780046463, + 0.13428238034248352, + 0.12782113254070282, + 0.1299300491809845, + 0.16199655830860138, + 0.29243209958076477 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7542636394500732, + "mean_weights": [ + 0.17849677801132202, + 0.20540779829025269, + 0.09721188247203827, + 0.13475516438484192, + 0.16575977206230164, + 0.21836858987808228 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7827293872833252, + "mean_weights": [ + 0.10179544985294342, + 0.09573351591825485, + 0.10741458088159561, + 0.10095835477113724, + 0.11035231500864029, + 0.12350213527679443, + 0.36024367809295654 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8723649978637695, + "mean_weights": [ + 0.13885334134101868, + 0.1545029878616333, + 0.07525748014450073, + 0.10369499027729034, + 0.11754631996154785, + 0.14988233149051666, + 0.26026254892349243 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7168539762496948, + "mean_weights": [ + 0.09823557734489441, + 0.09504838287830353, + 0.10787523537874222, + 0.10082508623600006, + 0.09352369606494904, + 0.09787303954362869, + 0.40661898255348206 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8875458240509033, + "mean_weights": [ + 0.144264817237854, + 0.16480503976345062, + 0.07487034052610397, + 0.10658131539821625, + 0.1236310601234436, + 0.15269996225833893, + 0.23314747214317322 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7091132402420044, + "mean_weights": [ + 0.10130499303340912, + 0.09899869561195374, + 0.10864657163619995, + 0.09762421250343323, + 0.08961810171604156, + 0.0953778326511383, + 0.4084296226501465 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9038714170455933, + "mean_weights": [ + 0.14696960151195526, + 0.1688399612903595, + 0.08132581412792206, + 0.11436077952384949, + 0.12918002903461456, + 0.14842385053634644, + 0.2108999788761139 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7605948448181152, + "mean_weights": [ + 0.1098497062921524, + 0.11961614340543747, + 0.10800998657941818, + 0.10506489127874374, + 0.09206781536340714, + 0.09388501197099686, + 0.3715064525604248 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9046406745910645, + "mean_weights": [ + 0.1550796627998352, + 0.1951897293329239, + 0.07557649910449982, + 0.11430470645427704, + 0.13275673985481262, + 0.16038885712623596, + 0.16670379042625427 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0288946628570557, + "mean_weights": [ + 0.12433864921331406, + 0.14156323671340942, + 0.07063110917806625, + 0.09557435661554337, + 0.10851188749074936, + 0.13083158433437347, + 0.12725013494491577, + 0.2012990415096283 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.046355724334717, + "mean_weights": [ + 0.12412874400615692, + 0.14707700908184052, + 0.08755914866924286, + 0.10208744555711746, + 0.11010868847370148, + 0.1277797818183899, + 0.1272171586751938, + 0.1740420162677765 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0275957584381104, + "mean_weights": [ + 0.10929238796234131, + 0.12230641394853592, + 0.11058878898620605, + 0.11738988757133484, + 0.103816457092762, + 0.09593221545219421, + 0.15372726321220398, + 0.18694660067558289 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0006895065307617, + "mean_weights": [ + 0.10473694652318954, + 0.1126377284526825, + 0.1065388023853302, + 0.14010563492774963, + 0.11665022373199463, + 0.09570855647325516, + 0.18249712884426117, + 0.14112496376037598 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0442771911621094, + "mean_weights": [ + 0.08640944957733154, + 0.09414205700159073, + 0.07781088352203369, + 0.07340539991855621, + 0.0658595934510231, + 0.07112829387187958, + 0.19808220863342285, + 0.1570216864347458, + 0.1761404275894165 + ], + "sources": 9 + }, + { + "entropy_mean": 2.114872694015503, + "mean_weights": [ + 0.08701683580875397, + 0.09773310273885727, + 0.10395046323537827, + 0.10442344844341278, + 0.09205754101276398, + 0.07338003814220428, + 0.15518538653850555, + 0.1225508376955986, + 0.1637023538351059 + ], + "sources": 9 + }, + { + "entropy_mean": 1.983813762664795, + "mean_weights": [ + 0.09266494959592819, + 0.09886855632066727, + 0.07565010339021683, + 0.07191943377256393, + 0.0560871958732605, + 0.05708993226289749, + 0.16357006132602692, + 0.15972918272018433, + 0.22442059218883514 + ], + "sources": 9 + }, + { + "entropy_mean": 2.102735996246338, + "mean_weights": [ + 0.09974014014005661, + 0.09861430525779724, + 0.09994262456893921, + 0.11571428179740906, + 0.09006108343601227, + 0.07509303092956543, + 0.1392432004213333, + 0.12204292416572571, + 0.15954840183258057 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0002355575561523, + "mean_weights": [ + 0.08056170493364334, + 0.12302123755216599, + 0.09211792051792145, + 0.10765204578638077, + 0.07560212910175323, + 0.07419829070568085, + 0.21178759634494781, + 0.144280344247818, + 0.09077870845794678 + ], + "sources": 9 + }, + { + "entropy_mean": 2.142282009124756, + "mean_weights": [ + 0.10089981555938721, + 0.13316139578819275, + 0.09534043073654175, + 0.13839571177959442, + 0.10801589488983154, + 0.09521867334842682, + 0.12368979305028915, + 0.0995693951845169, + 0.10570887476205826 + ], + "sources": 9 + }, + { + "entropy_mean": 2.089121103286743, + "mean_weights": [ + 0.08974401652812958, + 0.11299927532672882, + 0.09783615171909332, + 0.10564310848712921, + 0.08262784034013748, + 0.06969405710697174, + 0.16734099388122559, + 0.1368718296289444, + 0.13724273443222046 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05914194509387016, + 0.055478259921073914, + 0.07131855934858322, + 0.07814061641693115, + 0.15345832705497742, + 0.13287605345249176, + 0.1535882204771042, + 0.11014963686466217, + 0.14737600088119507, + 0.09671343863010406, + 0.10441362857818604, + 0.09596432745456696, + 0.10115501284599304, + 0.10277370363473892, + 0.094146229326725, + 0.09895699471235275, + 0.10367784649133682, + 0.06869825720787048, + 0.07840369641780853, + 0.07111510634422302, + 0.07408348470926285, + 0.07285869121551514, + 0.07572652399539948, + 0.07137154042720795, + 0.08140570670366287, + 0.05535653233528137, + 0.06326576322317123, + 0.06337618082761765, + 0.0647912248969078, + 0.059228237718343735, + 0.06690949946641922, + 0.06113220006227493, + 0.07740576565265656, + 0.05723513662815094, + 0.06246957555413246, + 0.05797118693590164, + 0.06878555566072464, + 0.0612582191824913, + 0.07573212683200836, + 0.068631112575531, + 0.07593145966529846, + 0.043081313371658325, + 0.056522712111473083, + 0.04909126088023186, + 0.06133647263050079, + 0.06029130890965462, + 0.06394942104816437, + 0.06984659284353256, + 0.06925318390130997, + 0.05301627889275551, + 0.05502115935087204, + 0.054338209331035614, + 0.059092920273542404, + 0.05665183439850807, + 0.06676284223794937, + 0.05922900140285492, + 0.07155967503786087, + 0.06734836101531982, + 0.06331570446491241, + 0.07430403679609299, + 0.07136080414056778, + 0.07700474560260773, + 0.06965775787830353, + 0.08010563999414444 + ], + "loss_nats": 1.3497004508972168, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.8327317237854004, + "mean_weights": [ + 0.10852579772472382, + 0.08185571432113647, + 0.06698062270879745, + 0.05133809894323349, + 0.05122953653335571, + 0.054709259420633316, + 0.07709662616252899, + 0.13766279816627502, + 0.3706015646457672 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.07071450352668762, + 0.12353726476430893, + 0.2539069354534149, + 0.2644505798816681, + 0.2833360433578491, + 0.28546804189682007, + 0.29032036662101746, + 0.2978324592113495, + 0.12009353935718536, + 0.12184728682041168, + 0.12877075374126434, + 0.13008765876293182, + 0.14095517992973328, + 0.14342248439788818, + 0.1594909131526947, + 0.17330849170684814, + 0.0411447174847126, + 0.04824596643447876, + 0.06196928396821022, + 0.07406596094369888, + 0.08537201583385468, + 0.10108275711536407, + 0.11382352560758591, + 0.1346256583929062, + 0.03765307366847992, + 0.05460091680288315, + 0.06756448745727539, + 0.0907488763332367, + 0.09880685061216354, + 0.12431846559047699, + 0.13172951340675354, + 0.15815801918506622, + 0.04659067839384079, + 0.06505855917930603, + 0.08199436217546463, + 0.10316865891218185, + 0.11639314144849777, + 0.13202176988124847, + 0.15105262398719788, + 0.16410958766937256, + 0.04452156648039818, + 0.05883970111608505, + 0.07945509254932404, + 0.10281188040971756, + 0.11789502203464508, + 0.14683762192726135, + 0.16081275045871735, + 0.20361240208148956, + 0.07435838133096695, + 0.0998431146144867, + 0.11974400281906128, + 0.1491125077009201, + 0.1621677130460739, + 0.1955009251832962, + 0.19795005023479462, + 0.24479439854621887, + 0.01625087857246399, + 0.07111947983503342, + 0.07255832850933075, + 0.13959142565727234, + 0.14124269783496857, + 0.21623636782169342, + 0.22023120522499084, + 0.3244384825229645 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00011634526890702546, + 0.00012781281839124858, + 0.00011786622053477913, + 0.0001117925567086786, + 0.0001259593409486115, + 0.00011713022104231641, + 0.00010976169141940773, + 9.978538582799956e-05, + 0.00010059804480988532, + 9.385724843014032e-05, + 9.017578850034624e-05, + 8.536517270840704e-05, + 9.590805711923167e-05, + 9.026959742186591e-05, + 8.354819874512032e-05, + 7.888357504270971e-05, + 9.291680908063427e-05, + 8.873649494489655e-05, + 9.039592987392098e-05, + 0.00010356695565860718, + 0.00019465028890408576, + 0.00014947899035178125, + 0.00011215243284823373, + 8.33522281027399e-05, + 6.745223072357476e-05, + 6.283520633587614e-05, + 5.942062853137031e-05, + 5.788906855741516e-05, + 0.00010315173858543858, + 6.46734333713539e-05, + 6.0739574109902605e-05, + 4.9852122174343094e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00011580668797250837, + "first_to_last_ratio": 1.7612715623004438, + "imbalance_abs_log_ratio": 0.5660360267678461, + "last_quartile_mean": 6.575175029865932e-05, + "mean": 9.644760370974836e-05, + "normalized": [ + 1.2063054387246115, + 1.325204706753435, + 1.2220751579218956, + 1.159101443775728, + 1.3059872521838534, + 1.2144440767529154, + 1.1380447745464686, + 1.0346072062950988, + 1.0430331178846848, + 0.9731423573010324, + 0.9349717881195196, + 0.8850937651629682, + 0.9944058061604059, + 0.9359444294077571, + 0.8662547904927963, + 0.8178904608154263, + 0.9633915774647989, + 0.9200487262694695, + 0.9372542851967641, + 1.0738157473594052, + 2.0181972534006216, + 1.5498465965170765, + 1.1628327561745115, + 0.8642229033868173, + 0.6993665796671015, + 0.6514957751047279, + 0.6160923262561516, + 0.6002126162888178, + 1.0695106422329144, + 0.6705551085124273, + 0.629767581294126, + 0.5168829525756723 + ], + "population_cv": 0.2990623914855925 + }, + "activation_output_rms_by_block": [ + 0.11052477359771729, + 0.26237499713897705, + 0.2805008888244629, + 0.2898934483528137, + 0.10940494388341904, + 0.11956621706485748, + 0.1299726366996765, + 0.1622796654701233, + 0.04207194223999977, + 0.06267905235290527, + 0.08533640205860138, + 0.11859145015478134, + 0.046815622597932816, + 0.07779109477996826, + 0.10524991899728775, + 0.13467493653297424, + 0.05643659830093384, + 0.08919516950845718, + 0.11048520356416702, + 0.13601729273796082, + 0.05171631649136543, + 0.0903087705373764, + 0.13047200441360474, + 0.18299677968025208, + 0.09183725714683533, + 0.13468314707279205, + 0.17454643547534943, + 0.2129679024219513, + 0.061451755464076996, + 0.1171356812119484, + 0.17424017190933228, + 0.2611335515975952 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.1830646963790059, + "first_to_last_ratio": 1.1926078648057303, + "imbalance_abs_log_ratio": 0.1761423923570425, + "last_quartile_mean": 0.15349948778748512, + "mean": 0.13166725088376552, + "normalized": [ + 0.8394249356302533, + 1.9927126554088912, + 2.1303770447222763, + 2.2017126233517903, + 0.8309199375629145, + 0.9080938218297676, + 0.9871295696331885, + 1.2324983196723847, + 0.31953232073737486, + 0.4760413233525905, + 0.6481216968214478, + 0.9006905616907931, + 0.3555601129643176, + 0.5908158198627648, + 0.7993629265503636, + 1.0228430807890405, + 0.42863049028611894, + 0.6774286613396201, + 0.8391244050633533, + 1.0330381459702191, + 0.39278040776457057, + 0.6858863531456281, + 0.9909222189865882, + 1.3898427927366672, + 0.6974950606959076, + 1.022905439042613, + 1.3256632481028803, + 1.6174705630480366, + 0.4667201225179826, + 0.8896341377656207, + 1.3233372060236124, + 1.9832839969304228 + ], + "population_cv": 0.5106184609405161 + }, + "bits_per_byte": 1.8782959666646515, + "branch_output_rms_by_sublayer": [ + 0.06572774797677994, + 0.08512533456087112, + 0.18250809609889984, + 0.017605070024728775, + 0.050045792013406754, + 0.007168244570493698, + 0.03047456406056881, + 0.018809039145708084, + 0.10768114775419235, + 0.00784416776150465, + 0.022024493664503098, + 0.008056504651904106, + 0.025202598422765732, + 0.013070429675281048, + 0.047531791031360626, + 0.042359206825494766, + 0.03554074838757515, + 0.020496424287557602, + 0.024192264303565025, + 0.02275150828063488, + 0.026202093809843063, + 0.024744367226958275, + 0.034759439527988434, + 0.03239097073674202, + 0.03065936081111431, + 0.029905376955866814, + 0.026576338335871696, + 0.03431443125009537, + 0.024248600006103516, + 0.03955564647912979, + 0.02839072048664093, + 0.044039808213710785, + 0.038578376173973083, + 0.03548697754740715, + 0.03354169800877571, + 0.03711722791194916, + 0.029022149741649628, + 0.03135069087147713, + 0.039004094898700714, + 0.03401976078748703, + 0.03717563673853874, + 0.034146230667829514, + 0.03800811246037483, + 0.053240738809108734, + 0.04156900942325592, + 0.07125747948884964, + 0.04407264664769173, + 0.10016607493162155, + 0.06643229722976685, + 0.05881091207265854, + 0.04714363068342209, + 0.06436190754175186, + 0.04642924666404724, + 0.07085702568292618, + 0.023332027718424797, + 0.08363397419452667, + 0.01394309289753437, + 0.059312932193279266, + 0.010770239867269993, + 0.07627109438180923, + 0.01487429253757, + 0.07864724844694138, + 0.024073146283626556, + 0.1078467145562172 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00046397832722905823, + 0.0003577824604571417, + 0.0001909347382856166, + 0.00025175893614286324, + 0.00021920850183836641, + 0.00012555039043477353, + 0.00015873849562840633, + 0.000286985595576038, + 0.00023109641241957234, + 0.0001958412218762804, + 0.00021405918509681047, + 0.00026584732741832384, + 0.00025040629990206423, + 0.0002670804405206366, + 0.00029396751102296045, + 0.0002883094298190267, + 0.00029999889470778046, + 0.00033731613208962287, + 0.00031450999988806523, + 0.00039012184898271963, + 0.0008664874114831617, + 0.0009338123657452615, + 0.0008563690710599772, + 0.0007475554611858251, + 0.00042572280224178354, + 0.00037209608685413205, + 0.0003514895396333438, + 0.00031386633481042563, + 0.0005082945923865056, + 0.0003965832677852886, + 0.0003501207530903734, + 0.00033489758998561187 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.000256867180699033, + "first_to_last_ratio": 0.6730722829396045, + "imbalance_abs_log_ratio": 0.3959025510340787, + "last_quartile_mean": 0.00038163387084843306, + "mean": 0.0003706496070499318, + "normalized": [ + 1.2517977043654434, + 0.9652848772855797, + 0.5151354126753329, + 0.6792370242792345, + 0.5914170625542735, + 0.3387306718980552, + 0.42827104793617654, + 0.7742773501372604, + 0.6234902399031557, + 0.528372938082996, + 0.5775243815865523, + 0.7172470235008517, + 0.6755876578288964, + 0.7205739205995086, + 0.7931143199171363, + 0.7778490097797064, + 0.8093867874177089, + 0.9100674212887581, + 0.8485372543392343, + 1.0525354446960595, + 2.3377534873966113, + 2.519393918093278, + 2.3104545499885343, + 2.0168791412886047, + 1.1485856025322392, + 1.0039025531841597, + 0.9483067915029333, + 0.8468006679099038, + 1.3713614764955924, + 1.0699681322793448, + 0.9446138520880913, + 0.9035422771687881 + ], + "population_cv": 0.5421526475176193 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.2618083953857422, + "mean_weights": [ + 0.9230123162269592, + 0.07698766887187958 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5963633060455322, + "mean_weights": [ + 0.6155085563659668, + 0.3844913840293884 + ], + "sources": 2 + }, + { + "entropy_mean": 0.59415203332901, + "mean_weights": [ + 0.7051531076431274, + 0.29484689235687256 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6797515749931335, + "mean_weights": [ + 0.4263187646865845, + 0.5736812353134155 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6653674840927124, + "mean_weights": [ + 0.5093975067138672, + 0.4906025528907776 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6868746280670166, + "mean_weights": [ + 0.4694281220436096, + 0.5305718779563904 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6332509517669678, + "mean_weights": [ + 0.6363139152526855, + 0.3636860251426697 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6836416721343994, + "mean_weights": [ + 0.5244400501251221, + 0.4755600094795227 + ], + "sources": 2 + }, + { + "entropy_mean": 0.9662269353866577, + "mean_weights": [ + 0.23947665095329285, + 0.1952630877494812, + 0.5652602910995483 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0825676918029785, + "mean_weights": [ + 0.3360655903816223, + 0.2621897757053375, + 0.40174466371536255 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9909573197364807, + "mean_weights": [ + 0.29322588443756104, + 0.1908101737499237, + 0.5159639120101929 + ], + "sources": 3 + }, + { + "entropy_mean": 1.067949652671814, + "mean_weights": [ + 0.3422330319881439, + 0.24122801423072815, + 0.4165389835834503 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9776483178138733, + "mean_weights": [ + 0.25922471284866333, + 0.19485288858413696, + 0.5459223985671997 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0666338205337524, + "mean_weights": [ + 0.3824150860309601, + 0.22134432196617126, + 0.39624059200286865 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0446889400482178, + "mean_weights": [ + 0.3259434700012207, + 0.22321224212646484, + 0.45084428787231445 + ], + "sources": 3 + }, + { + "entropy_mean": 1.088794231414795, + "mean_weights": [ + 0.3640216290950775, + 0.2939528822898865, + 0.3420255184173584 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3269968032836914, + "mean_weights": [ + 0.2434360533952713, + 0.1475134789943695, + 0.2533762454986572, + 0.35567426681518555 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3724898099899292, + "mean_weights": [ + 0.2855454087257385, + 0.21955126523971558, + 0.20736029744148254, + 0.28754302859306335 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3196265697479248, + "mean_weights": [ + 0.2502860128879547, + 0.1629919707775116, + 0.23060941696166992, + 0.35611259937286377 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3658894300460815, + "mean_weights": [ + 0.279031902551651, + 0.21006061136722565, + 0.1986178308725357, + 0.31228965520858765 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3331713676452637, + "mean_weights": [ + 0.2946619689464569, + 0.19263458251953125, + 0.21305972337722778, + 0.29964372515678406 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3705391883850098, + "mean_weights": [ + 0.28286588191986084, + 0.21221889555454254, + 0.21037471294403076, + 0.29454052448272705 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3430064916610718, + "mean_weights": [ + 0.32435888051986694, + 0.17917126417160034, + 0.20280948281288147, + 0.29366037249565125 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3731927871704102, + "mean_weights": [ + 0.26912301778793335, + 0.24305197596549988, + 0.1945190727710724, + 0.293305903673172 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5764957666397095, + "mean_weights": [ + 0.22172938287258148, + 0.14426849782466888, + 0.17165306210517883, + 0.2046574354171753, + 0.2576916217803955 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5870283842086792, + "mean_weights": [ + 0.2015727311372757, + 0.19003993272781372, + 0.14829128980636597, + 0.18626853823661804, + 0.27382752299308777 + ], + "sources": 5 + }, + { + "entropy_mean": 1.581924319267273, + "mean_weights": [ + 0.18218614161014557, + 0.17089596390724182, + 0.1979314684867859, + 0.19848117232322693, + 0.250505268573761 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5784693956375122, + "mean_weights": [ + 0.20249375700950623, + 0.19156721234321594, + 0.14371460676193237, + 0.1757340431213379, + 0.28649038076400757 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5356805324554443, + "mean_weights": [ + 0.17397668957710266, + 0.13229301571846008, + 0.17188337445259094, + 0.2026972770690918, + 0.3191496431827545 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5731475353240967, + "mean_weights": [ + 0.19923287630081177, + 0.18257781863212585, + 0.14229342341423035, + 0.18188734352588654, + 0.2940085530281067 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5363487005233765, + "mean_weights": [ + 0.227708637714386, + 0.13274484872817993, + 0.15277844667434692, + 0.1655799150466919, + 0.32118815183639526 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5632460117340088, + "mean_weights": [ + 0.21265456080436707, + 0.21500606834888458, + 0.10978172719478607, + 0.18234726786613464, + 0.28021037578582764 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7500195503234863, + "mean_weights": [ + 0.14760713279247284, + 0.11686383187770844, + 0.1550339013338089, + 0.15950019657611847, + 0.2371332347393036, + 0.18386171758174896 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7314503192901611, + "mean_weights": [ + 0.16667428612709045, + 0.17632552981376648, + 0.08000434190034866, + 0.12810909748077393, + 0.18869392573833466, + 0.2601928114891052 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7592062950134277, + "mean_weights": [ + 0.15652519464492798, + 0.11631648242473602, + 0.14736926555633545, + 0.14542044699192047, + 0.198613703250885, + 0.23575490713119507 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7304072380065918, + "mean_weights": [ + 0.16066193580627441, + 0.19433292746543884, + 0.08561266213655472, + 0.12697631120681763, + 0.16592314839363098, + 0.2664929926395416 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6932260990142822, + "mean_weights": [ + 0.1272033005952835, + 0.10737767815589905, + 0.16237998008728027, + 0.12267744541168213, + 0.16082434356212616, + 0.3195372521877289 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7273521423339844, + "mean_weights": [ + 0.171003520488739, + 0.21605968475341797, + 0.07823343575000763, + 0.12454713881015778, + 0.15885379910469055, + 0.25130242109298706 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7054250240325928, + "mean_weights": [ + 0.14685668051242828, + 0.13186603784561157, + 0.12616053223609924, + 0.12517386674880981, + 0.1507444977760315, + 0.3191983699798584 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7506941556930542, + "mean_weights": [ + 0.17476242780685425, + 0.21295547485351562, + 0.09478215128183365, + 0.13350678980350494, + 0.1624860316514969, + 0.22150711715221405 + ], + "sources": 6 + }, + { + "entropy_mean": 1.775676965713501, + "mean_weights": [ + 0.09844386577606201, + 0.09153492748737335, + 0.10916391015052795, + 0.09866812080144882, + 0.10917358100414276, + 0.12636438012123108, + 0.3666512072086334 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8761906623840332, + "mean_weights": [ + 0.13646182417869568, + 0.16472993791103363, + 0.07423210144042969, + 0.10240940749645233, + 0.11764147132635117, + 0.1542845070362091, + 0.2502407431602478 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7173192501068115, + "mean_weights": [ + 0.09652349352836609, + 0.09494321048259735, + 0.10608730465173721, + 0.09816491603851318, + 0.09510113298892975, + 0.10167565941810608, + 0.40750426054000854 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8882131576538086, + "mean_weights": [ + 0.1422848403453827, + 0.1774555891752243, + 0.07056589424610138, + 0.1058211699128151, + 0.12277501076459885, + 0.16191351413726807, + 0.21918398141860962 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7122434377670288, + "mean_weights": [ + 0.09876047074794769, + 0.09859075397253036, + 0.1078207939863205, + 0.09686246514320374, + 0.09128662198781967, + 0.09884604811668396, + 0.4078328609466553 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9042646884918213, + "mean_weights": [ + 0.14567604660987854, + 0.1752149760723114, + 0.07816295325756073, + 0.11239662766456604, + 0.1302127242088318, + 0.15784329175949097, + 0.20049338042736053 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7617065906524658, + "mean_weights": [ + 0.10847711563110352, + 0.11922790110111237, + 0.10886583477258682, + 0.10137523710727692, + 0.09351018071174622, + 0.09674951434135437, + 0.3717942237854004 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9010262489318848, + "mean_weights": [ + 0.15234190225601196, + 0.19616666436195374, + 0.07168153673410416, + 0.11421913653612137, + 0.13367216289043427, + 0.1689094752073288, + 0.1630091369152069 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0150485038757324, + "mean_weights": [ + 0.1217740997672081, + 0.14565055072307587, + 0.06278535723686218, + 0.09308129549026489, + 0.1060602143406868, + 0.135765939950943, + 0.12078039348125458, + 0.2141021490097046 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0347020626068115, + "mean_weights": [ + 0.12512989342212677, + 0.1512535661458969, + 0.0750519186258316, + 0.10000336915254593, + 0.11018894612789154, + 0.1341707855463028, + 0.12037566304206848, + 0.18382585048675537 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0271873474121094, + "mean_weights": [ + 0.10933255404233932, + 0.12632311880588531, + 0.10214807093143463, + 0.11638906598091125, + 0.10897155106067657, + 0.10483382642269135, + 0.1394592821598053, + 0.19254252314567566 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.017636299133301, + "mean_weights": [ + 0.1068490743637085, + 0.12393505126237869, + 0.09579624235630035, + 0.14464256167411804, + 0.1197507232427597, + 0.11283029615879059, + 0.15944911539554596, + 0.13674694299697876 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0312976837158203, + "mean_weights": [ + 0.08615848422050476, + 0.09558504819869995, + 0.0794157013297081, + 0.07059705257415771, + 0.0679299384355545, + 0.06907594949007034, + 0.19065847992897034, + 0.16998335719108582, + 0.17059597373008728 + ], + "sources": 9 + }, + { + "entropy_mean": 2.122809886932373, + "mean_weights": [ + 0.08954480290412903, + 0.10427558422088623, + 0.09500031918287277, + 0.10513864457607269, + 0.1005379781126976, + 0.08023279160261154, + 0.1318712830543518, + 0.1181824654340744, + 0.17521610856056213 + ], + "sources": 9 + }, + { + "entropy_mean": 1.9292302131652832, + "mean_weights": [ + 0.0891539454460144, + 0.08687238395214081, + 0.07028785347938538, + 0.06637713313102722, + 0.053483493626117706, + 0.05437715724110603, + 0.14970596134662628, + 0.15909138321876526, + 0.2706506848335266 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1081371307373047, + "mean_weights": [ + 0.09939657151699066, + 0.10370293259620667, + 0.09104302525520325, + 0.12695208191871643, + 0.09674926102161407, + 0.08559530228376389, + 0.11822725087404251, + 0.11419164389371872, + 0.1641419231891632 + ], + "sources": 9 + }, + { + "entropy_mean": 1.9956302642822266, + "mean_weights": [ + 0.0797709971666336, + 0.1222120076417923, + 0.0859479010105133, + 0.09724918007850647, + 0.07649146020412445, + 0.07940177619457245, + 0.20231372117996216, + 0.15381231904029846, + 0.1028006300330162 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1370532512664795, + "mean_weights": [ + 0.0996367484331131, + 0.13669438660144806, + 0.08467932790517807, + 0.15324927866458893, + 0.11039721965789795, + 0.10590086132287979, + 0.10783202946186066, + 0.09274056553840637, + 0.10886958241462708 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0651674270629883, + "mean_weights": [ + 0.08736689388751984, + 0.10770082473754883, + 0.08818213641643524, + 0.09740092605352402, + 0.08032247424125671, + 0.07209183275699615, + 0.14879563450813293, + 0.13829395174980164, + 0.17984531819820404 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06189696863293648, + 0.057800546288490295, + 0.06563128530979156, + 0.09368102252483368, + 0.15762142837047577, + 0.14606256783008575, + 0.1579342633485794, + 0.11808711290359497, + 0.14670249819755554, + 0.08914493024349213, + 0.09681173413991928, + 0.08896270394325256, + 0.09372898936271667, + 0.0935099646449089, + 0.0877278745174408, + 0.09024087339639664, + 0.09534886479377747, + 0.056276921182870865, + 0.06960605084896088, + 0.05864190682768822, + 0.06732676178216934, + 0.0639718547463417, + 0.06809315830469131, + 0.06017342209815979, + 0.07299839705228806, + 0.04610355570912361, + 0.05418815836310387, + 0.052105579525232315, + 0.055226605385541916, + 0.049103740602731705, + 0.05673142150044441, + 0.05152653902769089, + 0.0670500174164772, + 0.04761137068271637, + 0.054591380059719086, + 0.04658065736293793, + 0.059859346598386765, + 0.049645137041807175, + 0.06647583842277527, + 0.05658312514424324, + 0.06586667895317078, + 0.03494055196642876, + 0.050202250480651855, + 0.041106488555669785, + 0.05492720007896423, + 0.05137007683515549, + 0.05579669028520584, + 0.06046925112605095, + 0.06042403727769852, + 0.044273097068071365, + 0.04726185277104378, + 0.04548347741365433, + 0.05116242170333862, + 0.04755190759897232, + 0.056738123297691345, + 0.04970773309469223, + 0.05916130542755127, + 0.05979602038860321, + 0.051449015736579895, + 0.06289176642894745, + 0.05728663131594658, + 0.06691838055849075, + 0.05687704309821129, + 0.0696212500333786 + ], + "loss_nats": 1.3019355535507202, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.718531847000122, + "mean_weights": [ + 0.1014469787478447, + 0.07179208844900131, + 0.05965828523039818, + 0.048375453799963, + 0.04613318294286728, + 0.05054276064038277, + 0.061866357922554016, + 0.12668980658054352, + 0.43349504470825195 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.06572774797677994, + 0.11052477359771729, + 0.2562657296657562, + 0.26237499713897705, + 0.27904239296913147, + 0.2805008888244629, + 0.28450125455856323, + 0.2898934483528137, + 0.10768114775419235, + 0.10940494388341904, + 0.11819783598184586, + 0.11956621706485748, + 0.12732882797718048, + 0.1299726366996765, + 0.14650858938694, + 0.1622796654701233, + 0.03554074838757515, + 0.04207194223999977, + 0.05237022787332535, + 0.06267905235290527, + 0.07247184216976166, + 0.08533640205860138, + 0.10017570108175278, + 0.11859145015478134, + 0.03065936081111431, + 0.046815622597932816, + 0.05710037797689438, + 0.07779109477996826, + 0.08306355774402618, + 0.10524991899728775, + 0.11067607998847961, + 0.13467493653297424, + 0.038578376173973083, + 0.05643659830093384, + 0.07001541554927826, + 0.08919516950845718, + 0.09562809765338898, + 0.11048520356416702, + 0.12341359257698059, + 0.13601729273796082, + 0.03717563673853874, + 0.05171631649136543, + 0.06705759465694427, + 0.0903087705373764, + 0.10195975005626678, + 0.13047200441360474, + 0.1420367807149887, + 0.18299677968025208, + 0.06643229722976685, + 0.09183725714683533, + 0.10768710821866989, + 0.13468314707279205, + 0.14544399082660675, + 0.17454643547534943, + 0.17629387974739075, + 0.2129679024219513, + 0.01394309289753437, + 0.061451755464076996, + 0.06262189894914627, + 0.1171356812119484, + 0.1185564175248146, + 0.17424017190933228, + 0.17667441070079803, + 0.2611335515975952 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 7.9557801867581395, + "cross_entropy_nats": 5.514526605606079, + "step": 0 + }, + { + "bits_per_byte": 5.165484730095761, + "cross_entropy_nats": 3.580441176891327, + "step": 100 + }, + { + "bits_per_byte": 3.11034227121925, + "cross_entropy_nats": 2.15592497587204, + "step": 500 + }, + { + "bits_per_byte": 1.9526028337920993, + "cross_entropy_nats": 1.3534411489963531, + "step": 2000 + }, + { + "bits_per_byte": 1.7970257369820821, + "cross_entropy_nats": 1.245603322982788, + "step": 4000 + }, + { + "bits_per_byte": 1.7147710141124801, + "cross_entropy_nats": 1.1885886937379837, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 49, + 51, + 53, + 55 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.49.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "49": true, + "51": true, + "53": true, + "55": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_group_7_mlp_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "fa654184abc600358c72cbc14817a8baa7fef683e79e7fea6dd011ec51df9c15", + "final_model_state": "6c4d43757199967d30eb41027bfa26f89fdeb6ea181c801f3fb6eff297d8323d", + "final_optimizer_state": "f4230ebb568b35a562bf28869c8c219216afb42f74a39b9dc0e14bc62197e1e6", + "final_public_parameters": "d8d95bdd0c7f591062a8aaca2180702c6546ce493bbd20170e512fb6a0410a5a", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "73bbe569e1a47981386ebaf59ca891174746bf81897fe5403676ad6d50423c58" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "65136111a29a042e61a7909132560d95cd4bcf0f9b52f64d0fb2e57773856434", + "1": "d995676b4e7dec8f661cd8c2345fe7fc7a513c17f528c02fc946a441a6995a94", + "7999": "2345e7ac3decca2bdaebf13094fdc92fcabef42e3e461a2fefd6e8e7e76baccc" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "d37de1eb6551388d135651860f70c8fb415042f78532eb2d75bce3deba3d00b3", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073001, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 188.10089484973457, + "measured_steps": 7980, + "median_ms": 192.13708851020783, + "p95_ms": 205.6841156896553, + "peak_allocated_bytes": 12208699392, + "peak_reserved_bytes": 12712935424, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 7.958777844362678, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.516604423522949, + "step": 1, + "unclipped_grad_norm": 21.323143005371094 + }, + { + "bits_per_byte": 7.247847657046258, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.023825168609619, + "step": 10, + "unclipped_grad_norm": 11.153545379638672 + }, + { + "bits_per_byte": 6.733688324026805, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.6674370765686035, + "step": 20, + "unclipped_grad_norm": 3.4891698360443115 + }, + { + "bits_per_byte": 6.611945933064644, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.583051681518555, + "step": 30, + "unclipped_grad_norm": 3.0754928588867188 + }, + { + "bits_per_byte": 6.391857670963698, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.430498123168945, + "step": 40, + "unclipped_grad_norm": 3.231210231781006 + }, + { + "bits_per_byte": 6.094212112629658, + "learning_rate": 3.75e-05, + "loss_nats": 4.224185943603516, + "step": 50, + "unclipped_grad_norm": 2.8722939491271973 + }, + { + "bits_per_byte": 5.886082836139108, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.079921722412109, + "step": 60, + "unclipped_grad_norm": 2.6704916954040527 + }, + { + "bits_per_byte": 5.7157759842263065, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.961874008178711, + "step": 70, + "unclipped_grad_norm": 3.1779234409332275 + }, + { + "bits_per_byte": 5.566113929723077, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.8581361770629883, + "step": 80, + "unclipped_grad_norm": 2.558431386947632 + }, + { + "bits_per_byte": 5.355095344698031, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.711869239807129, + "step": 90, + "unclipped_grad_norm": 2.4269511699676514 + }, + { + "bits_per_byte": 5.181231203634784, + "learning_rate": 7.5e-05, + "loss_nats": 3.591355800628662, + "step": 100, + "unclipped_grad_norm": 2.278672933578491 + }, + { + "bits_per_byte": 4.992411107287205, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.4604756832122803, + "step": 110, + "unclipped_grad_norm": 2.103325128555298 + }, + { + "bits_per_byte": 4.776015312700283, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.310481548309326, + "step": 120, + "unclipped_grad_norm": 1.8947192430496216 + }, + { + "bits_per_byte": 4.5822957745949235, + "learning_rate": 9.75e-05, + "loss_nats": 3.1762053966522217, + "step": 130, + "unclipped_grad_norm": 1.6946765184402466 + }, + { + "bits_per_byte": 4.428558075231547, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.0696425437927246, + "step": 140, + "unclipped_grad_norm": 1.4522587060928345 + }, + { + "bits_per_byte": 4.271807927755657, + "learning_rate": 0.0001125, + "loss_nats": 2.960991621017456, + "step": 150, + "unclipped_grad_norm": 1.2268821001052856 + }, + { + "bits_per_byte": 4.167444383663182, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.8886523246765137, + "step": 160, + "unclipped_grad_norm": 1.0625066757202148 + }, + { + "bits_per_byte": 4.027399598996201, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.7915806770324707, + "step": 170, + "unclipped_grad_norm": 0.9791240692138672 + }, + { + "bits_per_byte": 3.948645303513103, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.736992359161377, + "step": 180, + "unclipped_grad_norm": 0.8488200306892395 + }, + { + "bits_per_byte": 3.839429440909185, + "learning_rate": 0.0001425, + "loss_nats": 2.661289691925049, + "step": 190, + "unclipped_grad_norm": 0.8027761578559875 + }, + { + "bits_per_byte": 3.780712500112812, + "learning_rate": 0.00015, + "loss_nats": 2.6205902099609375, + "step": 200, + "unclipped_grad_norm": 1.088740587234497 + }, + { + "bits_per_byte": 3.7342476033639054, + "learning_rate": 0.0001575, + "loss_nats": 2.588383197784424, + "step": 210, + "unclipped_grad_norm": 0.8067484498023987 + }, + { + "bits_per_byte": 3.632546694672089, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.5178894996643066, + "step": 220, + "unclipped_grad_norm": 0.7257278561592102 + }, + { + "bits_per_byte": 3.622913602431961, + "learning_rate": 0.0001725, + "loss_nats": 2.5112123489379883, + "step": 230, + "unclipped_grad_norm": 1.0309072732925415 + }, + { + "bits_per_byte": 3.5025432890583827, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.4277780055999756, + "step": 240, + "unclipped_grad_norm": 0.7052561044692993 + }, + { + "bits_per_byte": 3.5399536432046643, + "learning_rate": 0.0001875, + "loss_nats": 2.4537088871002197, + "step": 250, + "unclipped_grad_norm": 1.2091069221496582 + }, + { + "bits_per_byte": 3.534890817928565, + "learning_rate": 0.000195, + "loss_nats": 2.450199604034424, + "step": 260, + "unclipped_grad_norm": 0.9051746129989624 + }, + { + "bits_per_byte": 3.5328793088440738, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.448805332183838, + "step": 270, + "unclipped_grad_norm": 0.9858413934707642 + }, + { + "bits_per_byte": 3.4679641133106047, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.4038095474243164, + "step": 280, + "unclipped_grad_norm": 1.116882085800171 + }, + { + "bits_per_byte": 3.3828901433389027, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.3448407649993896, + "step": 290, + "unclipped_grad_norm": 1.141707420349121 + }, + { + "bits_per_byte": 3.407375313342241, + "learning_rate": 0.000225, + "loss_nats": 2.3618125915527344, + "step": 300, + "unclipped_grad_norm": 1.1243934631347656 + }, + { + "bits_per_byte": 3.356115540287279, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.326282024383545, + "step": 310, + "unclipped_grad_norm": 0.8094613552093506 + }, + { + "bits_per_byte": 3.3365989490673673, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.3127541542053223, + "step": 320, + "unclipped_grad_norm": 0.8834619522094727 + }, + { + "bits_per_byte": 3.3354218998048624, + "learning_rate": 0.0002475, + "loss_nats": 2.3119382858276367, + "step": 330, + "unclipped_grad_norm": 0.9597899317741394 + }, + { + "bits_per_byte": 3.4171411761888346, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.368581771850586, + "step": 340, + "unclipped_grad_norm": 1.1168477535247803 + }, + { + "bits_per_byte": 3.339147044022551, + "learning_rate": 0.0002625, + "loss_nats": 2.3145203590393066, + "step": 350, + "unclipped_grad_norm": 0.8856794834136963 + }, + { + "bits_per_byte": 3.2903775797520827, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.2807159423828125, + "step": 360, + "unclipped_grad_norm": 1.1153349876403809 + }, + { + "bits_per_byte": 3.284932609025766, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.2769417762756348, + "step": 370, + "unclipped_grad_norm": 0.8861335515975952 + }, + { + "bits_per_byte": 3.3366491680014194, + "learning_rate": 0.000285, + "loss_nats": 2.312788963317871, + "step": 380, + "unclipped_grad_norm": 1.1883227825164795 + }, + { + "bits_per_byte": 3.2632170476319224, + "learning_rate": 0.0002925, + "loss_nats": 2.261889696121216, + "step": 390, + "unclipped_grad_norm": 1.3043349981307983 + }, + { + "bits_per_byte": 3.2061074566905132, + "learning_rate": 0.0003, + "loss_nats": 2.222304344177246, + "step": 400, + "unclipped_grad_norm": 1.0857030153274536 + }, + { + "bits_per_byte": 3.1956760209850805, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.215073823928833, + "step": 410, + "unclipped_grad_norm": 1.0887941122055054 + }, + { + "bits_per_byte": 3.2259852114772367, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.2360825538635254, + "step": 420, + "unclipped_grad_norm": 1.0030070543289185 + }, + { + "bits_per_byte": 3.211075691508638, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.225748062133789, + "step": 430, + "unclipped_grad_norm": 1.3803348541259766 + }, + { + "bits_per_byte": 3.2379761858606932, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.244394063949585, + "step": 440, + "unclipped_grad_norm": 1.1100904941558838 + }, + { + "bits_per_byte": 3.1514530900728213, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.1844208240509033, + "step": 450, + "unclipped_grad_norm": 0.9338343739509583 + }, + { + "bits_per_byte": 3.1810839809899747, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.2049593925476074, + "step": 460, + "unclipped_grad_norm": 1.06135892868042 + }, + { + "bits_per_byte": 3.1537552498372707, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.18601655960083, + "step": 470, + "unclipped_grad_norm": 1.30195951461792 + }, + { + "bits_per_byte": 3.1425044887831275, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.178218126296997, + "step": 480, + "unclipped_grad_norm": 0.8835803866386414 + }, + { + "bits_per_byte": 3.0803637173733773, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.135145425796509, + "step": 490, + "unclipped_grad_norm": 1.013891577720642 + }, + { + "bits_per_byte": 3.045876724326462, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.111240863800049, + "step": 500, + "unclipped_grad_norm": 1.4184454679489136 + }, + { + "bits_per_byte": 3.085149306616286, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.138462543487549, + "step": 510, + "unclipped_grad_norm": 1.1447899341583252 + }, + { + "bits_per_byte": 2.994997032929907, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.0759737491607666, + "step": 520, + "unclipped_grad_norm": 1.2133339643478394 + }, + { + "bits_per_byte": 3.0813756632910545, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.1358468532562256, + "step": 530, + "unclipped_grad_norm": 1.1639283895492554 + }, + { + "bits_per_byte": 3.0182081554556532, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.092062473297119, + "step": 540, + "unclipped_grad_norm": 1.1934679746627808 + }, + { + "bits_per_byte": 2.9824199416723096, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.067255973815918, + "step": 550, + "unclipped_grad_norm": 1.076833724975586 + }, + { + "bits_per_byte": 2.972878344202441, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.0606422424316406, + "step": 560, + "unclipped_grad_norm": 0.9363606572151184 + }, + { + "bits_per_byte": 2.9325807453134747, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.032710075378418, + "step": 570, + "unclipped_grad_norm": 1.1381862163543701 + }, + { + "bits_per_byte": 2.98635868834237, + "learning_rate": 0.000299626473885547, + "loss_nats": 2.06998610496521, + "step": 580, + "unclipped_grad_norm": 0.9459874629974365 + }, + { + "bits_per_byte": 2.904629092999346, + "learning_rate": 0.00029958384005397226, + "loss_nats": 2.0133354663848877, + "step": 590, + "unclipped_grad_norm": 1.0313589572906494 + }, + { + "bits_per_byte": 2.9477090272139708, + "learning_rate": 0.0002995389065559004, + "loss_nats": 2.043196201324463, + "step": 600, + "unclipped_grad_norm": 0.888031542301178 + }, + { + "bits_per_byte": 2.8592906826135907, + "learning_rate": 0.000299491674159121, + "loss_nats": 1.9819092750549316, + "step": 610, + "unclipped_grad_norm": 0.8866042494773865 + }, + { + "bits_per_byte": 2.9142917662554217, + "learning_rate": 0.00029944214367070535, + "loss_nats": 2.020033121109009, + "step": 620, + "unclipped_grad_norm": 0.9619577527046204 + }, + { + "bits_per_byte": 2.847792266542205, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.9739391803741455, + "step": 630, + "unclipped_grad_norm": 1.1491894721984863 + }, + { + "bits_per_byte": 2.8079679639084127, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.9463350772857666, + "step": 640, + "unclipped_grad_norm": 1.1180716753005981 + }, + { + "bits_per_byte": 2.744830209071611, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.9025713205337524, + "step": 650, + "unclipped_grad_norm": 1.1899323463439941 + }, + { + "bits_per_byte": 2.6832347946477473, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.8598766326904297, + "step": 660, + "unclipped_grad_norm": 1.4504836797714233 + }, + { + "bits_per_byte": 2.7167132814300032, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.8830821514129639, + "step": 670, + "unclipped_grad_norm": 1.0202070474624634 + }, + { + "bits_per_byte": 2.6501164395363976, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.8369207382202148, + "step": 680, + "unclipped_grad_norm": 1.1102021932601929 + }, + { + "bits_per_byte": 2.7368354235636025, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.8970297574996948, + "step": 690, + "unclipped_grad_norm": 1.1015986204147339 + }, + { + "bits_per_byte": 2.63366629898137, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.8255183696746826, + "step": 700, + "unclipped_grad_norm": 0.9084121584892273 + }, + { + "bits_per_byte": 2.652852855494276, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.8388174772262573, + "step": 710, + "unclipped_grad_norm": 0.930149495601654 + }, + { + "bits_per_byte": 2.6655554940870028, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.847622275352478, + "step": 720, + "unclipped_grad_norm": 0.9548717141151428 + }, + { + "bits_per_byte": 2.61783357614294, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.8145439624786377, + "step": 730, + "unclipped_grad_norm": 1.0372314453125 + }, + { + "bits_per_byte": 2.5685440363362626, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.780379056930542, + "step": 740, + "unclipped_grad_norm": 1.0298105478286743 + }, + { + "bits_per_byte": 2.5352863752777144, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.757326602935791, + "step": 750, + "unclipped_grad_norm": 0.8861494660377502 + }, + { + "bits_per_byte": 2.483729760167363, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.721590280532837, + "step": 760, + "unclipped_grad_norm": 1.2945467233657837 + }, + { + "bits_per_byte": 2.6249443708255136, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.8194727897644043, + "step": 770, + "unclipped_grad_norm": 1.1100448369979858 + }, + { + "bits_per_byte": 2.5088113660038935, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.7389755249023438, + "step": 780, + "unclipped_grad_norm": 0.9117511510848999 + }, + { + "bits_per_byte": 2.4644654674962676, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.7082372903823853, + "step": 790, + "unclipped_grad_norm": 0.9106786251068115 + }, + { + "bits_per_byte": 2.4170245692037144, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.675353765487671, + "step": 800, + "unclipped_grad_norm": 1.1967017650604248 + }, + { + "bits_per_byte": 2.4787964378273637, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7181707620620728, + "step": 810, + "unclipped_grad_norm": 0.9779432415962219 + }, + { + "bits_per_byte": 2.410801548964894, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.6710402965545654, + "step": 820, + "unclipped_grad_norm": 1.0304938554763794 + }, + { + "bits_per_byte": 2.4708109393825017, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.7126356363296509, + "step": 830, + "unclipped_grad_norm": 1.0873920917510986 + }, + { + "bits_per_byte": 2.544278488178076, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.7635594606399536, + "step": 840, + "unclipped_grad_norm": 1.2037078142166138 + }, + { + "bits_per_byte": 2.430745689054851, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.6848645210266113, + "step": 850, + "unclipped_grad_norm": 0.9387081861495972 + }, + { + "bits_per_byte": 2.4231894593065424, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.6796269416809082, + "step": 860, + "unclipped_grad_norm": 1.0327205657958984 + }, + { + "bits_per_byte": 2.348273988573354, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.6276994943618774, + "step": 870, + "unclipped_grad_norm": 1.2000164985656738 + }, + { + "bits_per_byte": 2.3618205460338646, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.6370892524719238, + "step": 880, + "unclipped_grad_norm": 0.9521110653877258 + }, + { + "bits_per_byte": 2.298499629673894, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.593198537826538, + "step": 890, + "unclipped_grad_norm": 0.8883240222930908 + }, + { + "bits_per_byte": 2.325950984456663, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.6122263669967651, + "step": 900, + "unclipped_grad_norm": 1.2442853450775146 + }, + { + "bits_per_byte": 2.335013094295926, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.618507742881775, + "step": 910, + "unclipped_grad_norm": 0.8801536560058594 + }, + { + "bits_per_byte": 2.319716785345537, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.6079051494598389, + "step": 920, + "unclipped_grad_norm": 0.9751235246658325 + }, + { + "bits_per_byte": 2.3313837444147683, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.6159920692443848, + "step": 930, + "unclipped_grad_norm": 0.9184286594390869 + }, + { + "bits_per_byte": 2.3342377965058367, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.61797034740448, + "step": 940, + "unclipped_grad_norm": 0.9324998259544373 + }, + { + "bits_per_byte": 2.3099036272699562, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.6011031866073608, + "step": 950, + "unclipped_grad_norm": 0.8854913115501404 + }, + { + "bits_per_byte": 2.353852245854089, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.631566047668457, + "step": 960, + "unclipped_grad_norm": 0.945914626121521 + }, + { + "bits_per_byte": 2.259015508723509, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5658302307128906, + "step": 970, + "unclipped_grad_norm": 0.9183659553527832 + }, + { + "bits_per_byte": 2.33575932701802, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.6190249919891357, + "step": 980, + "unclipped_grad_norm": 0.8496978878974915 + }, + { + "bits_per_byte": 2.3026626417206795, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.5960841178894043, + "step": 990, + "unclipped_grad_norm": 1.0162022113800049 + }, + { + "bits_per_byte": 2.2601736398944188, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.5666329860687256, + "step": 1000, + "unclipped_grad_norm": 0.8914238810539246 + }, + { + "bits_per_byte": 2.249526709927455, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.5592530965805054, + "step": 1010, + "unclipped_grad_norm": 1.126776933670044 + }, + { + "bits_per_byte": 2.206921619811862, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.5297214984893799, + "step": 1020, + "unclipped_grad_norm": 0.8951566219329834 + }, + { + "bits_per_byte": 2.221434031839619, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.5397807359695435, + "step": 1030, + "unclipped_grad_norm": 1.0575919151306152 + }, + { + "bits_per_byte": 2.25792341889053, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.5650732517242432, + "step": 1040, + "unclipped_grad_norm": 0.9248005151748657 + }, + { + "bits_per_byte": 2.2513915178107604, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5605456829071045, + "step": 1050, + "unclipped_grad_norm": 0.9051158428192139 + }, + { + "bits_per_byte": 2.161204503628907, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.498032808303833, + "step": 1060, + "unclipped_grad_norm": 1.0043655633926392 + }, + { + "bits_per_byte": 2.2397568914798915, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.5524811744689941, + "step": 1070, + "unclipped_grad_norm": 0.9103208184242249 + }, + { + "bits_per_byte": 2.219961860348233, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.5387603044509888, + "step": 1080, + "unclipped_grad_norm": 1.0210990905761719 + }, + { + "bits_per_byte": 2.184187921124911, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.5139636993408203, + "step": 1090, + "unclipped_grad_norm": 0.9395274519920349 + }, + { + "bits_per_byte": 2.2472773488368203, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5576939582824707, + "step": 1100, + "unclipped_grad_norm": 0.8826550841331482 + }, + { + "bits_per_byte": 2.1246053916467456, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.4726642370224, + "step": 1110, + "unclipped_grad_norm": 0.9168148040771484 + }, + { + "bits_per_byte": 2.1635845715341975, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.4996825456619263, + "step": 1120, + "unclipped_grad_norm": 0.9577497243881226 + }, + { + "bits_per_byte": 2.0980280526956614, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.45424222946167, + "step": 1130, + "unclipped_grad_norm": 0.8399538993835449 + }, + { + "bits_per_byte": 2.2304591654089463, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.5460364818572998, + "step": 1140, + "unclipped_grad_norm": 0.9069536924362183 + }, + { + "bits_per_byte": 2.0914139439087793, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.449657678604126, + "step": 1150, + "unclipped_grad_norm": 0.983115553855896 + }, + { + "bits_per_byte": 2.1185865148144996, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.4684922695159912, + "step": 1160, + "unclipped_grad_norm": 0.8957574963569641 + }, + { + "bits_per_byte": 2.166541297267831, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.5017319917678833, + "step": 1170, + "unclipped_grad_norm": 0.8536692261695862 + }, + { + "bits_per_byte": 2.1418272183561946, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.4846014976501465, + "step": 1180, + "unclipped_grad_norm": 0.9590627551078796 + }, + { + "bits_per_byte": 2.07152105464878, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.4358689785003662, + "step": 1190, + "unclipped_grad_norm": 0.9030612111091614 + }, + { + "bits_per_byte": 2.045056020368709, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.417524814605713, + "step": 1200, + "unclipped_grad_norm": 1.0136886835098267 + }, + { + "bits_per_byte": 2.106827373044406, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.460341453552246, + "step": 1210, + "unclipped_grad_norm": 0.918593168258667 + }, + { + "bits_per_byte": 2.1170501937943422, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.467427372932434, + "step": 1220, + "unclipped_grad_norm": 0.842893660068512 + }, + { + "bits_per_byte": 2.078365448205167, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.4406131505966187, + "step": 1230, + "unclipped_grad_norm": 0.8629698157310486 + }, + { + "bits_per_byte": 2.0307797405205874, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.4076292514801025, + "step": 1240, + "unclipped_grad_norm": 1.0607963800430298 + }, + { + "bits_per_byte": 2.1121010490504633, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.4639968872070312, + "step": 1250, + "unclipped_grad_norm": 0.8639792203903198 + }, + { + "bits_per_byte": 2.0837644995637032, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.4443554878234863, + "step": 1260, + "unclipped_grad_norm": 0.8044229745864868 + }, + { + "bits_per_byte": 2.081569828956043, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.4428342580795288, + "step": 1270, + "unclipped_grad_norm": 0.8750306367874146 + }, + { + "bits_per_byte": 1.9746191498466392, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.368701696395874, + "step": 1280, + "unclipped_grad_norm": 0.8585606813430786 + }, + { + "bits_per_byte": 2.0945727492571766, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4518471956253052, + "step": 1290, + "unclipped_grad_norm": 0.9211471676826477 + }, + { + "bits_per_byte": 2.0360310587820325, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.411269187927246, + "step": 1300, + "unclipped_grad_norm": 0.8559306859970093 + }, + { + "bits_per_byte": 2.0431078008997354, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.4161744117736816, + "step": 1310, + "unclipped_grad_norm": 0.8761854767799377 + }, + { + "bits_per_byte": 2.0207526360273333, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.4006789922714233, + "step": 1320, + "unclipped_grad_norm": 0.8598175048828125 + }, + { + "bits_per_byte": 2.09540858494037, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.452426552772522, + "step": 1330, + "unclipped_grad_norm": 0.8781486749649048 + }, + { + "bits_per_byte": 2.0464659341404827, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.4185020923614502, + "step": 1340, + "unclipped_grad_norm": 0.8584818243980408 + }, + { + "bits_per_byte": 2.0464872599891897, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.4185168743133545, + "step": 1350, + "unclipped_grad_norm": 0.7844499349594116 + }, + { + "bits_per_byte": 2.0885683189676034, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.4476852416992188, + "step": 1360, + "unclipped_grad_norm": 0.8635284900665283 + }, + { + "bits_per_byte": 2.085897428399704, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4458339214324951, + "step": 1370, + "unclipped_grad_norm": 0.8268426656723022 + }, + { + "bits_per_byte": 2.039394695467607, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.4136006832122803, + "step": 1380, + "unclipped_grad_norm": 0.8343918919563293 + }, + { + "bits_per_byte": 2.0509312916874847, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.4215972423553467, + "step": 1390, + "unclipped_grad_norm": 0.8060122132301331 + }, + { + "bits_per_byte": 1.9702745241205428, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.3656902313232422, + "step": 1400, + "unclipped_grad_norm": 0.8047821521759033 + }, + { + "bits_per_byte": 1.9978851348380107, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.3848284482955933, + "step": 1410, + "unclipped_grad_norm": 0.799464762210846 + }, + { + "bits_per_byte": 2.0492087134564425, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.420403242111206, + "step": 1420, + "unclipped_grad_norm": 0.8452145457267761 + }, + { + "bits_per_byte": 2.064922080335174, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.4312949180603027, + "step": 1430, + "unclipped_grad_norm": 0.8104668855667114 + }, + { + "bits_per_byte": 2.0261362089472916, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.4044106006622314, + "step": 1440, + "unclipped_grad_norm": 0.8697406649589539 + }, + { + "bits_per_byte": 1.965508196934531, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.3623864650726318, + "step": 1450, + "unclipped_grad_norm": 0.7486980557441711 + }, + { + "bits_per_byte": 2.008056360792688, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.391878604888916, + "step": 1460, + "unclipped_grad_norm": 0.8696911931037903 + }, + { + "bits_per_byte": 1.94054801084951, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.3450853824615479, + "step": 1470, + "unclipped_grad_norm": 0.9338918924331665 + }, + { + "bits_per_byte": 2.001439844248694, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.3872923851013184, + "step": 1480, + "unclipped_grad_norm": 0.9493666291236877 + }, + { + "bits_per_byte": 2.029890762198278, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.4070130586624146, + "step": 1490, + "unclipped_grad_norm": 0.8383588790893555 + }, + { + "bits_per_byte": 2.0313993940116464, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.408058762550354, + "step": 1500, + "unclipped_grad_norm": 0.9135459065437317 + }, + { + "bits_per_byte": 2.0287118211266133, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.406195878982544, + "step": 1510, + "unclipped_grad_norm": 0.8665618896484375 + }, + { + "bits_per_byte": 1.9782192626771502, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.3711971044540405, + "step": 1520, + "unclipped_grad_norm": 0.7863375544548035 + }, + { + "bits_per_byte": 2.0515535249183063, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.4220285415649414, + "step": 1530, + "unclipped_grad_norm": 1.0084644556045532 + }, + { + "bits_per_byte": 1.999410448968513, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3858857154846191, + "step": 1540, + "unclipped_grad_norm": 0.829677402973175 + }, + { + "bits_per_byte": 2.0515982404075306, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.4220595359802246, + "step": 1550, + "unclipped_grad_norm": 0.820928692817688 + }, + { + "bits_per_byte": 1.969710077060411, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.3652989864349365, + "step": 1560, + "unclipped_grad_norm": 0.8289248943328857 + }, + { + "bits_per_byte": 2.0410830491511276, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.4147709608078003, + "step": 1570, + "unclipped_grad_norm": 0.8322929739952087 + }, + { + "bits_per_byte": 1.9838888427454937, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.3751269578933716, + "step": 1580, + "unclipped_grad_norm": 0.7805985808372498 + }, + { + "bits_per_byte": 2.002978401043313, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3883588314056396, + "step": 1590, + "unclipped_grad_norm": 0.8161291480064392 + }, + { + "bits_per_byte": 2.0205952719017937, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.4005699157714844, + "step": 1600, + "unclipped_grad_norm": 0.8045579195022583 + }, + { + "bits_per_byte": 1.9539882830317874, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.3544014692306519, + "step": 1610, + "unclipped_grad_norm": 0.7475293874740601 + }, + { + "bits_per_byte": 1.907571885390424, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.3222280740737915, + "step": 1620, + "unclipped_grad_norm": 0.7688040733337402 + }, + { + "bits_per_byte": 1.936934827337533, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.3425809144973755, + "step": 1630, + "unclipped_grad_norm": 0.8011300563812256 + }, + { + "bits_per_byte": 1.9866218190503546, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.377021312713623, + "step": 1640, + "unclipped_grad_norm": 0.7804743647575378 + }, + { + "bits_per_byte": 2.0408878488423983, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.4146356582641602, + "step": 1650, + "unclipped_grad_norm": 0.9009407758712769 + }, + { + "bits_per_byte": 1.9834102150281425, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.3747951984405518, + "step": 1660, + "unclipped_grad_norm": 0.8439423441886902 + }, + { + "bits_per_byte": 2.001540282116798, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.387362003326416, + "step": 1670, + "unclipped_grad_norm": 0.785163164138794 + }, + { + "bits_per_byte": 2.021726573779169, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.4013540744781494, + "step": 1680, + "unclipped_grad_norm": 0.878528356552124 + }, + { + "bits_per_byte": 2.049274754794374, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.4204490184783936, + "step": 1690, + "unclipped_grad_norm": 0.7973967790603638 + }, + { + "bits_per_byte": 1.9461947172252885, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.3489993810653687, + "step": 1700, + "unclipped_grad_norm": 0.7178767919540405 + }, + { + "bits_per_byte": 1.8749027489631283, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.2995835542678833, + "step": 1710, + "unclipped_grad_norm": 0.6961382031440735 + }, + { + "bits_per_byte": 1.888514659830967, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.309018611907959, + "step": 1720, + "unclipped_grad_norm": 0.7466060519218445 + }, + { + "bits_per_byte": 1.8629707646289175, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.2913129329681396, + "step": 1730, + "unclipped_grad_norm": 0.7574632167816162 + }, + { + "bits_per_byte": 1.9761370687231543, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.3697538375854492, + "step": 1740, + "unclipped_grad_norm": 0.777412474155426 + }, + { + "bits_per_byte": 1.9985331654664618, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.3852776288986206, + "step": 1750, + "unclipped_grad_norm": 0.7746729254722595 + }, + { + "bits_per_byte": 2.003487125724565, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.3887114524841309, + "step": 1760, + "unclipped_grad_norm": 0.8282725811004639 + }, + { + "bits_per_byte": 2.005511533507871, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.390114665031433, + "step": 1770, + "unclipped_grad_norm": 0.8189800977706909 + }, + { + "bits_per_byte": 1.8594673060481912, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.2888845205307007, + "step": 1780, + "unclipped_grad_norm": 0.7453064322471619 + }, + { + "bits_per_byte": 1.9473980798333754, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3498334884643555, + "step": 1790, + "unclipped_grad_norm": 0.717052698135376 + }, + { + "bits_per_byte": 1.884268752146466, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.3060755729675293, + "step": 1800, + "unclipped_grad_norm": 0.7678354978561401 + }, + { + "bits_per_byte": 1.9335232074923676, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.3402161598205566, + "step": 1810, + "unclipped_grad_norm": 0.7371082305908203 + }, + { + "bits_per_byte": 1.9131914185073664, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.3261232376098633, + "step": 1820, + "unclipped_grad_norm": 0.778931200504303 + }, + { + "bits_per_byte": 1.8850363107172667, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.3066076040267944, + "step": 1830, + "unclipped_grad_norm": 0.7472635507583618 + }, + { + "bits_per_byte": 1.8847960509540114, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.306441068649292, + "step": 1840, + "unclipped_grad_norm": 0.7614195346832275 + }, + { + "bits_per_byte": 1.906278403873285, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.32133150100708, + "step": 1850, + "unclipped_grad_norm": 0.7674909830093384 + }, + { + "bits_per_byte": 1.9001197051458876, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.3170626163482666, + "step": 1860, + "unclipped_grad_norm": 0.8020868301391602 + }, + { + "bits_per_byte": 1.8838683765352575, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.305798053741455, + "step": 1870, + "unclipped_grad_norm": 0.7525386214256287 + }, + { + "bits_per_byte": 1.8840861065712498, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.3059489727020264, + "step": 1880, + "unclipped_grad_norm": 0.712367832660675 + }, + { + "bits_per_byte": 1.9043379236235998, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3199864625930786, + "step": 1890, + "unclipped_grad_norm": 0.7926298379898071 + }, + { + "bits_per_byte": 1.9110392276144692, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.3246314525604248, + "step": 1900, + "unclipped_grad_norm": 0.7525349855422974 + }, + { + "bits_per_byte": 1.853286249576182, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.2846001386642456, + "step": 1910, + "unclipped_grad_norm": 0.7612677216529846 + }, + { + "bits_per_byte": 1.86284366944993, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.2912248373031616, + "step": 1920, + "unclipped_grad_norm": 0.7395919561386108 + }, + { + "bits_per_byte": 1.9429325503037227, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.3467382192611694, + "step": 1930, + "unclipped_grad_norm": 0.7171694040298462 + }, + { + "bits_per_byte": 1.8150103067481078, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.2580692768096924, + "step": 1940, + "unclipped_grad_norm": 0.7705104351043701 + }, + { + "bits_per_byte": 1.899291952647285, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.3164888620376587, + "step": 1950, + "unclipped_grad_norm": 0.7154866456985474 + }, + { + "bits_per_byte": 1.8214885492408082, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.2625596523284912, + "step": 1960, + "unclipped_grad_norm": 0.727989912033081 + }, + { + "bits_per_byte": 1.8986184685865062, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.3160220384597778, + "step": 1970, + "unclipped_grad_norm": 0.7109861373901367 + }, + { + "bits_per_byte": 1.944111835340689, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.3475556373596191, + "step": 1980, + "unclipped_grad_norm": 0.7743441462516785 + }, + { + "bits_per_byte": 1.9446501410378896, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.347928762435913, + "step": 1990, + "unclipped_grad_norm": 0.8268585205078125 + }, + { + "bits_per_byte": 1.9180298064240897, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.3294769525527954, + "step": 2000, + "unclipped_grad_norm": 0.8736605048179626 + }, + { + "bits_per_byte": 1.8851704571849397, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.306700587272644, + "step": 2010, + "unclipped_grad_norm": 0.7204143404960632 + }, + { + "bits_per_byte": 1.9064230412826604, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.3214317560195923, + "step": 2020, + "unclipped_grad_norm": 0.7198081016540527 + }, + { + "bits_per_byte": 1.9686688940920876, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.364577293395996, + "step": 2030, + "unclipped_grad_norm": 0.7342803478240967 + }, + { + "bits_per_byte": 1.901914516090292, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.3183066844940186, + "step": 2040, + "unclipped_grad_norm": 0.7853919863700867 + }, + { + "bits_per_byte": 1.9577235742258767, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.3569905757904053, + "step": 2050, + "unclipped_grad_norm": 0.7579192519187927 + }, + { + "bits_per_byte": 1.9172639676797976, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.3289461135864258, + "step": 2060, + "unclipped_grad_norm": 0.7506533265113831 + }, + { + "bits_per_byte": 1.871839222003309, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.2974600791931152, + "step": 2070, + "unclipped_grad_norm": 0.7474235892295837 + }, + { + "bits_per_byte": 1.8802275038164917, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.303274393081665, + "step": 2080, + "unclipped_grad_norm": 0.7812159061431885 + }, + { + "bits_per_byte": 1.8663983788606133, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.2936887741088867, + "step": 2090, + "unclipped_grad_norm": 0.7092841267585754 + }, + { + "bits_per_byte": 1.8633842109215917, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.2915995121002197, + "step": 2100, + "unclipped_grad_norm": 0.721014678478241 + }, + { + "bits_per_byte": 1.9053249320569015, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.3206706047058105, + "step": 2110, + "unclipped_grad_norm": 0.7848287224769592 + }, + { + "bits_per_byte": 1.851513624393739, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.2833714485168457, + "step": 2120, + "unclipped_grad_norm": 0.6954854726791382 + }, + { + "bits_per_byte": 1.8284772362412691, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.2674038410186768, + "step": 2130, + "unclipped_grad_norm": 0.7777084708213806 + }, + { + "bits_per_byte": 1.9647758948071572, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.3618788719177246, + "step": 2140, + "unclipped_grad_norm": 0.7766419649124146 + }, + { + "bits_per_byte": 1.821427323417101, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2625172138214111, + "step": 2150, + "unclipped_grad_norm": 0.7361568212509155 + }, + { + "bits_per_byte": 1.8668835419186973, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.2940250635147095, + "step": 2160, + "unclipped_grad_norm": 0.7416975498199463 + }, + { + "bits_per_byte": 1.950641844611299, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.3520818948745728, + "step": 2170, + "unclipped_grad_norm": 0.7165414094924927 + }, + { + "bits_per_byte": 1.809400404659614, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.2541807889938354, + "step": 2180, + "unclipped_grad_norm": 0.6813501715660095 + }, + { + "bits_per_byte": 1.8356222554713646, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.272356390953064, + "step": 2190, + "unclipped_grad_norm": 0.6741871237754822 + }, + { + "bits_per_byte": 1.8738968224382317, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.2988862991333008, + "step": 2200, + "unclipped_grad_norm": 0.7293112874031067 + }, + { + "bits_per_byte": 1.8312193276266258, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.2693045139312744, + "step": 2210, + "unclipped_grad_norm": 0.7025629281997681 + }, + { + "bits_per_byte": 1.8347312133572444, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2717387676239014, + "step": 2220, + "unclipped_grad_norm": 0.6917855143547058 + }, + { + "bits_per_byte": 1.905081060657978, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.3205015659332275, + "step": 2230, + "unclipped_grad_norm": 0.8129913210868835 + }, + { + "bits_per_byte": 1.8303934669371826, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.2687320709228516, + "step": 2240, + "unclipped_grad_norm": 0.7178182601928711 + }, + { + "bits_per_byte": 1.8533970064033376, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2846769094467163, + "step": 2250, + "unclipped_grad_norm": 0.7913292050361633 + }, + { + "bits_per_byte": 1.8357963019140378, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.2724770307540894, + "step": 2260, + "unclipped_grad_norm": 0.6933118104934692 + }, + { + "bits_per_byte": 1.8116917295170591, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.2557690143585205, + "step": 2270, + "unclipped_grad_norm": 0.7033843398094177 + }, + { + "bits_per_byte": 1.842161551805121, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.2768890857696533, + "step": 2280, + "unclipped_grad_norm": 0.7306208610534668 + }, + { + "bits_per_byte": 1.8574308594793247, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.2874729633331299, + "step": 2290, + "unclipped_grad_norm": 0.734154999256134 + }, + { + "bits_per_byte": 1.7915300313737612, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.2417939901351929, + "step": 2300, + "unclipped_grad_norm": 0.7330820560455322 + }, + { + "bits_per_byte": 1.8302302554015137, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.2686189413070679, + "step": 2310, + "unclipped_grad_norm": 0.7269130945205688 + }, + { + "bits_per_byte": 1.7933659461717217, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.2430665493011475, + "step": 2320, + "unclipped_grad_norm": 0.6893941164016724 + }, + { + "bits_per_byte": 1.8684828085890706, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.2951335906982422, + "step": 2330, + "unclipped_grad_norm": 0.7078773379325867 + }, + { + "bits_per_byte": 1.7973222350721696, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.2458088397979736, + "step": 2340, + "unclipped_grad_norm": 0.7075551748275757 + }, + { + "bits_per_byte": 1.8531663776685305, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2845170497894287, + "step": 2350, + "unclipped_grad_norm": 0.7025898098945618 + }, + { + "bits_per_byte": 1.8520264766586119, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.2837269306182861, + "step": 2360, + "unclipped_grad_norm": 0.7310325503349304 + }, + { + "bits_per_byte": 1.8732150832102117, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.2984137535095215, + "step": 2370, + "unclipped_grad_norm": 0.70386803150177 + }, + { + "bits_per_byte": 1.821263079985527, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.2624033689498901, + "step": 2380, + "unclipped_grad_norm": 0.7137768864631653 + }, + { + "bits_per_byte": 1.8898743546686885, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.3099610805511475, + "step": 2390, + "unclipped_grad_norm": 0.7540795207023621 + }, + { + "bits_per_byte": 1.833852725976637, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.271129846572876, + "step": 2400, + "unclipped_grad_norm": 0.70706707239151 + }, + { + "bits_per_byte": 1.82253540763661, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2632852792739868, + "step": 2410, + "unclipped_grad_norm": 0.7348983883857727 + }, + { + "bits_per_byte": 1.7905521380209553, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.2411161661148071, + "step": 2420, + "unclipped_grad_norm": 0.7241659164428711 + }, + { + "bits_per_byte": 1.8157608390364732, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.258589506149292, + "step": 2430, + "unclipped_grad_norm": 0.702774703502655 + }, + { + "bits_per_byte": 1.8805559906796396, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.303502082824707, + "step": 2440, + "unclipped_grad_norm": 0.7624387741088867 + }, + { + "bits_per_byte": 1.8173047272932685, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.2596596479415894, + "step": 2450, + "unclipped_grad_norm": 0.6831531524658203 + }, + { + "bits_per_byte": 1.813435633596808, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.2569777965545654, + "step": 2460, + "unclipped_grad_norm": 0.6722383499145508 + }, + { + "bits_per_byte": 1.804332075938687, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.250667691230774, + "step": 2470, + "unclipped_grad_norm": 0.7445098161697388 + }, + { + "bits_per_byte": 1.8386897380321536, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.2744826078414917, + "step": 2480, + "unclipped_grad_norm": 0.7045318484306335 + }, + { + "bits_per_byte": 1.8100850675926987, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.254655361175537, + "step": 2490, + "unclipped_grad_norm": 0.7058310508728027 + }, + { + "bits_per_byte": 1.8284113668859887, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2673581838607788, + "step": 2500, + "unclipped_grad_norm": 0.6749030947685242 + }, + { + "bits_per_byte": 1.7853604977393596, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.2375175952911377, + "step": 2510, + "unclipped_grad_norm": 0.7121540904045105 + }, + { + "bits_per_byte": 1.8378967260290253, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.273932933807373, + "step": 2520, + "unclipped_grad_norm": 0.708854615688324 + }, + { + "bits_per_byte": 1.843266024388962, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.2776546478271484, + "step": 2530, + "unclipped_grad_norm": 0.6758978366851807 + }, + { + "bits_per_byte": 1.8119417922914136, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.2559423446655273, + "step": 2540, + "unclipped_grad_norm": 0.7112060785293579 + }, + { + "bits_per_byte": 1.8093293758248075, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.254131555557251, + "step": 2550, + "unclipped_grad_norm": 0.7358750104904175 + }, + { + "bits_per_byte": 1.830769077046667, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.2689924240112305, + "step": 2560, + "unclipped_grad_norm": 0.799419641494751 + }, + { + "bits_per_byte": 1.8249698220595731, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2649726867675781, + "step": 2570, + "unclipped_grad_norm": 0.6938824653625488 + }, + { + "bits_per_byte": 1.8359280406245986, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.2725683450698853, + "step": 2580, + "unclipped_grad_norm": 0.7192572355270386 + }, + { + "bits_per_byte": 1.7947961538962967, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.2440578937530518, + "step": 2590, + "unclipped_grad_norm": 0.6899010539054871 + }, + { + "bits_per_byte": 1.8101101770597248, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.2546727657318115, + "step": 2600, + "unclipped_grad_norm": 0.7686203718185425 + }, + { + "bits_per_byte": 1.733514951775559, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.2015810012817383, + "step": 2610, + "unclipped_grad_norm": 0.6747013330459595 + }, + { + "bits_per_byte": 1.8015120764124894, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.2487130165100098, + "step": 2620, + "unclipped_grad_norm": 0.7034735083580017 + }, + { + "bits_per_byte": 1.8191664394888585, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.2609500885009766, + "step": 2630, + "unclipped_grad_norm": 0.7554193735122681 + }, + { + "bits_per_byte": 1.6790258604853463, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.1638120412826538, + "step": 2640, + "unclipped_grad_norm": 0.6473240256309509 + }, + { + "bits_per_byte": 1.7702973972635405, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2270766496658325, + "step": 2650, + "unclipped_grad_norm": 0.7774747014045715 + }, + { + "bits_per_byte": 1.8006040080159338, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.2480835914611816, + "step": 2660, + "unclipped_grad_norm": 0.7143652439117432 + }, + { + "bits_per_byte": 1.8092244664077812, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.254058837890625, + "step": 2670, + "unclipped_grad_norm": 0.770432710647583 + }, + { + "bits_per_byte": 1.7401871906984325, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.2062058448791504, + "step": 2680, + "unclipped_grad_norm": 0.6901031732559204 + }, + { + "bits_per_byte": 1.8980892779698013, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.31565523147583, + "step": 2690, + "unclipped_grad_norm": 0.70157390832901 + }, + { + "bits_per_byte": 1.8115494998647956, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.255670428276062, + "step": 2700, + "unclipped_grad_norm": 0.7487897872924805 + }, + { + "bits_per_byte": 1.7244038268808, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.1952656507492065, + "step": 2710, + "unclipped_grad_norm": 0.6885054111480713 + }, + { + "bits_per_byte": 1.7266999672524694, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.196857213973999, + "step": 2720, + "unclipped_grad_norm": 0.6634587645530701 + }, + { + "bits_per_byte": 1.7791412611188588, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.2332067489624023, + "step": 2730, + "unclipped_grad_norm": 0.744373083114624 + }, + { + "bits_per_byte": 1.7637409026646977, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.222532033920288, + "step": 2740, + "unclipped_grad_norm": 0.6819271445274353 + }, + { + "bits_per_byte": 1.7875560282602743, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.239039421081543, + "step": 2750, + "unclipped_grad_norm": 0.7268277406692505 + }, + { + "bits_per_byte": 1.8163767089092129, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.2590163946151733, + "step": 2760, + "unclipped_grad_norm": 0.7232861518859863 + }, + { + "bits_per_byte": 1.6994616989741196, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.1779770851135254, + "step": 2770, + "unclipped_grad_norm": 0.6630665063858032 + }, + { + "bits_per_byte": 1.7157783809920777, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.1892869472503662, + "step": 2780, + "unclipped_grad_norm": 0.6756263971328735 + }, + { + "bits_per_byte": 1.8017820891743441, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2489001750946045, + "step": 2790, + "unclipped_grad_norm": 0.7227063179016113 + }, + { + "bits_per_byte": 1.8514386399579628, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.2833194732666016, + "step": 2800, + "unclipped_grad_norm": 0.7472419142723083 + }, + { + "bits_per_byte": 1.7954293940167736, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.2444968223571777, + "step": 2810, + "unclipped_grad_norm": 0.7795827984809875 + }, + { + "bits_per_byte": 1.7820233463820176, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2352044582366943, + "step": 2820, + "unclipped_grad_norm": 0.672804594039917 + }, + { + "bits_per_byte": 1.8678590275143911, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.2947012186050415, + "step": 2830, + "unclipped_grad_norm": 0.7119723558425903 + }, + { + "bits_per_byte": 1.7464365242828344, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.2105375528335571, + "step": 2840, + "unclipped_grad_norm": 0.6869438886642456 + }, + { + "bits_per_byte": 1.7725790910925374, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.2286581993103027, + "step": 2850, + "unclipped_grad_norm": 0.7405384182929993 + }, + { + "bits_per_byte": 1.7485238777163563, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.211984395980835, + "step": 2860, + "unclipped_grad_norm": 0.7444901466369629 + }, + { + "bits_per_byte": 1.8269185574764997, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.266323447227478, + "step": 2870, + "unclipped_grad_norm": 0.7108240723609924 + }, + { + "bits_per_byte": 1.8165316652776402, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2591238021850586, + "step": 2880, + "unclipped_grad_norm": 0.7155829668045044 + }, + { + "bits_per_byte": 1.729610085687719, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.1988743543624878, + "step": 2890, + "unclipped_grad_norm": 0.686974287033081 + }, + { + "bits_per_byte": 1.691946917044669, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.172768235206604, + "step": 2900, + "unclipped_grad_norm": 0.6558113694190979 + }, + { + "bits_per_byte": 1.7561924121356782, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.2172998189926147, + "step": 2910, + "unclipped_grad_norm": 0.708652913570404 + }, + { + "bits_per_byte": 1.7338369032979741, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.2018041610717773, + "step": 2920, + "unclipped_grad_norm": 0.6739442348480225 + }, + { + "bits_per_byte": 1.7441622257078246, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.2089611291885376, + "step": 2930, + "unclipped_grad_norm": 0.6992833614349365 + }, + { + "bits_per_byte": 1.7762937443685232, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.23123300075531, + "step": 2940, + "unclipped_grad_norm": 0.7263411283493042 + }, + { + "bits_per_byte": 1.7671342923488718, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2248841524124146, + "step": 2950, + "unclipped_grad_norm": 0.762156069278717 + }, + { + "bits_per_byte": 1.8012826515562383, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.248553991317749, + "step": 2960, + "unclipped_grad_norm": 0.7459500432014465 + }, + { + "bits_per_byte": 1.7468738761639786, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.2108407020568848, + "step": 2970, + "unclipped_grad_norm": 0.7251895666122437 + }, + { + "bits_per_byte": 1.75555091684796, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.2168551683425903, + "step": 2980, + "unclipped_grad_norm": 0.7608514428138733 + }, + { + "bits_per_byte": 1.7761241694747725, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.231115460395813, + "step": 2990, + "unclipped_grad_norm": 0.7296990752220154 + }, + { + "bits_per_byte": 1.795146654538755, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.2443008422851562, + "step": 3000, + "unclipped_grad_norm": 0.6956633925437927 + }, + { + "bits_per_byte": 1.7609957155916256, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.2206292152404785, + "step": 3010, + "unclipped_grad_norm": 0.6949152946472168 + }, + { + "bits_per_byte": 1.710847638391841, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.1858692169189453, + "step": 3020, + "unclipped_grad_norm": 0.7081032991409302 + }, + { + "bits_per_byte": 1.8111713100155482, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.2554082870483398, + "step": 3030, + "unclipped_grad_norm": 0.7213758230209351 + }, + { + "bits_per_byte": 1.7164150607655717, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.1897282600402832, + "step": 3040, + "unclipped_grad_norm": 0.7003908157348633 + }, + { + "bits_per_byte": 1.7493662487402823, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.2125682830810547, + "step": 3050, + "unclipped_grad_norm": 0.6971246004104614 + }, + { + "bits_per_byte": 1.7548803564922457, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.2163903713226318, + "step": 3060, + "unclipped_grad_norm": 0.709033191204071 + }, + { + "bits_per_byte": 1.6898255110462763, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.1712977886199951, + "step": 3070, + "unclipped_grad_norm": 0.6854634284973145 + }, + { + "bits_per_byte": 1.7220010572655957, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.1936001777648926, + "step": 3080, + "unclipped_grad_norm": 0.7106940150260925 + }, + { + "bits_per_byte": 1.7844269759104763, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.236870527267456, + "step": 3090, + "unclipped_grad_norm": 0.7667668461799622 + }, + { + "bits_per_byte": 1.6982715790301488, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.177152156829834, + "step": 3100, + "unclipped_grad_norm": 0.6911866664886475 + }, + { + "bits_per_byte": 1.7838979572764222, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.2365038394927979, + "step": 3110, + "unclipped_grad_norm": 0.6885402202606201 + }, + { + "bits_per_byte": 1.7297457799992497, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.1989684104919434, + "step": 3120, + "unclipped_grad_norm": 0.6883302330970764 + }, + { + "bits_per_byte": 1.7958136032588012, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.2447631359100342, + "step": 3130, + "unclipped_grad_norm": 0.6977304220199585 + }, + { + "bits_per_byte": 1.7586135838945247, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.2189780473709106, + "step": 3140, + "unclipped_grad_norm": 0.690399706363678 + }, + { + "bits_per_byte": 1.7251435242621609, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.1957783699035645, + "step": 3150, + "unclipped_grad_norm": 0.7179538011550903 + }, + { + "bits_per_byte": 1.7378948339450822, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.204616904258728, + "step": 3160, + "unclipped_grad_norm": 0.696296215057373 + }, + { + "bits_per_byte": 1.6582524200529294, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.149412989616394, + "step": 3170, + "unclipped_grad_norm": 0.6602664589881897 + }, + { + "bits_per_byte": 1.7588497160741594, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.2191417217254639, + "step": 3180, + "unclipped_grad_norm": 0.713611900806427 + }, + { + "bits_per_byte": 1.7231459457723894, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.1943937540054321, + "step": 3190, + "unclipped_grad_norm": 0.6750349998474121 + }, + { + "bits_per_byte": 1.6910311094288246, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.172133445739746, + "step": 3200, + "unclipped_grad_norm": 0.6925067901611328 + }, + { + "bits_per_byte": 1.696170295001907, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.1756956577301025, + "step": 3210, + "unclipped_grad_norm": 0.66908198595047 + }, + { + "bits_per_byte": 1.7132325218113555, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.187522292137146, + "step": 3220, + "unclipped_grad_norm": 0.6956948637962341 + }, + { + "bits_per_byte": 1.7974027229527736, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.2458646297454834, + "step": 3230, + "unclipped_grad_norm": 0.7127658128738403 + }, + { + "bits_per_byte": 1.6758990439100094, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.161644697189331, + "step": 3240, + "unclipped_grad_norm": 0.6738332509994507 + }, + { + "bits_per_byte": 1.832743437878572, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.2703609466552734, + "step": 3250, + "unclipped_grad_norm": 0.7103776931762695 + }, + { + "bits_per_byte": 1.6853816513306321, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.1682175397872925, + "step": 3260, + "unclipped_grad_norm": 0.6714575886726379 + }, + { + "bits_per_byte": 1.7211172384228117, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.1929875612258911, + "step": 3270, + "unclipped_grad_norm": 0.7175151705741882 + }, + { + "bits_per_byte": 1.7737181321892017, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.2294477224349976, + "step": 3280, + "unclipped_grad_norm": 0.7096809148788452 + }, + { + "bits_per_byte": 1.7620688873330095, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.2213730812072754, + "step": 3290, + "unclipped_grad_norm": 0.7126203179359436 + }, + { + "bits_per_byte": 1.6882195370525195, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.17018461227417, + "step": 3300, + "unclipped_grad_norm": 0.6661962866783142 + }, + { + "bits_per_byte": 1.730336024457011, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.1993775367736816, + "step": 3310, + "unclipped_grad_norm": 0.7489113211631775 + }, + { + "bits_per_byte": 1.726321949385873, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.1965951919555664, + "step": 3320, + "unclipped_grad_norm": 0.6782962083816528 + }, + { + "bits_per_byte": 1.7297877437660603, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.1989974975585938, + "step": 3330, + "unclipped_grad_norm": 0.6996428370475769 + }, + { + "bits_per_byte": 1.7036374377370684, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.1808714866638184, + "step": 3340, + "unclipped_grad_norm": 0.6989784240722656 + }, + { + "bits_per_byte": 1.704051743942997, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.1811586618423462, + "step": 3350, + "unclipped_grad_norm": 0.6663082242012024 + }, + { + "bits_per_byte": 1.7350646874424838, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.2026551961898804, + "step": 3360, + "unclipped_grad_norm": 0.6982520222663879 + }, + { + "bits_per_byte": 1.687807638603703, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.1698991060256958, + "step": 3370, + "unclipped_grad_norm": 0.7006633877754211 + }, + { + "bits_per_byte": 1.7217919263621466, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.1934552192687988, + "step": 3380, + "unclipped_grad_norm": 0.7109994292259216 + }, + { + "bits_per_byte": 1.7432546732592216, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.2083320617675781, + "step": 3390, + "unclipped_grad_norm": 0.710293710231781 + }, + { + "bits_per_byte": 1.8022292440665872, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.2492101192474365, + "step": 3400, + "unclipped_grad_norm": 0.717724621295929 + }, + { + "bits_per_byte": 1.831886276346672, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.2697668075561523, + "step": 3410, + "unclipped_grad_norm": 0.7201820611953735 + }, + { + "bits_per_byte": 1.6916614258442368, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.1725703477859497, + "step": 3420, + "unclipped_grad_norm": 0.6613090634346008 + }, + { + "bits_per_byte": 1.619803634660738, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1227623224258423, + "step": 3430, + "unclipped_grad_norm": 0.680795431137085 + }, + { + "bits_per_byte": 1.7060680685417127, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.1825562715530396, + "step": 3440, + "unclipped_grad_norm": 0.7024274468421936 + }, + { + "bits_per_byte": 1.7857496944782623, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.2377873659133911, + "step": 3450, + "unclipped_grad_norm": 0.7458325028419495 + }, + { + "bits_per_byte": 1.6932861459469377, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.173696517944336, + "step": 3460, + "unclipped_grad_norm": 0.688740074634552 + }, + { + "bits_per_byte": 1.6982791462667868, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.1771574020385742, + "step": 3470, + "unclipped_grad_norm": 0.7061944603919983 + }, + { + "bits_per_byte": 1.6361734593178128, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.1341090202331543, + "step": 3480, + "unclipped_grad_norm": 0.6633186936378479 + }, + { + "bits_per_byte": 1.8413666479928332, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.2763381004333496, + "step": 3490, + "unclipped_grad_norm": 0.7541857361793518 + }, + { + "bits_per_byte": 1.70911130154873, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1846656799316406, + "step": 3500, + "unclipped_grad_norm": 0.685036301612854 + }, + { + "bits_per_byte": 1.7363929094550974, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.203575849533081, + "step": 3510, + "unclipped_grad_norm": 0.6899403929710388 + }, + { + "bits_per_byte": 1.7364856081039124, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.203640103340149, + "step": 3520, + "unclipped_grad_norm": 0.6868084073066711 + }, + { + "bits_per_byte": 1.6990805854198074, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.1777129173278809, + "step": 3530, + "unclipped_grad_norm": 0.698978066444397 + }, + { + "bits_per_byte": 1.7142506591044633, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.1882280111312866, + "step": 3540, + "unclipped_grad_norm": 0.7388191223144531 + }, + { + "bits_per_byte": 1.7127511423715904, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.1871886253356934, + "step": 3550, + "unclipped_grad_norm": 0.7400009632110596 + }, + { + "bits_per_byte": 1.6396853450484314, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.1365432739257812, + "step": 3560, + "unclipped_grad_norm": 0.6621814370155334 + }, + { + "bits_per_byte": 1.6909986047078116, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.172110915184021, + "step": 3570, + "unclipped_grad_norm": 0.7439326047897339 + }, + { + "bits_per_byte": 1.698092029142648, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.177027702331543, + "step": 3580, + "unclipped_grad_norm": 0.7215712070465088 + }, + { + "bits_per_byte": 1.7173618252585714, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.1903845071792603, + "step": 3590, + "unclipped_grad_norm": 0.709522545337677 + }, + { + "bits_per_byte": 1.7428047666445645, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.2080202102661133, + "step": 3600, + "unclipped_grad_norm": 0.7539828419685364 + }, + { + "bits_per_byte": 1.6780467632539844, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.1631333827972412, + "step": 3610, + "unclipped_grad_norm": 0.6853893995285034 + }, + { + "bits_per_byte": 1.6382623605951925, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.135556936264038, + "step": 3620, + "unclipped_grad_norm": 0.669718325138092 + }, + { + "bits_per_byte": 1.6957104133935, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.1753768920898438, + "step": 3630, + "unclipped_grad_norm": 0.7419127821922302 + }, + { + "bits_per_byte": 1.6747515756634528, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.1608493328094482, + "step": 3640, + "unclipped_grad_norm": 0.7065670490264893 + }, + { + "bits_per_byte": 1.7243671945761663, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.1952402591705322, + "step": 3650, + "unclipped_grad_norm": 0.7316760420799255 + }, + { + "bits_per_byte": 1.6266356454662632, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.127497911453247, + "step": 3660, + "unclipped_grad_norm": 0.7011439800262451 + }, + { + "bits_per_byte": 1.7177052746123445, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.1906225681304932, + "step": 3670, + "unclipped_grad_norm": 0.6995972394943237 + }, + { + "bits_per_byte": 1.6558486185418198, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.1477468013763428, + "step": 3680, + "unclipped_grad_norm": 0.6986536979675293 + }, + { + "bits_per_byte": 1.6278453714324321, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.1283364295959473, + "step": 3690, + "unclipped_grad_norm": 0.73860102891922 + }, + { + "bits_per_byte": 1.657545743340534, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.1489231586456299, + "step": 3700, + "unclipped_grad_norm": 0.7342231869697571 + }, + { + "bits_per_byte": 1.6973583511540673, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.1765191555023193, + "step": 3710, + "unclipped_grad_norm": 0.6928314566612244 + }, + { + "bits_per_byte": 1.67022224057033, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.1577098369598389, + "step": 3720, + "unclipped_grad_norm": 0.6940982937812805 + }, + { + "bits_per_byte": 1.6102929940680248, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.116170048713684, + "step": 3730, + "unclipped_grad_norm": 0.6923434734344482 + }, + { + "bits_per_byte": 1.6667588539472546, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.1553092002868652, + "step": 3740, + "unclipped_grad_norm": 0.6876158118247986 + }, + { + "bits_per_byte": 1.6907012467044698, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1719048023223877, + "step": 3750, + "unclipped_grad_norm": 0.7004653811454773 + }, + { + "bits_per_byte": 1.7679407189987675, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.2254431247711182, + "step": 3760, + "unclipped_grad_norm": 0.7061280608177185 + }, + { + "bits_per_byte": 1.733374613932455, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.2014837265014648, + "step": 3770, + "unclipped_grad_norm": 0.7066919803619385 + }, + { + "bits_per_byte": 1.5519569948433127, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.0757346153259277, + "step": 3780, + "unclipped_grad_norm": 0.670863926410675 + }, + { + "bits_per_byte": 1.654322616480714, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.1466890573501587, + "step": 3790, + "unclipped_grad_norm": 0.690252959728241 + }, + { + "bits_per_byte": 1.6559313421968849, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.1478041410446167, + "step": 3800, + "unclipped_grad_norm": 0.6939749717712402 + }, + { + "bits_per_byte": 1.6810659186898809, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.1652261018753052, + "step": 3810, + "unclipped_grad_norm": 0.7060471773147583 + }, + { + "bits_per_byte": 1.6584037647856886, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1495178937911987, + "step": 3820, + "unclipped_grad_norm": 0.7094374299049377 + }, + { + "bits_per_byte": 1.7198024310569655, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.1920762062072754, + "step": 3830, + "unclipped_grad_norm": 0.7122043371200562 + }, + { + "bits_per_byte": 1.6513688144521455, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.144641637802124, + "step": 3840, + "unclipped_grad_norm": 0.6882491111755371 + }, + { + "bits_per_byte": 1.5941935261381064, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.105010747909546, + "step": 3850, + "unclipped_grad_norm": 0.6913841366767883 + }, + { + "bits_per_byte": 1.6593587844459299, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.1501798629760742, + "step": 3860, + "unclipped_grad_norm": 0.7028048634529114 + }, + { + "bits_per_byte": 1.727495215030059, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.1974084377288818, + "step": 3870, + "unclipped_grad_norm": 0.7100816369056702 + }, + { + "bits_per_byte": 1.6252934928589298, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1265676021575928, + "step": 3880, + "unclipped_grad_norm": 0.6591659188270569 + }, + { + "bits_per_byte": 1.598235462398684, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.1078124046325684, + "step": 3890, + "unclipped_grad_norm": 0.6773467063903809 + }, + { + "bits_per_byte": 1.6802417778269463, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.1646548509597778, + "step": 3900, + "unclipped_grad_norm": 0.7679846286773682 + }, + { + "bits_per_byte": 1.6902781693833473, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1716115474700928, + "step": 3910, + "unclipped_grad_norm": 0.7588597536087036 + }, + { + "bits_per_byte": 1.6385531832578015, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.1357585191726685, + "step": 3920, + "unclipped_grad_norm": 0.6837491393089294 + }, + { + "bits_per_byte": 1.6743109561119425, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.1605439186096191, + "step": 3930, + "unclipped_grad_norm": 0.7004687190055847 + }, + { + "bits_per_byte": 1.685513734006495, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.1683090925216675, + "step": 3940, + "unclipped_grad_norm": 0.7021598219871521 + }, + { + "bits_per_byte": 1.7291545036455833, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1985585689544678, + "step": 3950, + "unclipped_grad_norm": 0.722118616104126 + }, + { + "bits_per_byte": 1.7163362927114765, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.189673662185669, + "step": 3960, + "unclipped_grad_norm": 0.7192909717559814 + }, + { + "bits_per_byte": 1.6383038084140504, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1355856657028198, + "step": 3970, + "unclipped_grad_norm": 0.6990612149238586 + }, + { + "bits_per_byte": 1.6849542744432382, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.1679213047027588, + "step": 3980, + "unclipped_grad_norm": 0.7282238006591797 + }, + { + "bits_per_byte": 1.694455627972805, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1745071411132812, + "step": 3990, + "unclipped_grad_norm": 0.7551571726799011 + }, + { + "bits_per_byte": 1.5908976506169716, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.1027262210845947, + "step": 4000, + "unclipped_grad_norm": 0.7165337204933167 + }, + { + "bits_per_byte": 1.6854710823090808, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.1682795286178589, + "step": 4010, + "unclipped_grad_norm": 0.7921661734580994 + }, + { + "bits_per_byte": 1.6380083422198681, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.1353808641433716, + "step": 4020, + "unclipped_grad_norm": 0.6990745663642883 + }, + { + "bits_per_byte": 1.6823289592778174, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.1661015748977661, + "step": 4030, + "unclipped_grad_norm": 0.7094166874885559 + }, + { + "bits_per_byte": 1.6057894563725312, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1130484342575073, + "step": 4040, + "unclipped_grad_norm": 0.7198181748390198 + }, + { + "bits_per_byte": 1.6562933656769512, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.148055076599121, + "step": 4050, + "unclipped_grad_norm": 0.7079614996910095 + }, + { + "bits_per_byte": 1.6577572820010953, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1490697860717773, + "step": 4060, + "unclipped_grad_norm": 0.7100932598114014 + }, + { + "bits_per_byte": 1.6862157671873168, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.1687957048416138, + "step": 4070, + "unclipped_grad_norm": 0.7280572056770325 + }, + { + "bits_per_byte": 1.7448933239566424, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.209467887878418, + "step": 4080, + "unclipped_grad_norm": 0.7571174502372742 + }, + { + "bits_per_byte": 1.6456920111124658, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.1407067775726318, + "step": 4090, + "unclipped_grad_norm": 0.756058394908905 + }, + { + "bits_per_byte": 1.631480912689021, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.1308563947677612, + "step": 4100, + "unclipped_grad_norm": 0.7193546295166016 + }, + { + "bits_per_byte": 1.6271056740510712, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.1278237104415894, + "step": 4110, + "unclipped_grad_norm": 0.7210359573364258 + }, + { + "bits_per_byte": 1.6738493546770266, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.1602239608764648, + "step": 4120, + "unclipped_grad_norm": 0.754092276096344 + }, + { + "bits_per_byte": 1.6235389258548274, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.1253514289855957, + "step": 4130, + "unclipped_grad_norm": 0.7249987125396729 + }, + { + "bits_per_byte": 1.6760534843304844, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.1617517471313477, + "step": 4140, + "unclipped_grad_norm": 0.7729062438011169 + }, + { + "bits_per_byte": 1.621320693623999, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1238138675689697, + "step": 4150, + "unclipped_grad_norm": 0.7151681780815125 + }, + { + "bits_per_byte": 1.6446339738443578, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.1399734020233154, + "step": 4160, + "unclipped_grad_norm": 0.7214731574058533 + }, + { + "bits_per_byte": 1.6990869487778895, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.1777173280715942, + "step": 4170, + "unclipped_grad_norm": 0.7596653699874878 + }, + { + "bits_per_byte": 1.720463876332184, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.1925346851348877, + "step": 4180, + "unclipped_grad_norm": 0.7532035708427429 + }, + { + "bits_per_byte": 1.6328731122477556, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.1318213939666748, + "step": 4190, + "unclipped_grad_norm": 0.7212545275688171 + }, + { + "bits_per_byte": 1.6776450117815687, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1628549098968506, + "step": 4200, + "unclipped_grad_norm": 0.7398514747619629 + }, + { + "bits_per_byte": 1.6659381527373374, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.154740333557129, + "step": 4210, + "unclipped_grad_norm": 0.7077314853668213 + }, + { + "bits_per_byte": 1.6822328209759851, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.1660349369049072, + "step": 4220, + "unclipped_grad_norm": 0.7291045188903809 + }, + { + "bits_per_byte": 1.6563177872133736, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.1480720043182373, + "step": 4230, + "unclipped_grad_norm": 0.7264421582221985 + }, + { + "bits_per_byte": 1.6143003618157792, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.1189477443695068, + "step": 4240, + "unclipped_grad_norm": 0.7086799144744873 + }, + { + "bits_per_byte": 1.6469608991105316, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1415863037109375, + "step": 4250, + "unclipped_grad_norm": 0.720244288444519 + }, + { + "bits_per_byte": 1.6140967343571575, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1188066005706787, + "step": 4260, + "unclipped_grad_norm": 0.7283206582069397 + }, + { + "bits_per_byte": 1.6061700539788908, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.1133122444152832, + "step": 4270, + "unclipped_grad_norm": 0.6953784227371216 + }, + { + "bits_per_byte": 1.6314439364190856, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1308307647705078, + "step": 4280, + "unclipped_grad_norm": 0.7203852534294128 + }, + { + "bits_per_byte": 1.6802823657325499, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.1646829843521118, + "step": 4290, + "unclipped_grad_norm": 0.711222767829895 + }, + { + "bits_per_byte": 1.5995525055389914, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.1087253093719482, + "step": 4300, + "unclipped_grad_norm": 0.7289605140686035 + }, + { + "bits_per_byte": 1.6895933344676115, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.1711368560791016, + "step": 4310, + "unclipped_grad_norm": 0.7529035806655884 + }, + { + "bits_per_byte": 1.6528860453980572, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.145693302154541, + "step": 4320, + "unclipped_grad_norm": 0.7355014085769653 + }, + { + "bits_per_byte": 1.6203823562808914, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.1231634616851807, + "step": 4330, + "unclipped_grad_norm": 0.7584924697875977 + }, + { + "bits_per_byte": 1.6461766582225972, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.141042709350586, + "step": 4340, + "unclipped_grad_norm": 0.7664118409156799 + }, + { + "bits_per_byte": 1.637568582581612, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.1350760459899902, + "step": 4350, + "unclipped_grad_norm": 0.7224692702293396 + }, + { + "bits_per_byte": 1.6123655569935724, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.1176066398620605, + "step": 4360, + "unclipped_grad_norm": 0.6963204145431519 + }, + { + "bits_per_byte": 1.6329797414912905, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.1318953037261963, + "step": 4370, + "unclipped_grad_norm": 0.758147656917572 + }, + { + "bits_per_byte": 1.6825791940348227, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.1662750244140625, + "step": 4380, + "unclipped_grad_norm": 0.7526863813400269 + }, + { + "bits_per_byte": 1.6432250919684896, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1389968395233154, + "step": 4390, + "unclipped_grad_norm": 0.7398001551628113 + }, + { + "bits_per_byte": 1.5952171668760418, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.1057202816009521, + "step": 4400, + "unclipped_grad_norm": 0.7230565547943115 + }, + { + "bits_per_byte": 1.6821843218684418, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.166001319885254, + "step": 4410, + "unclipped_grad_norm": 0.7536985874176025 + }, + { + "bits_per_byte": 1.623584673239957, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.1253831386566162, + "step": 4420, + "unclipped_grad_norm": 0.7180627584457397 + }, + { + "bits_per_byte": 1.6636365089208405, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.1531449556350708, + "step": 4430, + "unclipped_grad_norm": 0.7308915257453918 + }, + { + "bits_per_byte": 1.6118275952616736, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.1172337532043457, + "step": 4440, + "unclipped_grad_norm": 0.7632712721824646 + }, + { + "bits_per_byte": 1.599171048019378, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.1084609031677246, + "step": 4450, + "unclipped_grad_norm": 0.6959514617919922 + }, + { + "bits_per_byte": 1.5909119251769932, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.1027361154556274, + "step": 4460, + "unclipped_grad_norm": 0.6990306973457336 + }, + { + "bits_per_byte": 1.632260854010684, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.131397008895874, + "step": 4470, + "unclipped_grad_norm": 0.7412081956863403 + }, + { + "bits_per_byte": 1.6480567725618294, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.142345905303955, + "step": 4480, + "unclipped_grad_norm": 0.7477689981460571 + }, + { + "bits_per_byte": 1.6198278842145097, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.122779130935669, + "step": 4490, + "unclipped_grad_norm": 0.7336912155151367 + }, + { + "bits_per_byte": 1.650529539115935, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.1440598964691162, + "step": 4500, + "unclipped_grad_norm": 0.7142561078071594 + }, + { + "bits_per_byte": 1.586306229786887, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.0995436906814575, + "step": 4510, + "unclipped_grad_norm": 0.7236756682395935 + }, + { + "bits_per_byte": 1.6352588555805243, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.1334750652313232, + "step": 4520, + "unclipped_grad_norm": 0.726876974105835 + }, + { + "bits_per_byte": 1.651612685851069, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.144810676574707, + "step": 4530, + "unclipped_grad_norm": 0.712454617023468 + }, + { + "bits_per_byte": 1.711603330159732, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.1863930225372314, + "step": 4540, + "unclipped_grad_norm": 0.7713878750801086 + }, + { + "bits_per_byte": 1.5596071271189922, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.0810372829437256, + "step": 4550, + "unclipped_grad_norm": 0.7165849804878235 + }, + { + "bits_per_byte": 1.6289116638677816, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.129075527191162, + "step": 4560, + "unclipped_grad_norm": 0.828541100025177 + }, + { + "bits_per_byte": 1.609513052746362, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.1156294345855713, + "step": 4570, + "unclipped_grad_norm": 0.6911624073982239 + }, + { + "bits_per_byte": 1.5682516630819603, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.087029218673706, + "step": 4580, + "unclipped_grad_norm": 0.7403869032859802 + }, + { + "bits_per_byte": 1.613575798907694, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.1184455156326294, + "step": 4590, + "unclipped_grad_norm": 0.7791734933853149 + }, + { + "bits_per_byte": 1.6084557034088574, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.114896535873413, + "step": 4600, + "unclipped_grad_norm": 0.733103334903717 + }, + { + "bits_per_byte": 1.6640014561059713, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.1533979177474976, + "step": 4610, + "unclipped_grad_norm": 0.7711398005485535 + }, + { + "bits_per_byte": 1.6543915815237102, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.1467368602752686, + "step": 4620, + "unclipped_grad_norm": 0.7709856629371643 + }, + { + "bits_per_byte": 1.6312234546606794, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.1306779384613037, + "step": 4630, + "unclipped_grad_norm": 0.7350814342498779 + }, + { + "bits_per_byte": 1.6809548178974236, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.1651490926742554, + "step": 4640, + "unclipped_grad_norm": 0.7372580766677856 + }, + { + "bits_per_byte": 1.5952071918822919, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.1057133674621582, + "step": 4650, + "unclipped_grad_norm": 0.7462441325187683 + }, + { + "bits_per_byte": 1.5792171049183221, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.0946298837661743, + "step": 4660, + "unclipped_grad_norm": 0.7858431935310364 + }, + { + "bits_per_byte": 1.7203194109054591, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.192434549331665, + "step": 4670, + "unclipped_grad_norm": 0.7700279951095581 + }, + { + "bits_per_byte": 1.7093274837408645, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.184815526008606, + "step": 4680, + "unclipped_grad_norm": 0.8141517043113708 + }, + { + "bits_per_byte": 1.6729545289445873, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.1596037149429321, + "step": 4690, + "unclipped_grad_norm": 0.8173696398735046 + }, + { + "bits_per_byte": 1.6065360330599268, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.1135659217834473, + "step": 4700, + "unclipped_grad_norm": 0.7697358131408691 + }, + { + "bits_per_byte": 1.6622904007125372, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.1522119045257568, + "step": 4710, + "unclipped_grad_norm": 0.7910622954368591 + }, + { + "bits_per_byte": 1.603813031748816, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.1116784811019897, + "step": 4720, + "unclipped_grad_norm": 0.7602089643478394 + }, + { + "bits_per_byte": 1.5277098488287688, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.0589277744293213, + "step": 4730, + "unclipped_grad_norm": 0.7227722406387329 + }, + { + "bits_per_byte": 1.587534701862, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.1003952026367188, + "step": 4740, + "unclipped_grad_norm": 0.7166504859924316 + }, + { + "bits_per_byte": 1.630394154318219, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.1301031112670898, + "step": 4750, + "unclipped_grad_norm": 0.7454500794410706 + }, + { + "bits_per_byte": 1.6128519239302126, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.1179437637329102, + "step": 4760, + "unclipped_grad_norm": 0.7855631113052368 + }, + { + "bits_per_byte": 1.585962952415765, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.0993057489395142, + "step": 4770, + "unclipped_grad_norm": 0.7473867535591125 + }, + { + "bits_per_byte": 1.610390508231064, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.1162376403808594, + "step": 4780, + "unclipped_grad_norm": 0.7278242111206055 + }, + { + "bits_per_byte": 1.5724724612994354, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.0899548530578613, + "step": 4790, + "unclipped_grad_norm": 0.7452182769775391 + }, + { + "bits_per_byte": 1.5710837013937182, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.0889922380447388, + "step": 4800, + "unclipped_grad_norm": 0.7882425785064697 + }, + { + "bits_per_byte": 1.6297045038882592, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.1296250820159912, + "step": 4810, + "unclipped_grad_norm": 0.818852424621582 + }, + { + "bits_per_byte": 1.572377526876159, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.0898890495300293, + "step": 4820, + "unclipped_grad_norm": 0.7426112294197083 + }, + { + "bits_per_byte": 1.6040082320575453, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.1118137836456299, + "step": 4830, + "unclipped_grad_norm": 0.7535737752914429 + }, + { + "bits_per_byte": 1.6124343500539176, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.1176543235778809, + "step": 4840, + "unclipped_grad_norm": 0.7812339663505554 + }, + { + "bits_per_byte": 1.6146733921855005, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.1192063093185425, + "step": 4850, + "unclipped_grad_norm": 0.7442518472671509 + }, + { + "bits_per_byte": 1.5484468289392028, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.0733015537261963, + "step": 4860, + "unclipped_grad_norm": 0.7215056419372559 + }, + { + "bits_per_byte": 1.6174687981926246, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.1211439371109009, + "step": 4870, + "unclipped_grad_norm": 0.7703877091407776 + }, + { + "bits_per_byte": 1.4882243520171765, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.0315585136413574, + "step": 4880, + "unclipped_grad_norm": 0.7523868083953857 + }, + { + "bits_per_byte": 1.5491302879937316, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.073775291442871, + "step": 4890, + "unclipped_grad_norm": 0.7908336520195007 + }, + { + "bits_per_byte": 1.5940392577002824, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.1049038171768188, + "step": 4900, + "unclipped_grad_norm": 0.779721200466156 + }, + { + "bits_per_byte": 1.6902104082189073, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.1715645790100098, + "step": 4910, + "unclipped_grad_norm": 0.8124172687530518 + }, + { + "bits_per_byte": 1.628334490091486, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.1286754608154297, + "step": 4920, + "unclipped_grad_norm": 0.7750566005706787 + }, + { + "bits_per_byte": 1.5868837475284845, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.099943995475769, + "step": 4930, + "unclipped_grad_norm": 0.7316960692405701 + }, + { + "bits_per_byte": 1.5679647960203211, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.0868303775787354, + "step": 4940, + "unclipped_grad_norm": 0.7244730591773987 + }, + { + "bits_per_byte": 1.5223307474577328, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.0551992654800415, + "step": 4950, + "unclipped_grad_norm": 0.7124655246734619 + }, + { + "bits_per_byte": 1.5199752730715157, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.053566575050354, + "step": 4960, + "unclipped_grad_norm": 0.7967572808265686 + }, + { + "bits_per_byte": 1.6045529011128277, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.1121913194656372, + "step": 4970, + "unclipped_grad_norm": 0.8060715198516846 + }, + { + "bits_per_byte": 1.6645416536123314, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.1537723541259766, + "step": 4980, + "unclipped_grad_norm": 0.7765210270881653 + }, + { + "bits_per_byte": 1.589267599052094, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.1015963554382324, + "step": 4990, + "unclipped_grad_norm": 0.8595833778381348 + }, + { + "bits_per_byte": 1.5424119577204267, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.0691184997558594, + "step": 5000, + "unclipped_grad_norm": 0.7339364886283875 + }, + { + "bits_per_byte": 1.5465615551204446, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.0719947814941406, + "step": 5010, + "unclipped_grad_norm": 0.7331039309501648 + }, + { + "bits_per_byte": 1.5823095249634864, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.096773386001587, + "step": 5020, + "unclipped_grad_norm": 0.7315707802772522 + }, + { + "bits_per_byte": 1.5771629441364168, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.0932060480117798, + "step": 5030, + "unclipped_grad_norm": 0.7658902406692505 + }, + { + "bits_per_byte": 1.5384902373328022, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.066400170326233, + "step": 5040, + "unclipped_grad_norm": 0.7759442925453186 + }, + { + "bits_per_byte": 1.5112868815325775, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.047544240951538, + "step": 5050, + "unclipped_grad_norm": 0.7236356735229492 + }, + { + "bits_per_byte": 1.5259303443402912, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.0576943159103394, + "step": 5060, + "unclipped_grad_norm": 0.7110979557037354 + }, + { + "bits_per_byte": 1.6031350761391148, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.1112085580825806, + "step": 5070, + "unclipped_grad_norm": 0.7555773258209229 + }, + { + "bits_per_byte": 1.5663994099321679, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.0857453346252441, + "step": 5080, + "unclipped_grad_norm": 0.8156164288520813 + }, + { + "bits_per_byte": 1.5561516516978566, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.0786421298980713, + "step": 5090, + "unclipped_grad_norm": 0.7694247961044312 + }, + { + "bits_per_byte": 1.5601120681819254, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.0813872814178467, + "step": 5100, + "unclipped_grad_norm": 0.7503135800361633 + }, + { + "bits_per_byte": 1.480159397587616, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.025968313217163, + "step": 5110, + "unclipped_grad_norm": 0.7046789526939392 + }, + { + "bits_per_byte": 1.586371411211564, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0995888710021973, + "step": 5120, + "unclipped_grad_norm": 0.7313253283500671 + }, + { + "bits_per_byte": 1.5835048043869828, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.0976018905639648, + "step": 5130, + "unclipped_grad_norm": 0.7536035180091858 + }, + { + "bits_per_byte": 1.545830284888976, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.0714879035949707, + "step": 5140, + "unclipped_grad_norm": 0.7341952323913574 + }, + { + "bits_per_byte": 1.6291369611404118, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.1292316913604736, + "step": 5150, + "unclipped_grad_norm": 0.8132638931274414 + }, + { + "bits_per_byte": 1.4926222923650403, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.03460693359375, + "step": 5160, + "unclipped_grad_norm": 0.7804238200187683 + }, + { + "bits_per_byte": 1.673477872151163, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.1599664688110352, + "step": 5170, + "unclipped_grad_norm": 0.8154698610305786 + }, + { + "bits_per_byte": 1.5843855275420513, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.0982123613357544, + "step": 5180, + "unclipped_grad_norm": 0.791676938533783 + }, + { + "bits_per_byte": 1.5180251617933822, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.0522148609161377, + "step": 5190, + "unclipped_grad_norm": 0.7293723821640015 + }, + { + "bits_per_byte": 1.5458093030055706, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.0714733600616455, + "step": 5200, + "unclipped_grad_norm": 0.7410573959350586 + }, + { + "bits_per_byte": 1.5603046887508918, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.0815207958221436, + "step": 5210, + "unclipped_grad_norm": 0.7298501133918762 + }, + { + "bits_per_byte": 1.6592899913855848, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.150132179260254, + "step": 5220, + "unclipped_grad_norm": 0.8224835991859436 + }, + { + "bits_per_byte": 1.5529211295840495, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.0764029026031494, + "step": 5230, + "unclipped_grad_norm": 0.783123791217804 + }, + { + "bits_per_byte": 1.5484781297816599, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0733232498168945, + "step": 5240, + "unclipped_grad_norm": 0.7478788495063782 + }, + { + "bits_per_byte": 1.529591167046557, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0602318048477173, + "step": 5250, + "unclipped_grad_norm": 0.7350921630859375 + }, + { + "bits_per_byte": 1.5439278128051317, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.07016921043396, + "step": 5260, + "unclipped_grad_norm": 0.7814545035362244 + }, + { + "bits_per_byte": 1.543396042448664, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.069800615310669, + "step": 5270, + "unclipped_grad_norm": 0.7315354943275452 + }, + { + "bits_per_byte": 1.5462444191122537, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.071774959564209, + "step": 5280, + "unclipped_grad_norm": 0.7680234909057617 + }, + { + "bits_per_byte": 1.5416277168324923, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.0685749053955078, + "step": 5290, + "unclipped_grad_norm": 0.7800607681274414 + }, + { + "bits_per_byte": 1.5496768488581736, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.0741541385650635, + "step": 5300, + "unclipped_grad_norm": 0.7932760715484619 + }, + { + "bits_per_byte": 1.5575455710830997, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.0796083211898804, + "step": 5310, + "unclipped_grad_norm": 0.77089524269104 + }, + { + "bits_per_byte": 1.5471622905199085, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0724111795425415, + "step": 5320, + "unclipped_grad_norm": 0.7662466168403625 + }, + { + "bits_per_byte": 1.5395752758770958, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.0671522617340088, + "step": 5330, + "unclipped_grad_norm": 0.7971388101577759 + }, + { + "bits_per_byte": 1.4983589456672202, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.0385832786560059, + "step": 5340, + "unclipped_grad_norm": 0.7744171619415283 + }, + { + "bits_per_byte": 1.5835694698637073, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.097646713256836, + "step": 5350, + "unclipped_grad_norm": 0.7900452017784119 + }, + { + "bits_per_byte": 1.4958497187911317, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.03684401512146, + "step": 5360, + "unclipped_grad_norm": 0.762117326259613 + }, + { + "bits_per_byte": 1.4715757434830532, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.0200185775756836, + "step": 5370, + "unclipped_grad_norm": 0.7832645177841187 + }, + { + "bits_per_byte": 1.5852578235472274, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.098816990852356, + "step": 5380, + "unclipped_grad_norm": 0.7806663513183594 + }, + { + "bits_per_byte": 1.534045173738602, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.0633190870285034, + "step": 5390, + "unclipped_grad_norm": 0.7585294842720032 + }, + { + "bits_per_byte": 1.5504050234019267, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.0746588706970215, + "step": 5400, + "unclipped_grad_norm": 0.7929542660713196 + }, + { + "bits_per_byte": 1.5633890256114653, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.0836586952209473, + "step": 5410, + "unclipped_grad_norm": 0.7602052092552185 + }, + { + "bits_per_byte": 1.531367403864668, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.0614629983901978, + "step": 5420, + "unclipped_grad_norm": 0.8308344483375549 + }, + { + "bits_per_byte": 1.5704805582371424, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.0885741710662842, + "step": 5430, + "unclipped_grad_norm": 0.8202469348907471 + }, + { + "bits_per_byte": 1.515201206666215, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.0502574443817139, + "step": 5440, + "unclipped_grad_norm": 0.7972074747085571 + }, + { + "bits_per_byte": 1.516119937987124, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.0508942604064941, + "step": 5450, + "unclipped_grad_norm": 0.7874276041984558 + }, + { + "bits_per_byte": 1.504144442042245, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.0425934791564941, + "step": 5460, + "unclipped_grad_norm": 0.7748588919639587 + }, + { + "bits_per_byte": 1.5386909410863592, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0665392875671387, + "step": 5470, + "unclipped_grad_norm": 0.8019385933876038 + }, + { + "bits_per_byte": 1.548452676349332, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.073305606842041, + "step": 5480, + "unclipped_grad_norm": 0.7798033952713013 + }, + { + "bits_per_byte": 1.5648507061611483, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.0846718549728394, + "step": 5490, + "unclipped_grad_norm": 0.8339639902114868 + }, + { + "bits_per_byte": 1.5434290631176295, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.0698235034942627, + "step": 5500, + "unclipped_grad_norm": 0.8168340921401978 + }, + { + "bits_per_byte": 1.6309009871903117, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.130454421043396, + "step": 5510, + "unclipped_grad_norm": 0.8508331775665283 + }, + { + "bits_per_byte": 1.554397256676405, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.0774260759353638, + "step": 5520, + "unclipped_grad_norm": 0.8075413703918457 + }, + { + "bits_per_byte": 1.5142805835361464, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.0496193170547485, + "step": 5530, + "unclipped_grad_norm": 0.7754444479942322 + }, + { + "bits_per_byte": 1.5920397874013514, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.1035178899765015, + "step": 5540, + "unclipped_grad_norm": 0.8688627481460571 + }, + { + "bits_per_byte": 1.5369456611454033, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.0653295516967773, + "step": 5550, + "unclipped_grad_norm": 0.7904925346374512 + }, + { + "bits_per_byte": 1.5183832296724786, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.0524630546569824, + "step": 5560, + "unclipped_grad_norm": 0.8249436020851135 + }, + { + "bits_per_byte": 1.4868288847880757, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.0305912494659424, + "step": 5570, + "unclipped_grad_norm": 0.7715389728546143 + }, + { + "bits_per_byte": 1.567469829951138, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.0864872932434082, + "step": 5580, + "unclipped_grad_norm": 0.8076562285423279 + }, + { + "bits_per_byte": 1.5044120470469875, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.0427789688110352, + "step": 5590, + "unclipped_grad_norm": 0.8172252774238586 + }, + { + "bits_per_byte": 1.5483555061515946, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.0732382535934448, + "step": 5600, + "unclipped_grad_norm": 0.7971402406692505 + }, + { + "bits_per_byte": 1.5459828335002912, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.0715936422348022, + "step": 5610, + "unclipped_grad_norm": 0.7846203446388245 + }, + { + "bits_per_byte": 1.5657046000226822, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.085263729095459, + "step": 5620, + "unclipped_grad_norm": 0.8275474309921265 + }, + { + "bits_per_byte": 1.527136458670792, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.058530330657959, + "step": 5630, + "unclipped_grad_norm": 0.7675535678863525 + }, + { + "bits_per_byte": 1.5067444757579889, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.0443956851959229, + "step": 5640, + "unclipped_grad_norm": 0.7963153719902039 + }, + { + "bits_per_byte": 1.530226642941495, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.0606722831726074, + "step": 5650, + "unclipped_grad_norm": 0.8363965153694153 + }, + { + "bits_per_byte": 1.6120819576022998, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.1174100637435913, + "step": 5660, + "unclipped_grad_norm": 0.8411680459976196 + }, + { + "bits_per_byte": 1.5083466661334268, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.045506238937378, + "step": 5670, + "unclipped_grad_norm": 0.8423702716827393 + }, + { + "bits_per_byte": 1.5231550603033182, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0557706356048584, + "step": 5680, + "unclipped_grad_norm": 0.8063633441925049 + }, + { + "bits_per_byte": 1.6040151113635797, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.111818552017212, + "step": 5690, + "unclipped_grad_norm": 0.8261245489120483 + }, + { + "bits_per_byte": 1.5194806509676342, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.053223729133606, + "step": 5700, + "unclipped_grad_norm": 0.7742645144462585 + }, + { + "bits_per_byte": 1.5335094477811644, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.0629477500915527, + "step": 5710, + "unclipped_grad_norm": 0.8289422392845154 + }, + { + "bits_per_byte": 1.5419912881564164, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0688269138336182, + "step": 5720, + "unclipped_grad_norm": 0.7767664194107056 + }, + { + "bits_per_byte": 1.5612315032563913, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.0821632146835327, + "step": 5730, + "unclipped_grad_norm": 0.8206205368041992 + }, + { + "bits_per_byte": 1.5356351533458288, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.0644211769104004, + "step": 5740, + "unclipped_grad_norm": 0.8227375149726868 + }, + { + "bits_per_byte": 1.5339271936401102, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.0632373094558716, + "step": 5750, + "unclipped_grad_norm": 0.8271504044532776 + }, + { + "bits_per_byte": 1.5684253655593319, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.0871496200561523, + "step": 5760, + "unclipped_grad_norm": 0.8451591730117798 + }, + { + "bits_per_byte": 1.5106295638409797, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.047088623046875, + "step": 5770, + "unclipped_grad_norm": 0.7739905118942261 + }, + { + "bits_per_byte": 1.4943001551068578, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.0357699394226074, + "step": 5780, + "unclipped_grad_norm": 0.7823365330696106 + }, + { + "bits_per_byte": 1.5232613455815514, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 1.0558443069458008, + "step": 5790, + "unclipped_grad_norm": 0.8284541964530945 + }, + { + "bits_per_byte": 1.4850951276847277, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.029389500617981, + "step": 5800, + "unclipped_grad_norm": 0.7895095944404602 + }, + { + "bits_per_byte": 1.5627791751315059, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.0832359790802002, + "step": 5810, + "unclipped_grad_norm": 0.823820948600769 + }, + { + "bits_per_byte": 1.5000258015193824, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.039738655090332, + "step": 5820, + "unclipped_grad_norm": 0.774045467376709 + }, + { + "bits_per_byte": 1.5047940205145538, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.0430437326431274, + "step": 5830, + "unclipped_grad_norm": 0.780986487865448 + }, + { + "bits_per_byte": 1.51284814003711, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.04862642288208, + "step": 5840, + "unclipped_grad_norm": 0.8117980360984802 + }, + { + "bits_per_byte": 1.4582608466532545, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.010789394378662, + "step": 5850, + "unclipped_grad_norm": 0.8427215814590454 + }, + { + "bits_per_byte": 1.541775965877536, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.0686776638031006, + "step": 5860, + "unclipped_grad_norm": 0.8323033452033997 + }, + { + "bits_per_byte": 1.5345998177876345, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0637035369873047, + "step": 5870, + "unclipped_grad_norm": 0.8001008629798889 + }, + { + "bits_per_byte": 1.5461627273530938, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.0717183351516724, + "step": 5880, + "unclipped_grad_norm": 0.8404542803764343 + }, + { + "bits_per_byte": 1.557960049271679, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.0798956155776978, + "step": 5890, + "unclipped_grad_norm": 0.8062768578529358 + }, + { + "bits_per_byte": 1.5564311235055086, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.0788358449935913, + "step": 5900, + "unclipped_grad_norm": 0.8468642234802246 + }, + { + "bits_per_byte": 1.5528377179983812, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.0763450860977173, + "step": 5910, + "unclipped_grad_norm": 0.8097680807113647 + }, + { + "bits_per_byte": 1.4880272598992876, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.0314218997955322, + "step": 5920, + "unclipped_grad_norm": 0.808861494064331 + }, + { + "bits_per_byte": 1.591350996884646, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.1030404567718506, + "step": 5930, + "unclipped_grad_norm": 0.8698481321334839 + }, + { + "bits_per_byte": 1.4485551777344627, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.0040619373321533, + "step": 5940, + "unclipped_grad_norm": 0.774721086025238 + }, + { + "bits_per_byte": 1.5140621655695505, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.049467921257019, + "step": 5950, + "unclipped_grad_norm": 0.8082411289215088 + }, + { + "bits_per_byte": 1.53881270480317, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.0666236877441406, + "step": 5960, + "unclipped_grad_norm": 0.8784282207489014 + }, + { + "bits_per_byte": 1.5149937955892745, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.0501136779785156, + "step": 5970, + "unclipped_grad_norm": 0.8612354397773743 + }, + { + "bits_per_byte": 1.5512064625549473, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.0752143859863281, + "step": 5980, + "unclipped_grad_norm": 0.8223481774330139 + }, + { + "bits_per_byte": 1.5262275303609822, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.057900309562683, + "step": 5990, + "unclipped_grad_norm": 0.8561235070228577 + }, + { + "bits_per_byte": 1.4749209780249857, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.0223373174667358, + "step": 6000, + "unclipped_grad_norm": 0.7864241003990173 + }, + { + "bits_per_byte": 1.5060283399997962, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0438992977142334, + "step": 6010, + "unclipped_grad_norm": 0.8298590779304504 + }, + { + "bits_per_byte": 1.475602545270355, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0228097438812256, + "step": 6020, + "unclipped_grad_norm": 0.83128821849823 + }, + { + "bits_per_byte": 1.5114294551501426, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.0476430654525757, + "step": 6030, + "unclipped_grad_norm": 0.8141233921051025 + }, + { + "bits_per_byte": 1.5732303888417878, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.0904802083969116, + "step": 6040, + "unclipped_grad_norm": 0.8527133464813232 + }, + { + "bits_per_byte": 1.4789243901717701, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.025112271308899, + "step": 6050, + "unclipped_grad_norm": 0.8315216302871704 + }, + { + "bits_per_byte": 1.4705367962891909, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.0192984342575073, + "step": 6060, + "unclipped_grad_norm": 0.8039374351501465 + }, + { + "bits_per_byte": 1.5399969773370112, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 1.0674445629119873, + "step": 6070, + "unclipped_grad_norm": 0.8699690699577332 + }, + { + "bits_per_byte": 1.476775982897192, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.0236231088638306, + "step": 6080, + "unclipped_grad_norm": 0.8226972818374634 + }, + { + "bits_per_byte": 1.4930345347791585, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.0348926782608032, + "step": 6090, + "unclipped_grad_norm": 0.8191179037094116 + }, + { + "bits_per_byte": 1.4984270507969617, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.038630485534668, + "step": 6100, + "unclipped_grad_norm": 0.8075997233390808 + }, + { + "bits_per_byte": 1.560406846445504, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.0815916061401367, + "step": 6110, + "unclipped_grad_norm": 0.8406150937080383 + }, + { + "bits_per_byte": 1.5489887462720715, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0736771821975708, + "step": 6120, + "unclipped_grad_norm": 0.8380064368247986 + }, + { + "bits_per_byte": 1.4647881042014508, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.015313744544983, + "step": 6130, + "unclipped_grad_norm": 0.831690788269043 + }, + { + "bits_per_byte": 1.4808624626643432, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.0264556407928467, + "step": 6140, + "unclipped_grad_norm": 0.8637996315956116 + }, + { + "bits_per_byte": 1.548437541876056, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.0732951164245605, + "step": 6150, + "unclipped_grad_norm": 0.816342830657959 + }, + { + "bits_per_byte": 1.4100234967045557, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 0.9773538112640381, + "step": 6160, + "unclipped_grad_norm": 0.8161224722862244 + }, + { + "bits_per_byte": 1.5108202926007865, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 1.0472208261489868, + "step": 6170, + "unclipped_grad_norm": 0.8210566639900208 + }, + { + "bits_per_byte": 1.4860711292283741, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0300660133361816, + "step": 6180, + "unclipped_grad_norm": 0.8188468813896179 + }, + { + "bits_per_byte": 1.5124393372760092, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.0483430624008179, + "step": 6190, + "unclipped_grad_norm": 0.8787809014320374 + }, + { + "bits_per_byte": 1.5045876413335184, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.0429006814956665, + "step": 6200, + "unclipped_grad_norm": 0.8184348940849304 + }, + { + "bits_per_byte": 1.5048669411585196, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.043094277381897, + "step": 6210, + "unclipped_grad_norm": 0.827497124671936 + }, + { + "bits_per_byte": 1.5117109907496051, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0478382110595703, + "step": 6220, + "unclipped_grad_norm": 0.8387815952301025 + }, + { + "bits_per_byte": 1.4541616401699395, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.0079480409622192, + "step": 6230, + "unclipped_grad_norm": 0.8376311659812927 + }, + { + "bits_per_byte": 1.4883141269609268, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.031620740890503, + "step": 6240, + "unclipped_grad_norm": 0.8739249110221863 + }, + { + "bits_per_byte": 1.5538819966544202, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.0770689249038696, + "step": 6250, + "unclipped_grad_norm": 0.847855269908905 + }, + { + "bits_per_byte": 1.4089668352976545, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 0.9766213893890381, + "step": 6260, + "unclipped_grad_norm": 0.8132977485656738 + }, + { + "bits_per_byte": 1.504553072820695, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.0428767204284668, + "step": 6270, + "unclipped_grad_norm": 0.8422287702560425 + }, + { + "bits_per_byte": 1.509997527599059, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.0466505289077759, + "step": 6280, + "unclipped_grad_norm": 0.8678255677223206 + }, + { + "bits_per_byte": 1.4391085146878704, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 0.997514009475708, + "step": 6290, + "unclipped_grad_norm": 0.8154304623603821 + }, + { + "bits_per_byte": 1.519478415193173, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.0532221794128418, + "step": 6300, + "unclipped_grad_norm": 0.8448652029037476 + }, + { + "bits_per_byte": 1.4847654369430237, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.029160976409912, + "step": 6310, + "unclipped_grad_norm": 0.8183073401451111 + }, + { + "bits_per_byte": 1.4313677475551863, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 0.9921485185623169, + "step": 6320, + "unclipped_grad_norm": 0.8498275876045227 + }, + { + "bits_per_byte": 1.5128925115610325, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 1.0486571788787842, + "step": 6330, + "unclipped_grad_norm": 0.8627044558525085 + }, + { + "bits_per_byte": 1.461181112064905, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 1.0128135681152344, + "step": 6340, + "unclipped_grad_norm": 0.822685956954956 + }, + { + "bits_per_byte": 1.5772647578657275, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.0932766199111938, + "step": 6350, + "unclipped_grad_norm": 0.9062409400939941 + }, + { + "bits_per_byte": 1.4535081060966608, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.0074950456619263, + "step": 6360, + "unclipped_grad_norm": 0.9099148511886597 + }, + { + "bits_per_byte": 1.4330259182834801, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 0.9932978749275208, + "step": 6370, + "unclipped_grad_norm": 0.8210288286209106 + }, + { + "bits_per_byte": 1.4872781034721294, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.030902624130249, + "step": 6380, + "unclipped_grad_norm": 0.8931319713592529 + }, + { + "bits_per_byte": 1.5242339074721805, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0565184354782104, + "step": 6390, + "unclipped_grad_norm": 0.8513842821121216 + }, + { + "bits_per_byte": 1.527535114455492, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.0588066577911377, + "step": 6400, + "unclipped_grad_norm": 0.8990474343299866 + }, + { + "bits_per_byte": 1.4879281978923908, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.031353235244751, + "step": 6410, + "unclipped_grad_norm": 0.8824620246887207 + }, + { + "bits_per_byte": 1.4718911596647355, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0202372074127197, + "step": 6420, + "unclipped_grad_norm": 0.8558386564254761 + }, + { + "bits_per_byte": 1.5203593103308923, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.053832769393921, + "step": 6430, + "unclipped_grad_norm": 0.8756915330886841 + }, + { + "bits_per_byte": 1.46260134479573, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.0137979984283447, + "step": 6440, + "unclipped_grad_norm": 0.8763707280158997 + }, + { + "bits_per_byte": 1.530507662593005, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0608670711517334, + "step": 6450, + "unclipped_grad_norm": 0.9259318709373474 + }, + { + "bits_per_byte": 1.5846079011096168, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.0983664989471436, + "step": 6460, + "unclipped_grad_norm": 0.9025880694389343 + }, + { + "bits_per_byte": 1.5613885234166291, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.0822720527648926, + "step": 6470, + "unclipped_grad_norm": 0.9080148935317993 + }, + { + "bits_per_byte": 1.4372277124180348, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 0.9962103366851807, + "step": 6480, + "unclipped_grad_norm": 0.8463495969772339 + }, + { + "bits_per_byte": 1.5543678476431075, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0774056911468506, + "step": 6490, + "unclipped_grad_norm": 0.9173493981361389 + }, + { + "bits_per_byte": 1.5280765158404082, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.0591819286346436, + "step": 6500, + "unclipped_grad_norm": 0.8818939924240112 + }, + { + "bits_per_byte": 1.4639863210831283, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 1.0147579908370972, + "step": 6510, + "unclipped_grad_norm": 0.8582631349563599 + }, + { + "bits_per_byte": 1.4175933130822822, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 0.9826008081436157, + "step": 6520, + "unclipped_grad_norm": 0.8186147809028625 + }, + { + "bits_per_byte": 1.4593926644785826, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.0115739107131958, + "step": 6530, + "unclipped_grad_norm": 0.8432228565216064 + }, + { + "bits_per_byte": 1.4987111661361872, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.0388274192810059, + "step": 6540, + "unclipped_grad_norm": 0.8676847815513611 + }, + { + "bits_per_byte": 1.4334030762368222, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 0.9935593008995056, + "step": 6550, + "unclipped_grad_norm": 0.8410716652870178 + }, + { + "bits_per_byte": 1.5210352021487832, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.0543012619018555, + "step": 6560, + "unclipped_grad_norm": 0.8957542181015015 + }, + { + "bits_per_byte": 1.4683194239716169, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 1.017761468887329, + "step": 6570, + "unclipped_grad_norm": 0.8648741245269775 + }, + { + "bits_per_byte": 1.517477912998337, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.051835536956787, + "step": 6580, + "unclipped_grad_norm": 0.9248769283294678 + }, + { + "bits_per_byte": 1.4883793083856038, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.0316659212112427, + "step": 6590, + "unclipped_grad_norm": 0.8476209044456482 + }, + { + "bits_per_byte": 1.4281028289112068, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9898854494094849, + "step": 6600, + "unclipped_grad_norm": 0.8366096615791321 + }, + { + "bits_per_byte": 1.4440667745122453, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.000950813293457, + "step": 6610, + "unclipped_grad_norm": 0.8324962258338928 + }, + { + "bits_per_byte": 1.431520984097105, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 0.9922547340393066, + "step": 6620, + "unclipped_grad_norm": 0.8581469655036926 + }, + { + "bits_per_byte": 1.4652533172720346, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.0156362056732178, + "step": 6630, + "unclipped_grad_norm": 0.835750162601471 + }, + { + "bits_per_byte": 1.511406753440229, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.047627329826355, + "step": 6640, + "unclipped_grad_norm": 0.9299710392951965 + }, + { + "bits_per_byte": 1.5100805952194256, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.046708106994629, + "step": 6650, + "unclipped_grad_norm": 0.8319801092147827 + }, + { + "bits_per_byte": 1.506217864881047, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.0440306663513184, + "step": 6660, + "unclipped_grad_norm": 0.8589498400688171 + }, + { + "bits_per_byte": 1.493190523043491, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.0350008010864258, + "step": 6670, + "unclipped_grad_norm": 0.9032244086265564 + }, + { + "bits_per_byte": 1.4646532698031742, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.015220284461975, + "step": 6680, + "unclipped_grad_norm": 0.8491512537002563 + }, + { + "bits_per_byte": 1.4953650716810003, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.0365080833435059, + "step": 6690, + "unclipped_grad_norm": 0.8763020038604736 + }, + { + "bits_per_byte": 1.544096527785628, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.0702861547470093, + "step": 6700, + "unclipped_grad_norm": 0.9039784669876099 + }, + { + "bits_per_byte": 1.5190818232002834, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.0529472827911377, + "step": 6710, + "unclipped_grad_norm": 0.8633220791816711 + }, + { + "bits_per_byte": 1.5275646954714406, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.0588271617889404, + "step": 6720, + "unclipped_grad_norm": 0.919657826423645 + }, + { + "bits_per_byte": 1.4703340287438238, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 1.019157886505127, + "step": 6730, + "unclipped_grad_norm": 0.8617632985115051 + }, + { + "bits_per_byte": 1.4327182413210866, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 0.9930846095085144, + "step": 6740, + "unclipped_grad_norm": 0.8640421032905579 + }, + { + "bits_per_byte": 1.5101160236455033, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 1.0467326641082764, + "step": 6750, + "unclipped_grad_norm": 0.8495616316795349 + }, + { + "bits_per_byte": 1.4808132756261965, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 1.0264215469360352, + "step": 6760, + "unclipped_grad_norm": 0.85832679271698 + }, + { + "bits_per_byte": 1.3797458650288328, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 0.9563669562339783, + "step": 6770, + "unclipped_grad_norm": 0.8209432363510132 + }, + { + "bits_per_byte": 1.4773923687178843, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 1.0240503549575806, + "step": 6780, + "unclipped_grad_norm": 0.8844631314277649 + }, + { + "bits_per_byte": 1.4581760592063793, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.0107306241989136, + "step": 6790, + "unclipped_grad_norm": 0.8933369517326355 + }, + { + "bits_per_byte": 1.508762520183213, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0457944869995117, + "step": 6800, + "unclipped_grad_norm": 0.9430736899375916 + }, + { + "bits_per_byte": 1.5071114867349302, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0446500778198242, + "step": 6810, + "unclipped_grad_norm": 0.9123260974884033 + }, + { + "bits_per_byte": 1.4829224708563777, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 1.027883529663086, + "step": 6820, + "unclipped_grad_norm": 0.9451769590377808 + }, + { + "bits_per_byte": 1.4736034189367258, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.0214240550994873, + "step": 6830, + "unclipped_grad_norm": 0.8638833165168762 + }, + { + "bits_per_byte": 1.432557867499157, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 0.9929734468460083, + "step": 6840, + "unclipped_grad_norm": 0.8747947216033936 + }, + { + "bits_per_byte": 1.4988186552929763, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.038901925086975, + "step": 6850, + "unclipped_grad_norm": 0.8931675553321838 + }, + { + "bits_per_byte": 1.4656223920407863, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.0158920288085938, + "step": 6860, + "unclipped_grad_norm": 0.9086399674415588 + }, + { + "bits_per_byte": 1.4122386332476686, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 0.9788892269134521, + "step": 6870, + "unclipped_grad_norm": 0.8658599257469177 + }, + { + "bits_per_byte": 1.4283602869395484, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 0.9900639057159424, + "step": 6880, + "unclipped_grad_norm": 0.8174008131027222 + }, + { + "bits_per_byte": 1.4241431003577414, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 0.9871407747268677, + "step": 6890, + "unclipped_grad_norm": 0.9087112545967102 + }, + { + "bits_per_byte": 1.534203053812094, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 1.063428521156311, + "step": 6900, + "unclipped_grad_norm": 0.9002585411071777 + }, + { + "bits_per_byte": 1.5000476433160421, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.039753794670105, + "step": 6910, + "unclipped_grad_norm": 0.9408553838729858 + }, + { + "bits_per_byte": 1.447142340257625, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 1.0030826330184937, + "step": 6920, + "unclipped_grad_norm": 0.8834111094474792 + }, + { + "bits_per_byte": 1.43785665297224, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.9966462850570679, + "step": 6930, + "unclipped_grad_norm": 0.933394193649292 + }, + { + "bits_per_byte": 1.4172609566094898, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 0.9823704361915588, + "step": 6940, + "unclipped_grad_norm": 0.8906471729278564 + }, + { + "bits_per_byte": 1.4269259516313526, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 0.9890697002410889, + "step": 6950, + "unclipped_grad_norm": 0.9011527895927429 + }, + { + "bits_per_byte": 1.4799932623468826, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 1.025853157043457, + "step": 6960, + "unclipped_grad_norm": 0.8841021060943604 + }, + { + "bits_per_byte": 1.4803606172891255, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 1.0261077880859375, + "step": 6970, + "unclipped_grad_norm": 0.894124448299408 + }, + { + "bits_per_byte": 1.4519604342215464, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.0064222812652588, + "step": 6980, + "unclipped_grad_norm": 0.8639485239982605 + }, + { + "bits_per_byte": 1.4960554100415637, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.0369865894317627, + "step": 6990, + "unclipped_grad_norm": 0.9485885500907898 + }, + { + "bits_per_byte": 1.5266048602969753, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 1.0581618547439575, + "step": 7000, + "unclipped_grad_norm": 0.895354688167572 + }, + { + "bits_per_byte": 1.515237666988198, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 1.0502827167510986, + "step": 7010, + "unclipped_grad_norm": 0.9304215312004089 + }, + { + "bits_per_byte": 1.4472281596004053, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 1.0031421184539795, + "step": 7020, + "unclipped_grad_norm": 0.8737574219703674 + }, + { + "bits_per_byte": 1.5096649131522903, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 1.0464199781417847, + "step": 7030, + "unclipped_grad_norm": 0.9219122529029846 + }, + { + "bits_per_byte": 1.5077884104487262, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.045119285583496, + "step": 7040, + "unclipped_grad_norm": 0.899664044380188 + }, + { + "bits_per_byte": 1.484001490007891, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.0286314487457275, + "step": 7050, + "unclipped_grad_norm": 0.911277174949646 + }, + { + "bits_per_byte": 1.4946274380914497, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 1.0359967947006226, + "step": 7060, + "unclipped_grad_norm": 0.9353191256523132 + }, + { + "bits_per_byte": 1.465652832969989, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 1.0159131288528442, + "step": 7070, + "unclipped_grad_norm": 0.9585835337638855 + }, + { + "bits_per_byte": 1.433203920327123, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 0.9934212565422058, + "step": 7080, + "unclipped_grad_norm": 0.8950881958007812 + }, + { + "bits_per_byte": 1.465784399697899, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 1.0160043239593506, + "step": 7090, + "unclipped_grad_norm": 0.9099149703979492 + }, + { + "bits_per_byte": 1.4697200506802435, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 1.0187323093414307, + "step": 7100, + "unclipped_grad_norm": 0.8818370699882507 + }, + { + "bits_per_byte": 1.4770736848658355, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 1.023829460144043, + "step": 7110, + "unclipped_grad_norm": 0.8931179046630859 + }, + { + "bits_per_byte": 1.5174638104209661, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.051825761795044, + "step": 7120, + "unclipped_grad_norm": 0.9178721308708191 + }, + { + "bits_per_byte": 1.4773056894618495, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 1.023990273475647, + "step": 7130, + "unclipped_grad_norm": 0.9431551098823547 + }, + { + "bits_per_byte": 1.4591236836126333, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.0113874673843384, + "step": 7140, + "unclipped_grad_norm": 0.9280264377593994 + }, + { + "bits_per_byte": 1.429918793721667, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 0.9911441802978516, + "step": 7150, + "unclipped_grad_norm": 0.8666377067565918 + }, + { + "bits_per_byte": 1.5222081238276675, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0551142692565918, + "step": 7160, + "unclipped_grad_norm": 0.9168943166732788 + }, + { + "bits_per_byte": 1.4509791012157232, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.005742073059082, + "step": 7170, + "unclipped_grad_norm": 0.8682712316513062 + }, + { + "bits_per_byte": 1.4650968130597495, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 1.0155277252197266, + "step": 7180, + "unclipped_grad_norm": 0.8913494348526001 + }, + { + "bits_per_byte": 1.4268142488996172, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 0.9889922738075256, + "step": 7190, + "unclipped_grad_norm": 0.9602050185203552 + }, + { + "bits_per_byte": 1.4729980400056886, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 1.0210044384002686, + "step": 7200, + "unclipped_grad_norm": 0.9293481111526489 + }, + { + "bits_per_byte": 1.533011042058964, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 1.0626022815704346, + "step": 7210, + "unclipped_grad_norm": 0.9510640501976013 + }, + { + "bits_per_byte": 1.5085778108161865, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 1.0456664562225342, + "step": 7220, + "unclipped_grad_norm": 0.9551865458488464 + }, + { + "bits_per_byte": 1.4407144886816274, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 0.9986271858215332, + "step": 7230, + "unclipped_grad_norm": 0.9295659065246582 + }, + { + "bits_per_byte": 1.4591302189533661, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 1.0113919973373413, + "step": 7240, + "unclipped_grad_norm": 0.9115588068962097 + }, + { + "bits_per_byte": 1.4913319065356165, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 1.0337125062942505, + "step": 7250, + "unclipped_grad_norm": 0.9422294497489929 + }, + { + "bits_per_byte": 1.3775590196317868, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 0.9548511505126953, + "step": 7260, + "unclipped_grad_norm": 0.8549509644508362 + }, + { + "bits_per_byte": 1.4166838688245198, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 0.9819704294204712, + "step": 7270, + "unclipped_grad_norm": 0.9206053614616394 + }, + { + "bits_per_byte": 1.4830483621568094, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.027970790863037, + "step": 7280, + "unclipped_grad_norm": 0.9591236114501953 + }, + { + "bits_per_byte": 1.421203314915218, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 0.9851030707359314, + "step": 7290, + "unclipped_grad_norm": 0.9255973696708679 + }, + { + "bits_per_byte": 1.446030128454495, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 1.0023117065429688, + "step": 7300, + "unclipped_grad_norm": 0.9709646701812744 + }, + { + "bits_per_byte": 1.4516477697622778, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 1.0062055587768555, + "step": 7310, + "unclipped_grad_norm": 0.8941022753715515 + }, + { + "bits_per_byte": 1.397832592489471, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 0.9689037203788757, + "step": 7320, + "unclipped_grad_norm": 0.8796373605728149 + }, + { + "bits_per_byte": 1.4256914601634594, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 0.9882140159606934, + "step": 7330, + "unclipped_grad_norm": 0.9331433773040771 + }, + { + "bits_per_byte": 1.3916424209369658, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.9646130204200745, + "step": 7340, + "unclipped_grad_norm": 0.9056965708732605 + }, + { + "bits_per_byte": 1.4744784666643156, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 1.0220305919647217, + "step": 7350, + "unclipped_grad_norm": 0.9329537749290466 + }, + { + "bits_per_byte": 1.4227147844423258, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 0.9861507415771484, + "step": 7360, + "unclipped_grad_norm": 0.9041620492935181 + }, + { + "bits_per_byte": 1.475013332708499, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 1.0224013328552246, + "step": 7370, + "unclipped_grad_norm": 0.939182460308075 + }, + { + "bits_per_byte": 1.4122973653229383, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 0.9789299368858337, + "step": 7380, + "unclipped_grad_norm": 0.9086670279502869 + }, + { + "bits_per_byte": 1.482260337650556, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 1.0274245738983154, + "step": 7390, + "unclipped_grad_norm": 0.9115052819252014 + }, + { + "bits_per_byte": 1.4016735670225156, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.9715660810470581, + "step": 7400, + "unclipped_grad_norm": 0.9169398546218872 + }, + { + "bits_per_byte": 1.3998930306381328, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 0.9703319072723389, + "step": 7410, + "unclipped_grad_norm": 0.8746247887611389 + }, + { + "bits_per_byte": 1.4749039517425502, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0223255157470703, + "step": 7420, + "unclipped_grad_norm": 0.8939687013626099 + }, + { + "bits_per_byte": 1.4887745245172865, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 1.0319398641586304, + "step": 7430, + "unclipped_grad_norm": 0.9612749814987183 + }, + { + "bits_per_byte": 1.502158902338034, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 1.0412172079086304, + "step": 7440, + "unclipped_grad_norm": 0.9905859231948853 + }, + { + "bits_per_byte": 1.450549488553868, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 1.0054442882537842, + "step": 7450, + "unclipped_grad_norm": 0.8959178328514099 + }, + { + "bits_per_byte": 1.4430171643940295, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 1.0002232789993286, + "step": 7460, + "unclipped_grad_norm": 0.9044798016548157 + }, + { + "bits_per_byte": 1.428750515574356, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 0.9903343915939331, + "step": 7470, + "unclipped_grad_norm": 0.9605315923690796 + }, + { + "bits_per_byte": 1.4683799618647204, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 1.017803430557251, + "step": 7480, + "unclipped_grad_norm": 0.9531717896461487 + }, + { + "bits_per_byte": 1.5187091367958636, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 1.0526889562606812, + "step": 7490, + "unclipped_grad_norm": 0.9383010268211365 + }, + { + "bits_per_byte": 1.4713956776475998, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.019893765449524, + "step": 7500, + "unclipped_grad_norm": 0.9227973222732544 + }, + { + "bits_per_byte": 1.4330450083577257, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 0.9933111071586609, + "step": 7510, + "unclipped_grad_norm": 0.8817017674446106 + }, + { + "bits_per_byte": 1.4488226107565545, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 1.0042473077774048, + "step": 7520, + "unclipped_grad_norm": 0.9354658722877502 + }, + { + "bits_per_byte": 1.498810400125735, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.0388962030410767, + "step": 7530, + "unclipped_grad_norm": 0.9494727849960327 + }, + { + "bits_per_byte": 1.4547981479607828, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.0083892345428467, + "step": 7540, + "unclipped_grad_norm": 0.8941510319709778 + }, + { + "bits_per_byte": 1.478494605527264, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 1.0248143672943115, + "step": 7550, + "unclipped_grad_norm": 0.9451413750648499 + }, + { + "bits_per_byte": 1.4515287577678808, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 1.0061230659484863, + "step": 7560, + "unclipped_grad_norm": 0.9424342513084412 + }, + { + "bits_per_byte": 1.4043914088541005, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 0.9734499454498291, + "step": 7570, + "unclipped_grad_norm": 0.9017007350921631 + }, + { + "bits_per_byte": 1.5796333029334102, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 1.0949183702468872, + "step": 7580, + "unclipped_grad_norm": 1.0103739500045776 + }, + { + "bits_per_byte": 1.438846929075908, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 0.9973326921463013, + "step": 7590, + "unclipped_grad_norm": 0.9061667323112488 + }, + { + "bits_per_byte": 1.4099923678447495, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 0.9773322343826294, + "step": 7600, + "unclipped_grad_norm": 0.9454378485679626 + }, + { + "bits_per_byte": 1.4439556737197878, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 1.0008738040924072, + "step": 7610, + "unclipped_grad_norm": 0.9156769514083862 + }, + { + "bits_per_byte": 1.4071500105739398, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 0.9753620624542236, + "step": 7620, + "unclipped_grad_norm": 0.8833504915237427 + }, + { + "bits_per_byte": 1.4007524279444943, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 0.9709275960922241, + "step": 7630, + "unclipped_grad_norm": 0.9482987523078918 + }, + { + "bits_per_byte": 1.4210770796494847, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.9850155711174011, + "step": 7640, + "unclipped_grad_norm": 0.9340207576751709 + }, + { + "bits_per_byte": 1.5566732750779235, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 1.0790036916732788, + "step": 7650, + "unclipped_grad_norm": 0.9742102026939392 + }, + { + "bits_per_byte": 1.4584527792916175, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 1.0109224319458008, + "step": 7660, + "unclipped_grad_norm": 0.9492388963699341 + }, + { + "bits_per_byte": 1.495809302868179, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 1.0368160009384155, + "step": 7670, + "unclipped_grad_norm": 0.9652732014656067 + }, + { + "bits_per_byte": 1.4521128108502108, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 1.0065279006958008, + "step": 7680, + "unclipped_grad_norm": 0.963537335395813 + }, + { + "bits_per_byte": 1.4431084871816378, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 1.00028657913208, + "step": 7690, + "unclipped_grad_norm": 0.9126168489456177 + }, + { + "bits_per_byte": 1.392462520207605, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 0.9651814699172974, + "step": 7700, + "unclipped_grad_norm": 0.933846652507782 + }, + { + "bits_per_byte": 1.4605151952407642, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 1.0123519897460938, + "step": 7710, + "unclipped_grad_norm": 0.912322461605072 + }, + { + "bits_per_byte": 1.4861072455850552, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 1.0300910472869873, + "step": 7720, + "unclipped_grad_norm": 0.9802721738815308 + }, + { + "bits_per_byte": 1.4782785953177804, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 1.0246646404266357, + "step": 7730, + "unclipped_grad_norm": 0.942146360874176 + }, + { + "bits_per_byte": 1.4248471973303738, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 0.9876288175582886, + "step": 7740, + "unclipped_grad_norm": 0.9768857359886169 + }, + { + "bits_per_byte": 1.4418222289358347, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9993950128555298, + "step": 7750, + "unclipped_grad_norm": 0.9538165926933289 + }, + { + "bits_per_byte": 1.4568244475532484, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 1.009793758392334, + "step": 7760, + "unclipped_grad_norm": 0.9020839333534241 + }, + { + "bits_per_byte": 1.3845216512606422, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 0.9596772789955139, + "step": 7770, + "unclipped_grad_norm": 0.9030863046646118 + }, + { + "bits_per_byte": 1.4714175194442594, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 1.0199089050292969, + "step": 7780, + "unclipped_grad_norm": 0.9440871477127075 + }, + { + "bits_per_byte": 1.4171394508666553, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 0.9822862148284912, + "step": 7790, + "unclipped_grad_norm": 0.9084553122520447 + }, + { + "bits_per_byte": 1.4521059315441764, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 1.0065231323242188, + "step": 7800, + "unclipped_grad_norm": 0.9615890383720398 + }, + { + "bits_per_byte": 1.4126611946208385, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9791821241378784, + "step": 7810, + "unclipped_grad_norm": 0.9781654477119446 + }, + { + "bits_per_byte": 1.3957367259147313, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 0.9674509763717651, + "step": 7820, + "unclipped_grad_norm": 0.9221798777580261 + }, + { + "bits_per_byte": 1.4359687134223935, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 0.9953376650810242, + "step": 7830, + "unclipped_grad_norm": 0.9617613554000854 + }, + { + "bits_per_byte": 1.4187247869423087, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.9833850860595703, + "step": 7840, + "unclipped_grad_norm": 0.9338092803955078 + }, + { + "bits_per_byte": 1.3754349479023056, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.9533788561820984, + "step": 7850, + "unclipped_grad_norm": 0.9112406373023987 + }, + { + "bits_per_byte": 1.4409369482405183, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 0.9987813830375671, + "step": 7860, + "unclipped_grad_norm": 0.9407899379730225 + }, + { + "bits_per_byte": 1.468287779163858, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0177395343780518, + "step": 7870, + "unclipped_grad_norm": 0.9693545699119568 + }, + { + "bits_per_byte": 1.3773409456304928, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9546999931335449, + "step": 7880, + "unclipped_grad_norm": 0.9440149664878845 + }, + { + "bits_per_byte": 1.4559502597389127, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 1.0091878175735474, + "step": 7890, + "unclipped_grad_norm": 0.9741380214691162 + }, + { + "bits_per_byte": 1.4439381314893998, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 1.000861644744873, + "step": 7900, + "unclipped_grad_norm": 0.9557946920394897 + }, + { + "bits_per_byte": 1.44968682357714, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.0048463344573975, + "step": 7910, + "unclipped_grad_norm": 0.9438835382461548 + }, + { + "bits_per_byte": 1.4609620061677058, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 1.0126616954803467, + "step": 7920, + "unclipped_grad_norm": 0.9663562178611755 + }, + { + "bits_per_byte": 1.4159809757304433, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.9814832210540771, + "step": 7930, + "unclipped_grad_norm": 0.9176503419876099 + }, + { + "bits_per_byte": 1.4283059404218756, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 0.9900262355804443, + "step": 7940, + "unclipped_grad_norm": 0.957091212272644 + }, + { + "bits_per_byte": 1.4865411578131822, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 1.030391812324524, + "step": 7950, + "unclipped_grad_norm": 0.971011757850647 + }, + { + "bits_per_byte": 1.4189291023315336, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 0.9835267066955566, + "step": 7960, + "unclipped_grad_norm": 0.9379769563674927 + }, + { + "bits_per_byte": 1.4431967142815303, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 1.0003477334976196, + "step": 7970, + "unclipped_grad_norm": 0.9736126065254211 + }, + { + "bits_per_byte": 1.4850192833356972, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 1.029336929321289, + "step": 7980, + "unclipped_grad_norm": 0.9676840305328369 + }, + { + "bits_per_byte": 1.5188785397069635, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 1.0528063774108887, + "step": 7990, + "unclipped_grad_norm": 1.0497169494628906 + }, + { + "bits_per_byte": 1.4251737923843621, + "learning_rate": 3e-05, + "loss_nats": 0.9878551959991455, + "step": 8000, + "unclipped_grad_norm": 0.9529033899307251 + } + ], + "variant": "uniform_group_7_mlp_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073002.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073002.json new file mode 100644 index 0000000..ce78de9 --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073002.json @@ -0,0 +1,14029 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "e27039c108dac596ee7206bda41310dc6f7a3fca1ac25fbb1d3f6b92523b0fd7", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.00040311424527317286, + 0.00035970969474874437, + 0.00032544726855121553, + 0.00029308689408935606, + 0.00026978139067068696, + 0.00025443214690312743, + 0.00023969457834027708, + 0.0002233884297311306, + 0.0002095247182296589, + 0.0001984660339076072, + 0.0001901928335428238, + 0.00018121300672646612, + 0.0001734284160193056, + 0.0001677417167229578, + 0.00016109299031086266, + 0.00015624657680746168, + 0.0001518603676231578, + 0.00014820833166595548, + 0.00014440594532061368, + 0.0001414585713064298, + 0.00013923499500378966, + 0.00013687640603166074, + 0.00013471563579514623, + 0.00013265138841234148, + 0.00013098257477395236, + 0.0001293915556743741, + 0.00012844328011851758, + 0.0001269879430765286, + 0.0001256384130101651, + 0.00012456141121219844, + 0.0001231025526067242, + 0.00012163266364950687 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00029608183103846386, + "first_to_last_ratio": 2.343484699021421, + "imbalance_abs_log_ratio": 0.8516390092453431, + "last_quartile_mean": 0.0001263425492652459, + "mean": 0.0001858347804954974, + "normalized": [ + 2.16920774571012, + 1.9356424765570717, + 1.751272112160408, + 1.5771369240348272, + 1.451727119925344, + 1.369130935687747, + 1.289826251583108, + 1.2020808437231323, + 1.127478492836466, + 1.067970341065492, + 1.0234512239081746, + 0.9751296621832141, + 0.9332398141881068, + 0.9026389800429313, + 0.8668613586828855, + 0.8407822065969367, + 0.8171794710239251, + 0.7975274126338607, + 0.777066300159633, + 0.7612061150730458, + 0.7492407752334779, + 0.7365489154759011, + 0.7249215428669998, + 0.7138135717041164, + 0.7048334785593375, + 0.6962720074755282, + 0.6911692191098191, + 0.683337870004401, + 0.6760758813563923, + 0.6702804011180051, + 0.662430102042749, + 0.6545204472768429 + ], + "population_cv": 0.39438035124793563 + }, + "activation_output_rms_by_block": [ + 0.0036361459642648697, + 0.005152894649654627, + 0.006386642809957266, + 0.007284422870725393, + 0.0037505170330405235, + 0.0054016136564314365, + 0.006585344672203064, + 0.007922882214188576, + 0.0044892593286931515, + 0.006135277915745974, + 0.007316684816032648, + 0.0082859443500638, + 0.0044554853811860085, + 0.006464940495789051, + 0.007893112488090992, + 0.009196000173687935, + 0.004961193539202213, + 0.006907128728926182, + 0.008217990398406982, + 0.009676996618509293, + 0.005381465889513493, + 0.007502692751586437, + 0.0091153085231781, + 0.010543142445385456, + 0.006136673036962748, + 0.008469858206808567, + 0.010564134456217289, + 0.012301022186875343, + 0.005773921962827444, + 0.008156762458384037, + 0.010457510128617287, + 0.012218860909342766 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.005765057983808219, + "first_to_last_ratio": 0.622587017371887, + "imbalance_abs_log_ratio": 0.4738718734697682, + "last_quartile_mean": 0.009259842918254435, + "mean": 0.007398182220640592, + "normalized": [ + 0.49149180917985336, + 0.6965082091758006, + 0.8632718983507628, + 0.9846233376628902, + 0.5069511565390686, + 0.730127144119427, + 0.8901300989627224, + 1.0709228264321597, + 0.6068057253534959, + 0.8292953231982888, + 0.9889841312125875, + 1.119997332175246, + 0.6022405569783624, + 0.8738552664669654, + 1.0668989020126547, + 1.2430080659586231, + 0.6705962885532487, + 0.9336251153230055, + 1.1108121094231989, + 1.3080235563150786, + 0.7274038039370601, + 1.01412651484229, + 1.232101109614035, + 1.4250990487866828, + 0.8294838994154143, + 1.1448566626512724, + 1.427936504016329, + 1.6627087330393202, + 0.7804514393709395, + 1.1025360304896303, + 1.4135242707919937, + 1.6516031296515918 + ], + "population_cv": 0.30590027610070303 + }, + "bits_per_byte": 8.038491115106986, + "branch_output_rms_by_sublayer": [ + 0.0023513585329055786, + 0.0027412420604377985, + 0.0023208872880786657, + 0.002683233004063368, + 0.002778682392090559, + 0.002711885841563344, + 0.002497476991266012, + 0.0027249811682850122, + 0.002628475893288851, + 0.0027012599166482687, + 0.002920303260907531, + 0.002700963756069541, + 0.002932395786046982, + 0.0026683746837079525, + 0.0032228652853518724, + 0.002738022478297353, + 0.003561305580660701, + 0.0027370047755539417, + 0.003354643704369664, + 0.002654933836311102, + 0.0034162327647209167, + 0.002663377905264497, + 0.0036349669098854065, + 0.002684312639757991, + 0.0035210708156228065, + 0.0027356327045708895, + 0.0038089773152023554, + 0.002652956871315837, + 0.003915763925760984, + 0.002679797587916255, + 0.0037244565319269896, + 0.002698024269193411, + 0.004162859171628952, + 0.0026484597474336624, + 0.004167158622294664, + 0.0026680538430809975, + 0.004440344404429197, + 0.002654345240443945, + 0.0041700867004692554, + 0.002591830212622881, + 0.004737259820103645, + 0.002620960585772991, + 0.004552462603896856, + 0.002651234855875373, + 0.004706811159849167, + 0.0026270023081451654, + 0.005186605267226696, + 0.002615687670186162, + 0.005580419674515724, + 0.00258000404573977, + 0.004905648995190859, + 0.002626793459057808, + 0.005227713380008936, + 0.002573782578110695, + 0.005349088925868273, + 0.002592637902125716, + 0.005235649179667234, + 0.002581534907221794, + 0.004925078712403774, + 0.002551236655563116, + 0.005810525268316269, + 0.002563327783718705, + 0.005564783234149218, + 0.002589041367173195 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.010012756197717748, + 0.008904953530422711, + 0.008980942528705073, + 0.007787912846174446, + 0.007907551845183929, + 0.008289745083748344, + 0.007663833460373458, + 0.007350080301903749, + 0.00709564946584838, + 0.006310668475383808, + 0.006285078227083486, + 0.006362421438950662, + 0.005604026016423943, + 0.005959901601525332, + 0.006167718108848569, + 0.005523196426059091, + 0.0054641843609823, + 0.00557989785603791, + 0.005260407000687673, + 0.004763357766419076, + 0.004970214176820314, + 0.004845025235106639, + 0.004921304265254701, + 0.004820106373346748, + 0.0054804096177630225, + 0.0042302744374162005, + 0.004588429583035324, + 0.004469681656036737, + 0.004349111730454971, + 0.003928502929307311, + 0.004468266803351117, + 0.003904642007649288 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.008362221974278682, + "first_to_last_ratio": 1.8887369414995314, + "imbalance_abs_log_ratio": 0.6359083207487641, + "last_quartile_mean": 0.004427414845626747, + "mean": 0.00600782035481319, + "normalized": [ + 1.6666204390908572, + 1.4822269982304765, + 1.4948753455169401, + 1.2962958920592769, + 1.3162097696294732, + 1.3798257261648947, + 1.2756429133626719, + 1.2234187888150154, + 1.1810688480662828, + 1.0504089840715678, + 1.0461494944748422, + 1.0590232502297414, + 0.9327885465041001, + 0.9920239370590588, + 1.0266149359654666, + 0.9193344840336578, + 0.9095119424808776, + 0.9287724210274617, + 0.875593258455752, + 0.7928595538984271, + 0.8272907449435313, + 0.8064530809788657, + 0.8191497039874, + 0.8023053434820334, + 0.9122126318860999, + 0.7041279844573081, + 0.7637428072161453, + 0.7439772483303089, + 0.7239084182952745, + 0.6538982022256998, + 0.7437417465006834, + 0.6499265585598059 + ], + "population_cv": 0.2648157248292805 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.027978340163826942, + 0.014031016267836094, + 0.0140878576785326, + 0.014151837676763535, + 0.014187770895659924, + 0.014265215955674648, + 0.014337530359625816, + 0.014401593245565891, + 0.014477464370429516, + 0.009722241200506687, + 0.009757626801729202, + 0.009786579757928848, + 0.009825476445257664, + 0.009839320555329323, + 0.00987118761986494, + 0.00996283907443285, + 0.009999874979257584, + 0.007562503684312105, + 0.007595349568873644, + 0.007635751739144325, + 0.007658981718122959, + 0.007660485804080963, + 0.007689184043556452, + 0.007726665120571852, + 0.007755559403449297, + 0.006238248664885759, + 0.00625597033649683, + 0.006290202960371971, + 0.006315375678241253, + 0.006327399518340826, + 0.00635211868211627, + 0.006382675375789404, + 0.006408242508769035, + 0.005390914157032967, + 0.005408597644418478, + 0.0054517327807843685, + 0.005475745070725679, + 0.005509276874363422, + 0.005517482291907072, + 0.0055727302096784115, + 0.005585856735706329, + 0.0048178439028561115, + 0.004830927588045597, + 0.004884926602244377, + 0.004899685736745596, + 0.004938251338899136, + 0.004948681220412254, + 0.004975184798240662, + 0.004989070352166891, + 0.004412420094013214, + 0.004419817589223385, + 0.004455919377505779, + 0.004463871940970421, + 0.004530393984168768, + 0.004542885348200798, + 0.0045819394290447235, + 0.004592607729136944, + 0.004127359017729759, + 0.004137600306421518, + 0.004185030702501535, + 0.004193917382508516, + 0.004221425391733646, + 0.004230021964758635, + 0.004281661007553339 + ], + "loss_nats": 5.571857452392578, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0023513585329055786, + 0.0036361459642648697, + 0.004408807028084993, + 0.005152894649654627, + 0.005783648230135441, + 0.006386642809957266, + 0.006751363631337881, + 0.007284422870725393, + 0.002628475893288851, + 0.0037505170330405235, + 0.004723479505628347, + 0.0054016136564314365, + 0.006024772301316261, + 0.006585344672203064, + 0.007445912808179855, + 0.007922882214188576, + 0.003561305580660701, + 0.0044892593286931515, + 0.005543973762542009, + 0.006135277915745974, + 0.006855612155050039, + 0.007316684816032648, + 0.007876631803810596, + 0.0082859443500638, + 0.0035210708156228065, + 0.0044554853811860085, + 0.0058924350887537, + 0.006464940495789051, + 0.007390674669295549, + 0.007893112488090992, + 0.008790239691734314, + 0.009196000173687935, + 0.004162859171628952, + 0.004961193539202213, + 0.006355749908834696, + 0.006907128728926182, + 0.007784162648022175, + 0.008217990398406982, + 0.00937180407345295, + 0.009676996618509293, + 0.004737259820103645, + 0.005381465889513493, + 0.006999304983764887, + 0.007502692751586437, + 0.008728929795324802, + 0.0091153085231781, + 0.010217384435236454, + 0.010543142445385456, + 0.005580419674515724, + 0.006136673036962748, + 0.008008211851119995, + 0.008469858206808567, + 0.010209310799837112, + 0.010564134456217289, + 0.012040933594107628, + 0.012301022186875343, + 0.005235649179667234, + 0.005773921962827444, + 0.007745372597128153, + 0.008156762458384037, + 0.010180563665926456, + 0.010457510128617287, + 0.011906592175364494, + 0.012218860909342766 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.928340008016676e-05, + 6.579236651305109e-05, + 6.220132490852848e-05, + 5.8788795286091045e-05, + 5.369330392568372e-05, + 5.012521069147624e-05, + 4.67321697215084e-05, + 4.331160744186491e-05, + 3.891469896188937e-05, + 3.624656528700143e-05, + 3.3648047974566e-05, + 3.116594234597869e-05, + 2.860806489479728e-05, + 2.6935140340356156e-05, + 2.535484054533299e-05, + 2.395138108113315e-05, + 2.2848669686936773e-05, + 2.1811480110045522e-05, + 2.0979441615054384e-05, + 2.0198793208692223e-05, + 1.9655508367577568e-05, + 1.9088720364379697e-05, + 1.860289557953365e-05, + 1.8176880985265598e-05, + 1.7723919881973416e-05, + 1.735088153509423e-05, + 1.702679946902208e-05, + 1.675463681749534e-05, + 1.6367963326047175e-05, + 1.6178253645193763e-05, + 1.6008341845008545e-05, + 1.5869158232817426e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.624102232104633e-05, + "first_to_last_ratio": 3.3758128099861024, + "imbalance_abs_log_ratio": 1.2166361279174553, + "last_quartile_mean": 1.6659994344081497e-05, + "mean": 3.091860014592385e-05, + "normalized": [ + 2.24083237123207, + 2.1279219046960907, + 2.0117768791265536, + 1.9014054649508918, + 1.7366020347710462, + 1.6211992281314358, + 1.5114581352632595, + 1.4008269209295006, + 1.25861775042295, + 1.17232232752879, + 1.0882785060048066, + 1.0079997865002777, + 0.9252703796348563, + 0.8711629961651789, + 0.8200513744370036, + 0.7746592979013243, + 0.7389943134262186, + 0.7054485004852666, + 0.6785378870983656, + 0.6532893828750889, + 0.6357179262583417, + 0.6173863070866181, + 0.6016732805410068, + 0.5878946944388731, + 0.5732445776433397, + 0.5611794018230054, + 0.5506976185423067, + 0.5418950644084767, + 0.5293888872328214, + 0.5232531087707288, + 0.5177576529809033, + 0.5132560386926035 + ], + "population_cv": 0.5273540961183735 + }, + "activation_output_rms_by_block": [ + 0.005867196712642908, + 0.009899199940264225, + 0.014524669386446476, + 0.01919807307422161, + 0.006808924954384565, + 0.012262510135769844, + 0.017801832407712936, + 0.02365170046687126, + 0.00828799232840538, + 0.014784816652536392, + 0.021556563675403595, + 0.028364917263388634, + 0.009276608005166054, + 0.016745325177907944, + 0.025116128847002983, + 0.03308028355240822, + 0.010395217686891556, + 0.019817844033241272, + 0.029506469145417213, + 0.04013796150684357, + 0.013481441885232925, + 0.024827372282743454, + 0.03669177368283272, + 0.05052504315972328, + 0.015275463461875916, + 0.02962263859808445, + 0.044035423547029495, + 0.060379285365343094, + 0.019455187022686005, + 0.03753510117530823, + 0.055977120995521545, + 0.07986074686050415 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013751763384789228, + "first_to_last_ratio": 0.32154613940119064, + "imbalance_abs_log_ratio": 1.1346142326895097, + "last_quartile_mean": 0.04276762087829411, + "mean": 0.026085963530931622, + "normalized": [ + 0.22491776873358874, + 0.3794837759596718, + 0.5568001875500495, + 0.7359541483471506, + 0.26101872550388383, + 0.47008078199716424, + 0.6824295520694217, + 0.9066830304667906, + 0.3177184664303403, + 0.5667728790238163, + 0.8263663962361498, + 1.0873632185276472, + 0.35561684329452686, + 0.6419285666044903, + 0.9628215885996064, + 1.2681258069376287, + 0.3984985133696653, + 0.7597129394795833, + 1.1311243730916707, + 1.5386804270905956, + 0.5168082777255748, + 0.9517521656159725, + 1.406571531824967, + 1.9368670472843696, + 0.5855817226671701, + 1.135577704958423, + 1.688088825809105, + 2.314627377813663, + 0.7458105582190543, + 1.4389003162869836, + 2.1458713199972954, + 3.0614451624839805 + ], + "population_cv": 0.6546347225491245 + }, + "bits_per_byte": 5.118622295449385, + "branch_output_rms_by_sublayer": [ + 0.0030817349907010794, + 0.004966990556567907, + 0.0026547613088041544, + 0.004918823484331369, + 0.0032732647377997637, + 0.0055610197596251965, + 0.002746741520240903, + 0.0058339363895356655, + 0.0031059745233505964, + 0.0061521683819592, + 0.003051348263397813, + 0.006149548105895519, + 0.0035144330468028784, + 0.006374536082148552, + 0.0038315753918141127, + 0.006469159387052059, + 0.004718332085758448, + 0.006541167385876179, + 0.004048469010740519, + 0.00703774206340313, + 0.004866529256105423, + 0.006700266618281603, + 0.005042063072323799, + 0.006732675712555647, + 0.00518860574811697, + 0.006989923305809498, + 0.005118499975651503, + 0.00668067904189229, + 0.0065872641280293465, + 0.0069386838003993034, + 0.00584368035197258, + 0.007169649470597506, + 0.006155831273645163, + 0.007253303658217192, + 0.007110248785465956, + 0.007558699697256088, + 0.0075478320941329, + 0.007293254602700472, + 0.008073437958955765, + 0.007723369635641575, + 0.008931404910981655, + 0.007892200723290443, + 0.008837316185235977, + 0.007910354062914848, + 0.009106472134590149, + 0.008088595233857632, + 0.01139669120311737, + 0.008185946382582188, + 0.010470997542142868, + 0.00779342744499445, + 0.011258648708462715, + 0.008266177959740162, + 0.011105097830295563, + 0.008234836161136627, + 0.012935677543282509, + 0.008376535028219223, + 0.013468709774315357, + 0.009116943925619125, + 0.013869118876755238, + 0.009177922271192074, + 0.015341006219387054, + 0.008169475942850113, + 0.02037828043103218, + 0.008922291919589043 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00029021677449823154, + 0.00022704444231256654, + 0.00023188005847476576, + 0.00020123043684219116, + 0.00020768758363724898, + 0.0001907782297390571, + 0.00021041302301598886, + 0.000204073024488347, + 0.000209162340468506, + 0.00017997197990423242, + 0.00018986854887106271, + 0.00018948324906254967, + 0.0001857667248834063, + 0.00017157329461936358, + 0.00020658003247296997, + 0.00017892353711884711, + 0.00017636225458367538, + 0.00018333423263757547, + 0.00017382060269345646, + 0.00017214026504407782, + 0.00017751244438689144, + 0.0001671217969928599, + 0.00016197265333609264, + 0.00016758888174946927, + 0.00016281467377731164, + 0.00015525214216758539, + 0.00014554877311962573, + 0.00015848049378210734, + 0.00015086785706136313, + 0.0001471472273371441, + 0.00014728041250258974, + 0.00017235102128479145 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00022041544662604963, + "first_to_last_ratio": 1.4223303865978427, + "imbalance_abs_log_ratio": 0.35229664377407166, + "last_quartile_mean": 0.00015496782512906481, + "mean": 0.00018419528165206098, + "normalized": [ + 1.575592881073044, + 1.2326289809173627, + 1.2588816412397568, + 1.0924842104387291, + 1.1275401941596106, + 1.035738961540997, + 1.1423366610087893, + 1.10791667765863, + 1.1355466795485403, + 0.9770716073183339, + 1.0308002852631066, + 1.0287084846205643, + 1.0085313978580284, + 0.9314749709140763, + 1.1215272759439792, + 0.9713795897162446, + 0.9574743337715788, + 0.9953253470622988, + 0.9436756530050431, + 0.9345530650955832, + 0.9637187380413293, + 0.9073076980796264, + 0.8793528904939804, + 0.9098435108996946, + 0.8839242369132092, + 0.8428670961336119, + 0.7901873045508447, + 0.8603938839295132, + 0.8190647214641888, + 0.798865345612384, + 0.7995884106347401, + 0.9356972650925828 + ], + "population_cv": 0.15934631462522764 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6923106908798218, + "mean_weights": [ + 0.48025596141815186, + 0.5197440385818481 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927922964096069, + "mean_weights": [ + 0.5129902958869934, + 0.4870097041130066 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928635835647583, + "mean_weights": [ + 0.5082216262817383, + 0.4917783737182617 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6926431655883789, + "mean_weights": [ + 0.5150535702705383, + 0.48494642972946167 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6925322413444519, + "mean_weights": [ + 0.5161769986152649, + 0.4838230013847351 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928005218505859, + "mean_weights": [ + 0.5107828378677368, + 0.48921722173690796 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6924358606338501, + "mean_weights": [ + 0.5171115398406982, + 0.48288846015930176 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929624080657959, + "mean_weights": [ + 0.5073212385177612, + 0.49267876148223877 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0978729724884033, + "mean_weights": [ + 0.34929248690605164, + 0.32299426198005676, + 0.3277132511138916 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0983192920684814, + "mean_weights": [ + 0.3433057367801666, + 0.3268842399120331, + 0.3298100233078003 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0981415510177612, + "mean_weights": [ + 0.3443463146686554, + 0.3302139937877655, + 0.3254396915435791 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984491109848022, + "mean_weights": [ + 0.3400939702987671, + 0.32976919412612915, + 0.33013686537742615 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982602834701538, + "mean_weights": [ + 0.34348270297050476, + 0.33093971014022827, + 0.32557758688926697 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098495602607727, + "mean_weights": [ + 0.33823511004447937, + 0.3338121473789215, + 0.3279527425765991 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0980885028839111, + "mean_weights": [ + 0.34462738037109375, + 0.33185723423957825, + 0.323515385389328 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985525846481323, + "mean_weights": [ + 0.33589011430740356, + 0.33452272415161133, + 0.3295871913433075 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3860300779342651, + "mean_weights": [ + 0.2575128674507141, + 0.25002357363700867, + 0.24574732780456543, + 0.2467162311077118 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862011432647705, + "mean_weights": [ + 0.25315096974372864, + 0.25240254402160645, + 0.2467591017484665, + 0.24768739938735962 + ], + "sources": 4 + }, + { + "entropy_mean": 1.385886311531067, + "mean_weights": [ + 0.2581905722618103, + 0.25243544578552246, + 0.24489089846611023, + 0.24448305368423462 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861608505249023, + "mean_weights": [ + 0.2537953853607178, + 0.2533895969390869, + 0.247644305229187, + 0.2451707124710083 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3855739831924438, + "mean_weights": [ + 0.2592586278915405, + 0.2544729709625244, + 0.24628368020057678, + 0.23998472094535828 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861018419265747, + "mean_weights": [ + 0.251406192779541, + 0.2558833658695221, + 0.24896499514579773, + 0.24374544620513916 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3856627941131592, + "mean_weights": [ + 0.2574523091316223, + 0.25216349959373474, + 0.24600929021835327, + 0.24437491595745087 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862518072128296, + "mean_weights": [ + 0.25003281235694885, + 0.25259900093078613, + 0.2502778172492981, + 0.24709036946296692 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6089379787445068, + "mean_weights": [ + 0.2049759179353714, + 0.20672938227653503, + 0.20000213384628296, + 0.1957574188709259, + 0.1925351321697235 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093037128448486, + "mean_weights": [ + 0.2015664279460907, + 0.20420445501804352, + 0.2009846717119217, + 0.1967204511165619, + 0.1965239942073822 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085624694824219, + "mean_weights": [ + 0.20553870499134064, + 0.20927682518959045, + 0.2015843391418457, + 0.19414088129997253, + 0.18945923447608948 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093416213989258, + "mean_weights": [ + 0.20106898248195648, + 0.20404034852981567, + 0.20018696784973145, + 0.19849355518817902, + 0.19621016085147858 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6080294847488403, + "mean_weights": [ + 0.20543605089187622, + 0.21290913224220276, + 0.2034062147140503, + 0.19301313161849976, + 0.18523548543453217 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609381914138794, + "mean_weights": [ + 0.20110207796096802, + 0.2025112807750702, + 0.1994754821062088, + 0.19839243590831757, + 0.19851873815059662 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6076724529266357, + "mean_weights": [ + 0.20587486028671265, + 0.21558518707752228, + 0.20289722084999084, + 0.1925787776708603, + 0.18306395411491394 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093553304672241, + "mean_weights": [ + 0.20062199234962463, + 0.20333243906497955, + 0.20117762684822083, + 0.19841420650482178, + 0.19645372033119202 + ], + "sources": 5 + }, + { + "entropy_mean": 1.790160059928894, + "mean_weights": [ + 0.17127585411071777, + 0.18119347095489502, + 0.17149242758750916, + 0.16200974583625793, + 0.1543208658695221, + 0.15970763564109802 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791698694229126, + "mean_weights": [ + 0.16690848767757416, + 0.1692683845758438, + 0.1683572381734848, + 0.16567713022232056, + 0.16515520215034485, + 0.16463355720043182 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7896753549575806, + "mean_weights": [ + 0.1720653474330902, + 0.18266692757606506, + 0.17346777021884918, + 0.16240555047988892, + 0.1566377580165863, + 0.15275666117668152 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917170524597168, + "mean_weights": [ + 0.16645747423171997, + 0.16856339573860168, + 0.16665786504745483, + 0.16677311062812805, + 0.1645956039428711, + 0.16695253551006317 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7894043922424316, + "mean_weights": [ + 0.17307308316230774, + 0.18355457484722137, + 0.17263716459274292, + 0.1638392210006714, + 0.1555517464876175, + 0.1513442099094391 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791741132736206, + "mean_weights": [ + 0.1662009358406067, + 0.1674235761165619, + 0.16676823794841766, + 0.1668626070022583, + 0.1653302013874054, + 0.16741445660591125 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7883269786834717, + "mean_weights": [ + 0.17291778326034546, + 0.1867799162864685, + 0.1759256273508072, + 0.1628587543964386, + 0.15401706099510193, + 0.14750084280967712 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917324304580688, + "mean_weights": [ + 0.1666288673877716, + 0.1680488884449005, + 0.16745373606681824, + 0.1664467751979828, + 0.1653839647769928, + 0.16603776812553406 + ], + "sources": 6 + }, + { + "entropy_mean": 1.941745638847351, + "mean_weights": [ + 0.15034958720207214, + 0.16411583125591278, + 0.1530914604663849, + 0.1417817771434784, + 0.13223649561405182, + 0.12712612748146057, + 0.1312987506389618 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458884000778198, + "mean_weights": [ + 0.14314880967140198, + 0.14330869913101196, + 0.1431296020746231, + 0.1423678696155548, + 0.14219951629638672, + 0.143891841173172, + 0.14195363223552704 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9410674571990967, + "mean_weights": [ + 0.15053144097328186, + 0.1655072271823883, + 0.15479665994644165, + 0.14239241182804108, + 0.1326075792312622, + 0.12793134152889252, + 0.12623336911201477 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459002017974854, + "mean_weights": [ + 0.14251236617565155, + 0.14261719584465027, + 0.1436205804347992, + 0.14209839701652527, + 0.14353320002555847, + 0.14268487691879272, + 0.14293336868286133 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9416589736938477, + "mean_weights": [ + 0.1504504680633545, + 0.16386832296848297, + 0.15372931957244873, + 0.14195366203784943, + 0.13413497805595398, + 0.12904158234596252, + 0.12682168185710907 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458904266357422, + "mean_weights": [ + 0.14262166619300842, + 0.1436314880847931, + 0.1429048478603363, + 0.14244344830513, + 0.14211009442806244, + 0.14198479056358337, + 0.14430366456508636 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9401452541351318, + "mean_weights": [ + 0.15069669485092163, + 0.16666433215141296, + 0.15620410442352295, + 0.1427108645439148, + 0.13369205594062805, + 0.12736594676971436, + 0.12266598641872406 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458465576171875, + "mean_weights": [ + 0.14223381876945496, + 0.14147613942623138, + 0.14203029870986938, + 0.14167660474777222, + 0.1428503394126892, + 0.14363214373588562, + 0.14610067009925842 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.07940673828125, + "mean_weights": [ + 0.12429625540971756, + 0.12343983352184296, + 0.12440137565135956, + 0.12428686022758484, + 0.12552641332149506, + 0.12572240829467773, + 0.12628617882728577, + 0.12604068219661713 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079404592514038, + "mean_weights": [ + 0.12455789744853973, + 0.12376029789447784, + 0.12392541766166687, + 0.12458688020706177, + 0.12515239417552948, + 0.12512771785259247, + 0.1266307234764099, + 0.12625865638256073 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0794081687927246, + "mean_weights": [ + 0.12450386583805084, + 0.1242513507604599, + 0.12410619854927063, + 0.12468918412923813, + 0.1250263750553131, + 0.12550246715545654, + 0.1268608421087265, + 0.12505970895290375 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079289674758911, + "mean_weights": [ + 0.12426267564296722, + 0.12249956279993057, + 0.12263091653585434, + 0.12315285950899124, + 0.1255517154932022, + 0.1262110471725464, + 0.12732245028018951, + 0.1283687949180603 + ], + "sources": 8 + }, + { + "entropy_mean": 2.190581798553467, + "mean_weights": [ + 0.11896128952503204, + 0.1322615146636963, + 0.1255320906639099, + 0.1168743222951889, + 0.10873264074325562, + 0.1038634330034256, + 0.09847979247570038, + 0.09484302997589111, + 0.10045188665390015 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197192668914795, + "mean_weights": [ + 0.11042845994234085, + 0.11004112660884857, + 0.11079689115285873, + 0.11040763556957245, + 0.11129189282655716, + 0.11132515221834183, + 0.11097458004951477, + 0.11236311495304108, + 0.11237115412950516 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1903457641601562, + "mean_weights": [ + 0.11926907300949097, + 0.13218426704406738, + 0.12624800205230713, + 0.11612105369567871, + 0.11070699244737625, + 0.10494911670684814, + 0.10015220940113068, + 0.09518154710531235, + 0.0951877236366272 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197110176086426, + "mean_weights": [ + 0.11036704480648041, + 0.1091790497303009, + 0.10961900651454926, + 0.10945796966552734, + 0.11106431484222412, + 0.11170250922441483, + 0.11177249252796173, + 0.11316000670194626, + 0.11367759108543396 + ], + "sources": 9 + }, + { + "entropy_mean": 2.189466953277588, + "mean_weights": [ + 0.11985911428928375, + 0.13407184183597565, + 0.12662780284881592, + 0.11625541001558304, + 0.10988743603229523, + 0.10406351089477539, + 0.1002032607793808, + 0.09441744536161423, + 0.09461415559053421 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1969714164733887, + "mean_weights": [ + 0.109768807888031, + 0.10752680897712708, + 0.10840100049972534, + 0.11001579463481903, + 0.11048678308725357, + 0.1114589273929596, + 0.11389660835266113, + 0.11433040350675583, + 0.11411485821008682 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1906027793884277, + "mean_weights": [ + 0.11944250017404556, + 0.13184568285942078, + 0.12487314641475677, + 0.11625705659389496, + 0.11063406616449356, + 0.10504259169101715, + 0.10130102187395096, + 0.09607821702957153, + 0.09452571719884872 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.02856963500380516, + 0.013709262944757938, + 0.015033171512186527, + 0.014952408149838448, + 0.015676075592637062, + 0.015783414244651794, + 0.01654042676091194, + 0.01675565168261528, + 0.017751825973391533, + 0.012115196324884892, + 0.012944136746227741, + 0.013099951669573784, + 0.014040441252291203, + 0.014190124347805977, + 0.015272513963282108, + 0.015522786416113377, + 0.016727523878216743, + 0.012802046723663807, + 0.01376014482229948, + 0.013935120776295662, + 0.015030023641884327, + 0.015211713500320911, + 0.016274208202958107, + 0.016450518742203712, + 0.017627116292715073, + 0.01436984445899725, + 0.015271615236997604, + 0.015504637733101845, + 0.016412612050771713, + 0.01673644594848156, + 0.017763817682862282, + 0.01788381114602089, + 0.01905091665685177, + 0.016091881319880486, + 0.01706043817102909, + 0.01734146475791931, + 0.01837068609893322, + 0.018590064719319344, + 0.019711896777153015, + 0.019785139709711075, + 0.021146487444639206, + 0.018274880945682526, + 0.019461458548903465, + 0.019498655572533607, + 0.020824171602725983, + 0.020760269835591316, + 0.022214392200112343, + 0.022032229229807854, + 0.02401469647884369, + 0.02150564268231392, + 0.022301122546195984, + 0.022986235097050667, + 0.023831279948353767, + 0.024493295699357986, + 0.025312449783086777, + 0.026194127276539803, + 0.02722339890897274, + 0.023568831384181976, + 0.02563568577170372, + 0.02509884163737297, + 0.027448825538158417, + 0.026545312255620956, + 0.029361043125391006, + 0.028476910665631294 + ], + "loss_nats": 3.5479586124420166, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1936135292053223, + "mean_weights": [ + 0.10073192417621613, + 0.0949646532535553, + 0.1022535115480423, + 0.11061976104974747, + 0.11558850109577179, + 0.1188843697309494, + 0.11974476277828217, + 0.11931920051574707, + 0.11789333820343018 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.0030817349907010794, + 0.005867196712642908, + 0.006682469509541988, + 0.009899199940264225, + 0.010634200647473335, + 0.014524669386446476, + 0.014864049851894379, + 0.01919807307422161, + 0.0031059745233505964, + 0.006808924954384565, + 0.007717950735241175, + 0.012262510135769844, + 0.012947632931172848, + 0.017801832407712936, + 0.01853417046368122, + 0.02365170046687126, + 0.004718332085758448, + 0.00828799232840538, + 0.009585161693394184, + 0.014784816652536392, + 0.016341527923941612, + 0.021556563675403595, + 0.023111101239919662, + 0.028364917263388634, + 0.00518860574811697, + 0.009276608005166054, + 0.01194683089852333, + 0.016745325177907944, + 0.019882459193468094, + 0.025116128847002983, + 0.027493691071867943, + 0.03308028355240822, + 0.006155831273645163, + 0.010395217686891556, + 0.014268205501139164, + 0.019817844033241272, + 0.024034317582845688, + 0.029506469145417213, + 0.03419799357652664, + 0.04013796150684357, + 0.008931404910981655, + 0.013481441885232925, + 0.01909547857940197, + 0.024827372282743454, + 0.030591756105422974, + 0.03669177368283272, + 0.04402819275856018, + 0.05052504315972328, + 0.010470997542142868, + 0.015275463461875916, + 0.023567812517285347, + 0.02962263859808445, + 0.03773915022611618, + 0.044035423547029495, + 0.05374709144234657, + 0.060379285365343094, + 0.013468709774315357, + 0.019455187022686005, + 0.030445825308561325, + 0.03753510117530823, + 0.04988862946629524, + 0.055977120995521545, + 0.0727088525891304, + 0.07986074686050415 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00014720449689775705, + 0.00015340348181780428, + 0.00015196509775705636, + 0.000144509831443429, + 0.00013877956371288747, + 0.00012416215031407773, + 0.0001118314394261688, + 0.00010269604536006227, + 9.509945084573701e-05, + 8.64348403410986e-05, + 7.960450602695346e-05, + 7.462542271241546e-05, + 7.089111750246957e-05, + 6.518122245324776e-05, + 5.945420343778096e-05, + 5.5216671171365306e-05, + 5.822240200359374e-05, + 5.333530862117186e-05, + 4.839406756218523e-05, + 4.397402153699659e-05, + 4.4552143663167953e-05, + 4.008696123491973e-05, + 3.62531682185363e-05, + 3.390324127394706e-05, + 2.848051008186303e-05, + 2.699211472645402e-05, + 2.583425521152094e-05, + 2.5249151804018766e-05, + 2.516436688893009e-05, + 2.4853252398315817e-05, + 2.470965227985289e-05, + 2.4676644898136146e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00013431901334115537, + "first_to_last_ratio": 5.2172867378125805, + "imbalance_abs_log_ratio": 1.651977484732799, + "last_quartile_mean": 2.5744993536136462e-05, + "mean": 6.955440011324754e-05, + "normalized": [ + 2.1163937386862752, + 2.2055180055903696, + 2.184838019012871, + 2.0776518984872276, + 1.9952664890636458, + 1.7851084922293714, + 1.6078269562254344, + 1.4764852431025781, + 1.3672672137333852, + 1.2426940668076578, + 1.1444927408955072, + 1.0729072868274523, + 1.019218300884568, + 0.937125794300872, + 0.8547870924194361, + 0.7938630925069049, + 0.8370771929424567, + 0.7668142998046426, + 0.6957729127616752, + 0.6322248695323183, + 0.640536667567095, + 0.5763396876351558, + 0.5212203420561371, + 0.4874348886446617, + 0.4094710045013895, + 0.3880719937560502, + 0.37142517467562014, + 0.363013005114104, + 0.36179403241143343, + 0.35732106606986885, + 0.35525649332926407, + 0.3547819384245708 + ], + "population_cv": 0.6134400078428953 + }, + "activation_output_rms_by_block": [ + 0.0207632165402174, + 0.03715304285287857, + 0.04801575094461441, + 0.05743842571973801, + 0.016815170645713806, + 0.02669147215783596, + 0.03357178345322609, + 0.0404425784945488, + 0.02355853095650673, + 0.0294625386595726, + 0.03687986731529236, + 0.044238943606615067, + 0.0264891404658556, + 0.03747721388936043, + 0.040315404534339905, + 0.04906824976205826, + 0.022600771859288216, + 0.038448579609394073, + 0.049529142677783966, + 0.06041477620601654, + 0.032174400985240936, + 0.04235849902033806, + 0.05741666257381439, + 0.08149656653404236, + 0.024131158366799355, + 0.044305022805929184, + 0.06695609539747238, + 0.09163036942481995, + 0.028238570317626, + 0.05507355183362961, + 0.0897025391459465, + 0.1233445405960083 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.03511143010109663, + "first_to_last_ratio": 0.536685484875199, + "imbalance_abs_log_ratio": 0.6223430453016575, + "last_quartile_mean": 0.06542273098602891, + "mean": 0.04613133054226637, + "normalized": [ + 0.4500892624632572, + 0.8053754881151385, + 1.0408490364400285, + 1.2451066345704438, + 0.36450651754575863, + 0.578597492074952, + 0.7277436626820682, + 0.8766835471501211, + 0.5106839685649641, + 0.6386665702732871, + 0.7994537959728335, + 0.9589782710924095, + 0.5742114990935664, + 0.8124026220102869, + 0.8739267664825361, + 1.0636643076466443, + 0.4899223931679355, + 0.8334591514581783, + 1.0736551947575945, + 1.3096257033094552, + 0.6974522652400447, + 0.9182154193781258, + 1.2446348695970995, + 1.7666207667557672, + 0.5230969513157646, + 0.9604106858642656, + 1.4514234601606815, + 1.986293660896395, + 0.6121343127476738, + 1.1938426918592824, + 1.944503618072741, + 2.6737694132406995 + ], + "population_cv": 0.5044573961884824 + }, + "bits_per_byte": 3.05721416463664, + "branch_output_rms_by_sublayer": [ + 0.011415470391511917, + 0.016195835545659065, + 0.01380860898643732, + 0.01344599761068821, + 0.010809495113790035, + 0.011634104885160923, + 0.011394373141229153, + 0.010538598522543907, + 0.014016781933605671, + 0.008524156175553799, + 0.014518543146550655, + 0.007663691882044077, + 0.015515416860580444, + 0.008351503871381283, + 0.012641542591154575, + 0.008738826029002666, + 0.02041158638894558, + 0.0090224863961339, + 0.018009882420301437, + 0.008264589123427868, + 0.014741327613592148, + 0.00873782392591238, + 0.016969164833426476, + 0.008828580379486084, + 0.023472262546420097, + 0.009660127572715282, + 0.017038796097040176, + 0.009518579579889774, + 0.020082581788301468, + 0.009755227714776993, + 0.013996563851833344, + 0.009853951632976532, + 0.017843423411250114, + 0.011314678005874157, + 0.02033001184463501, + 0.011529330164194107, + 0.028434108942747116, + 0.012490257620811462, + 0.024664754047989845, + 0.013800890184938908, + 0.026457639411091805, + 0.016111411154270172, + 0.02041458524763584, + 0.01804531179368496, + 0.015643378719687462, + 0.021163320168852806, + 0.017674533650279045, + 0.025270802900195122, + 0.014530857093632221, + 0.01729007065296173, + 0.014537091366946697, + 0.01891391910612583, + 0.012820921838283539, + 0.022136952728033066, + 0.015762539580464363, + 0.02368040196597576, + 0.006902170367538929, + 0.02691379003226757, + 0.005232888739556074, + 0.02736916020512581, + 0.010495252907276154, + 0.031240658834576607, + 0.007725848816335201, + 0.031242473050951958 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0005524629869673688, + 0.0004338901684238589, + 0.0003649233835635166, + 0.00037225283519649925, + 0.0003880921020980952, + 0.0003220203107157402, + 0.00027234437562079087, + 0.000247125973981451, + 0.0002816674071781532, + 0.0002181649755319825, + 0.00020388941074850873, + 0.00019544308951821647, + 0.0002520557844052774, + 0.00020211878381793617, + 0.00017277572869134819, + 0.00016087962345884917, + 0.00018315709567718978, + 0.00015808771964392651, + 0.00016125565940260976, + 0.00016165531916595836, + 0.000189977881489246, + 0.0001469711146741148, + 0.00013468467057933776, + 0.0001301469553137439, + 9.214152131052285e-05, + 8.692135464132448e-05, + 8.841260918553467e-05, + 8.74385267079575e-05, + 8.524937669747115e-05, + 8.3436873067033e-05, + 9.023426032781702e-05, + 8.666425702512732e-05 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0003691390170709151, + "first_to_last_ratio": 4.215727743223085, + "imbalance_abs_log_ratio": 1.438822232119502, + "last_quartile_mean": 8.75623473703485e-05, + "mean": 0.00020645444171332836, + "normalized": [ + 2.6759559270441344, + 2.1016267067110896, + 1.767573419758291, + 1.8030749646616453, + 1.8797953624896055, + 1.5597645080597513, + 1.3191499943554315, + 1.1970000351195977, + 1.3643078097068095, + 1.0567221209748288, + 0.9875758015011307, + 0.9466644936106452, + 1.2208784771764254, + 0.9789994448198288, + 0.8368709690017488, + 0.7792499988071849, + 0.8871550263448048, + 0.765726899997815, + 0.7810713979528752, + 0.7830072234068198, + 0.9201927549373782, + 0.7118815824664652, + 0.6523699343139041, + 0.6303906771570409, + 0.44630437856341043, + 0.4210195427135389, + 0.4282427079398936, + 0.4235245605874151, + 0.4129210105144838, + 0.4041418163474971, + 0.43706620976027005, + 0.4197742431982388 + ], + "population_cv": 0.5565468806242639 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6509300470352173, + "mean_weights": [ + 0.3797916769981384, + 0.6202082633972168 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6688297390937805, + "mean_weights": [ + 0.42257827520370483, + 0.5774217247962952 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6901689171791077, + "mean_weights": [ + 0.5026094913482666, + 0.497390478849411 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6681818962097168, + "mean_weights": [ + 0.4313230514526367, + 0.5686769485473633 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6888339519500732, + "mean_weights": [ + 0.5125775933265686, + 0.4874223470687866 + ], + "sources": 2 + }, + { + "entropy_mean": 0.678235650062561, + "mean_weights": [ + 0.46284395456314087, + 0.5371560454368591 + ], + "sources": 2 + }, + { + "entropy_mean": 0.674854040145874, + "mean_weights": [ + 0.4335355758666992, + 0.5664644241333008 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6759839057922363, + "mean_weights": [ + 0.4481176435947418, + 0.5518823266029358 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0920528173446655, + "mean_weights": [ + 0.3036450743675232, + 0.3394566774368286, + 0.3568982481956482 + ], + "sources": 3 + }, + { + "entropy_mean": 1.083463191986084, + "mean_weights": [ + 0.29528260231018066, + 0.31751662492752075, + 0.3872007727622986 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0810730457305908, + "mean_weights": [ + 0.2716212272644043, + 0.3391823172569275, + 0.3891964554786682 + ], + "sources": 3 + }, + { + "entropy_mean": 1.075951337814331, + "mean_weights": [ + 0.2946985065937042, + 0.2994706630706787, + 0.40583083033561707 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0884246826171875, + "mean_weights": [ + 0.29286426305770874, + 0.3257332444190979, + 0.38140249252319336 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0633609294891357, + "mean_weights": [ + 0.2971906363964081, + 0.2828219532966614, + 0.41998744010925293 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0838993787765503, + "mean_weights": [ + 0.28456783294677734, + 0.33104029297828674, + 0.3843918442726135 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0559102296829224, + "mean_weights": [ + 0.2804597020149231, + 0.2869969606399536, + 0.4325433373451233 + ], + "sources": 3 + }, + { + "entropy_mean": 1.379939079284668, + "mean_weights": [ + 0.2295229434967041, + 0.236092671751976, + 0.259260892868042, + 0.2751235067844391 + ], + "sources": 4 + }, + { + "entropy_mean": 1.359201192855835, + "mean_weights": [ + 0.23423539102077484, + 0.21753183007240295, + 0.2471584975719452, + 0.3010742664337158 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3836932182312012, + "mean_weights": [ + 0.2409345507621765, + 0.2401750534772873, + 0.25370562076568604, + 0.26518476009368896 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3563683032989502, + "mean_weights": [ + 0.24123775959014893, + 0.21958640217781067, + 0.23174384236335754, + 0.30743199586868286 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3794562816619873, + "mean_weights": [ + 0.2466629445552826, + 0.23648479580879211, + 0.24682480096817017, + 0.27002739906311035 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3506522178649902, + "mean_weights": [ + 0.2452215552330017, + 0.21543733775615692, + 0.22423096001148224, + 0.31511014699935913 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3806893825531006, + "mean_weights": [ + 0.263094961643219, + 0.22825336456298828, + 0.2420026957988739, + 0.2666490077972412 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3482398986816406, + "mean_weights": [ + 0.24399645626544952, + 0.2222413867712021, + 0.21783240139484406, + 0.3159297704696655 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6075217723846436, + "mean_weights": [ + 0.20089995861053467, + 0.1914183348417282, + 0.18963690102100372, + 0.20569932460784912, + 0.21234549582004547 + ], + "sources": 5 + }, + { + "entropy_mean": 1.581428050994873, + "mean_weights": [ + 0.19768138229846954, + 0.17632916569709778, + 0.17089398205280304, + 0.21458101272583008, + 0.24051448702812195 + ], + "sources": 5 + }, + { + "entropy_mean": 1.604072093963623, + "mean_weights": [ + 0.18868398666381836, + 0.20233267545700073, + 0.18790358304977417, + 0.20127445459365845, + 0.2198053002357483 + ], + "sources": 5 + }, + { + "entropy_mean": 1.585512638092041, + "mean_weights": [ + 0.20036093890666962, + 0.1812903732061386, + 0.17134693264961243, + 0.20505717396736145, + 0.2419445812702179 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6031458377838135, + "mean_weights": [ + 0.19205427169799805, + 0.1957930028438568, + 0.1873190701007843, + 0.19722896814346313, + 0.2276046872138977 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5870397090911865, + "mean_weights": [ + 0.20018050074577332, + 0.18308719992637634, + 0.17480550706386566, + 0.1946839541196823, + 0.24724282324314117 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6008273363113403, + "mean_weights": [ + 0.18832530081272125, + 0.19471973180770874, + 0.18355414271354675, + 0.19772151112556458, + 0.23567935824394226 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5872669219970703, + "mean_weights": [ + 0.1991586834192276, + 0.1815168708562851, + 0.17452490329742432, + 0.1933296024799347, + 0.2514699399471283 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7806007862091064, + "mean_weights": [ + 0.15636348724365234, + 0.16336596012115479, + 0.14888212084770203, + 0.15434490144252777, + 0.17244650423526764, + 0.20459705591201782 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7769626379013062, + "mean_weights": [ + 0.15559840202331543, + 0.15784364938735962, + 0.146101713180542, + 0.15522915124893188, + 0.18186213076114655, + 0.20336496829986572 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7779756784439087, + "mean_weights": [ + 0.1444942057132721, + 0.17399287223815918, + 0.15615272521972656, + 0.15364304184913635, + 0.16771897673606873, + 0.20399817824363708 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7767877578735352, + "mean_weights": [ + 0.16275008022785187, + 0.15631918609142303, + 0.1448238492012024, + 0.15303537249565125, + 0.17191651463508606, + 0.2111549973487854 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7772514820098877, + "mean_weights": [ + 0.14449408650398254, + 0.17503975331783295, + 0.1530744880437851, + 0.15281377732753754, + 0.1632731407880783, + 0.21130473911762238 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7782056331634521, + "mean_weights": [ + 0.16952891647815704, + 0.164504736661911, + 0.15233942866325378, + 0.1473768949508667, + 0.15854719281196594, + 0.20770281553268433 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7776892185211182, + "mean_weights": [ + 0.1409088671207428, + 0.19116732478141785, + 0.17183402180671692, + 0.15369077026844025, + 0.1531701385974884, + 0.18922889232635498 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7802293300628662, + "mean_weights": [ + 0.18974950909614563, + 0.16210713982582092, + 0.14752522110939026, + 0.15355613827705383, + 0.15477655827999115, + 0.192285418510437 + ], + "sources": 6 + }, + { + "entropy_mean": 1.923887014389038, + "mean_weights": [ + 0.11810784041881561, + 0.15446443855762482, + 0.13537141680717468, + 0.12457357347011566, + 0.1259126365184784, + 0.1416103094816208, + 0.19995978474617004 + ], + "sources": 7 + }, + { + "entropy_mean": 1.935656189918518, + "mean_weights": [ + 0.1431342363357544, + 0.1474619209766388, + 0.13855193555355072, + 0.13083775341510773, + 0.12335601449012756, + 0.13681572675704956, + 0.17984242737293243 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9216810464859009, + "mean_weights": [ + 0.11365681886672974, + 0.1653227061033249, + 0.1454201489686966, + 0.12679800391197205, + 0.12142163515090942, + 0.1342659443616867, + 0.1931147575378418 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9345595836639404, + "mean_weights": [ + 0.14107602834701538, + 0.1496107280254364, + 0.14346632361412048, + 0.13347145915031433, + 0.1228245198726654, + 0.12806901335716248, + 0.18148189783096313 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9258065223693848, + "mean_weights": [ + 0.11847450584173203, + 0.17573018372058868, + 0.1523495614528656, + 0.1305684745311737, + 0.12221411615610123, + 0.1318148672580719, + 0.16884827613830566 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9327702522277832, + "mean_weights": [ + 0.1419830024242401, + 0.1573026478290558, + 0.14094775915145874, + 0.12695229053497314, + 0.12230762839317322, + 0.1348869502544403, + 0.17561975121498108 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9022786617279053, + "mean_weights": [ + 0.11001560091972351, + 0.19611316919326782, + 0.16768787801265717, + 0.13166987895965576, + 0.11657917499542236, + 0.12608671188354492, + 0.15184757113456726 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9218614101409912, + "mean_weights": [ + 0.1404634714126587, + 0.16902993619441986, + 0.14651942253112793, + 0.12828423082828522, + 0.12441351264715195, + 0.14463797211647034, + 0.14665144681930542 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.055537223815918, + "mean_weights": [ + 0.1172759085893631, + 0.15376049280166626, + 0.14282414317131042, + 0.12410750240087509, + 0.1104062870144844, + 0.12431743741035461, + 0.12324342131614685, + 0.10406480729579926 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0585291385650635, + "mean_weights": [ + 0.12015682458877563, + 0.15133792161941528, + 0.13978728652000427, + 0.11943931132555008, + 0.1080939993262291, + 0.12182489037513733, + 0.1284182071685791, + 0.11094155162572861 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0481176376342773, + "mean_weights": [ + 0.12562295794487, + 0.1536586880683899, + 0.13427801430225372, + 0.1104816198348999, + 0.10579568147659302, + 0.12777721881866455, + 0.12624382972717285, + 0.11614200472831726 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0622897148132324, + "mean_weights": [ + 0.1263139843940735, + 0.13862255215644836, + 0.13181570172309875, + 0.11378266662359238, + 0.10710503160953522, + 0.12409691512584686, + 0.1342143714427948, + 0.12404878437519073 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1263952255249023, + "mean_weights": [ + 0.08320179581642151, + 0.17807114124298096, + 0.15383821725845337, + 0.10982780158519745, + 0.08595264703035355, + 0.09522677958011627, + 0.1013219803571701, + 0.07260454446077347, + 0.11995510756969452 + ], + "sources": 9 + }, + { + "entropy_mean": 2.191333055496216, + "mean_weights": [ + 0.10514402389526367, + 0.12333709001541138, + 0.118657186627388, + 0.11529222875833511, + 0.10812272876501083, + 0.11375586688518524, + 0.10901863873004913, + 0.10086873173713684, + 0.10580351948738098 + ], + "sources": 9 + }, + { + "entropy_mean": 2.119330406188965, + "mean_weights": [ + 0.08629368990659714, + 0.1845894604921341, + 0.16208021342754364, + 0.11681709438562393, + 0.09349106252193451, + 0.10148751735687256, + 0.10388819873332977, + 0.07384954392910004, + 0.07750321924686432 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1750831604003906, + "mean_weights": [ + 0.11003721505403519, + 0.12489983439445496, + 0.11925545334815979, + 0.10618631541728973, + 0.10310288518667221, + 0.12184539437294006, + 0.10719410330057144, + 0.10030166804790497, + 0.10717712342739105 + ], + "sources": 9 + }, + { + "entropy_mean": 2.112546443939209, + "mean_weights": [ + 0.08790051192045212, + 0.1874711960554123, + 0.16362935304641724, + 0.1165957897901535, + 0.09222754836082458, + 0.10501022636890411, + 0.10432598739862442, + 0.07483046501874924, + 0.0680089071393013 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1820497512817383, + "mean_weights": [ + 0.10189752280712128, + 0.12980656325817108, + 0.12679147720336914, + 0.1196061372756958, + 0.11064215004444122, + 0.11909371614456177, + 0.10348135232925415, + 0.09426192939281464, + 0.09441913664340973 + ], + "sources": 9 + }, + { + "entropy_mean": 2.109631061553955, + "mean_weights": [ + 0.08837950229644775, + 0.19031818211078644, + 0.1584751158952713, + 0.11744856089353561, + 0.09628002345561981, + 0.11098644137382507, + 0.10107049345970154, + 0.07331392168998718, + 0.06372775137424469 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.04403940215706825, + 0.017738768830895424, + 0.023364512249827385, + 0.027133116498589516, + 0.030052900314331055, + 0.03147675842046738, + 0.03452444076538086, + 0.036009032279253006, + 0.038523897528648376, + 0.025462795048952103, + 0.026331108063459396, + 0.02717023342847824, + 0.02767380140721798, + 0.028582124039530754, + 0.02971894107758999, + 0.03044123761355877, + 0.031637098640203476, + 0.023042550310492516, + 0.024380329996347427, + 0.024008087813854218, + 0.025297485291957855, + 0.024779317900538445, + 0.026564402505755424, + 0.025300834327936172, + 0.027964461594820023, + 0.020810944959521294, + 0.02211145870387554, + 0.02145211398601532, + 0.022944999858736992, + 0.021871959790587425, + 0.02329310216009617, + 0.022764835506677628, + 0.024271579459309578, + 0.019149763509631157, + 0.020352035760879517, + 0.020554374903440475, + 0.0214078426361084, + 0.021508626639842987, + 0.022155046463012695, + 0.02263982780277729, + 0.02316490188241005, + 0.018644461408257484, + 0.019499700516462326, + 0.019409380853176117, + 0.020626679062843323, + 0.02070835418999195, + 0.022632429376244545, + 0.022218424826860428, + 0.024623846635222435, + 0.02211650088429451, + 0.02275349758565426, + 0.023270584642887115, + 0.024167504161596298, + 0.02471146360039711, + 0.026470327749848366, + 0.026444725692272186, + 0.029257889837026596, + 0.02242361381649971, + 0.026425527408719063, + 0.02482469007372856, + 0.02938293106853962, + 0.02654915116727352, + 0.0303161833435297, + 0.028063541278243065 + ], + "loss_nats": 2.1190993785858154, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1659064292907715, + "mean_weights": [ + 0.12335751950740814, + 0.09263164550065994, + 0.09668299555778503, + 0.0968317836523056, + 0.09424621611833572, + 0.1124739944934845, + 0.13164183497428894, + 0.12291045486927032, + 0.1292235553264618 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.011415470391511917, + 0.0207632165402174, + 0.028523780405521393, + 0.03715304285287857, + 0.04056267440319061, + 0.04801575094461441, + 0.0512080043554306, + 0.05743842571973801, + 0.014016781933605671, + 0.016815170645713806, + 0.02324935607612133, + 0.02669147215783596, + 0.029545176774263382, + 0.03357178345322609, + 0.03596904128789902, + 0.0404425784945488, + 0.02041158638894558, + 0.02355853095650673, + 0.02596368081867695, + 0.0294625386595726, + 0.032832056283950806, + 0.03687986731529236, + 0.03985746577382088, + 0.044238943606615067, + 0.023472262546420097, + 0.0264891404658556, + 0.03353423997759819, + 0.03747721388936043, + 0.03523857891559601, + 0.040315404534339905, + 0.04393065348267555, + 0.04906824976205826, + 0.017843423411250114, + 0.022600771859288216, + 0.03249615058302879, + 0.038448579609394073, + 0.04308200255036354, + 0.049529142677783966, + 0.05269951745867729, + 0.06041477620601654, + 0.026457639411091805, + 0.032174400985240936, + 0.032601792365312576, + 0.04235849902033806, + 0.042124100029468536, + 0.05741666257381439, + 0.061759110540151596, + 0.08149656653404236, + 0.014530857093632221, + 0.024131158366799355, + 0.03063076362013817, + 0.044305022805929184, + 0.04961260035634041, + 0.06695609539747238, + 0.07227867096662521, + 0.09163036942481995, + 0.006902170367538929, + 0.028238570317626, + 0.029817141592502594, + 0.05507355183362961, + 0.06039740890264511, + 0.0897025391459465, + 0.09399373084306717, + 0.1233445405960083 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.154222112149e-05, + 7.623876444995403e-05, + 7.118327630450949e-05, + 6.8867695517838e-05, + 6.886413029860705e-05, + 6.499009759863839e-05, + 5.939946640864946e-05, + 5.699929897673428e-05, + 6.077714351704344e-05, + 5.829890869790688e-05, + 5.366237019188702e-05, + 4.9881105951499194e-05, + 5.357278132578358e-05, + 5.056819281890057e-05, + 4.9049616791307926e-05, + 4.906796311843209e-05, + 6.259966176003218e-05, + 6.433785165427253e-05, + 6.160033080959693e-05, + 5.752341166953556e-05, + 0.00012387374590616673, + 9.820309787755832e-05, + 7.636630471097305e-05, + 5.9136196796316653e-05, + 2.7780890377471223e-05, + 2.6253310352331027e-05, + 2.4885601305868477e-05, + 2.419860174995847e-05, + 3.810717316810042e-05, + 3.5119224776281044e-05, + 3.229025242035277e-05, + 3.166312671964988e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 6.726061883455259e-05, + "first_to_last_ratio": 2.2392385524029077, + "imbalance_abs_log_ratio": 0.8061358761139943, + "last_quartile_mean": 3.0037272608751664e-05, + "mean": 5.646568172323896e-05, + "normalized": [ + 1.267003584091081, + 1.350178765637346, + 1.26064671730003, + 1.2196380777865448, + 1.219574938210042, + 1.150966314675541, + 1.051956951477008, + 1.0094502932969232, + 1.0763554368286332, + 1.0324662152075539, + 0.9503537113907152, + 0.8833880054080738, + 0.9487671040325937, + 0.895556225934819, + 0.8686624387485455, + 0.8689873498549779, + 1.108632001768053, + 1.1394151224387628, + 1.090933978474305, + 1.0187322620398167, + 2.1937881935672334, + 1.7391643008737812, + 1.3524374873445266, + 1.0472944803211084, + 0.4919960147410697, + 0.4649427679100568, + 0.4407208156600823, + 0.4285541414086801, + 0.6748731619832204, + 0.621956978194552, + 0.5718562396646568, + 0.5607499237296667 + ], + "population_cv": 0.3708584212042403 + }, + "activation_output_rms_by_block": [ + 0.08216902613639832, + 0.18183350563049316, + 0.2180991768836975, + 0.23346203565597534, + 0.07513487339019775, + 0.08731291443109512, + 0.09794493764638901, + 0.10548584163188934, + 0.03381669893860817, + 0.05014139786362648, + 0.07702979445457458, + 0.11409901082515717, + 0.051963530480861664, + 0.08275293558835983, + 0.11449820548295975, + 0.15335743129253387, + 0.049225207418203354, + 0.07410992681980133, + 0.10182089358568192, + 0.1329580694437027, + 0.06232793629169464, + 0.08574102073907852, + 0.11530966311693192, + 0.15233835577964783, + 0.09091266989707947, + 0.12394227832555771, + 0.16820983588695526, + 0.227275088429451, + 0.06458000093698502, + 0.11642003804445267, + 0.19070100784301758, + 0.27311408519744873 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.13518028892576694, + "first_to_last_ratio": 0.8616006050857627, + "imbalance_abs_log_ratio": 0.1489634508586789, + "last_quartile_mean": 0.15689437557011843, + "mean": 0.11837773106526583, + "normalized": [ + 0.6941257058820939, + 1.536044862443274, + 1.8424003815671355, + 1.9721786653205868, + 0.634704455931608, + 0.737578881140714, + 0.827393267002123, + 0.891095315669898, + 0.28566774032832093, + 0.4235712022219935, + 0.6507118675226621, + 0.9638553619704904, + 0.43896373087445245, + 0.6990583012841766, + 0.9672275727250832, + 1.2954922341600084, + 0.41583165157189705, + 0.6260461841335843, + 0.8601355395935444, + 1.123167915512743, + 0.5265174094047388, + 0.7243002545116063, + 0.9740823893081514, + 1.2868835583244815, + 0.7679879406284286, + 1.047006706499755, + 1.4209584384939362, + 1.9199142134608593, + 0.545541803815954, + 0.9834623201239275, + 1.6109533957689843, + 2.3071407328027878 + ], + "population_cv": 0.500198998304589 + }, + "bits_per_byte": 2.0628794423908765, + "branch_output_rms_by_sublayer": [ + 0.04650525376200676, + 0.06399014592170715, + 0.1112780049443245, + 0.031699467450380325, + 0.04443240538239479, + 0.027440544217824936, + 0.029028279706835747, + 0.015319430269300938, + 0.07375111430883408, + 0.0059572625905275345, + 0.023948675021529198, + 0.008174876682460308, + 0.02697572112083435, + 0.012948136776685715, + 0.018753280863165855, + 0.015564307570457458, + 0.029962820932269096, + 0.014876189641654491, + 0.0314951054751873, + 0.016637088730931282, + 0.03273233398795128, + 0.02445702627301216, + 0.03909192234277725, + 0.03150094673037529, + 0.04035414755344391, + 0.026773061603307724, + 0.0292054396122694, + 0.0282891932874918, + 0.030428806319832802, + 0.031183898448944092, + 0.03996886685490608, + 0.03284372761845589, + 0.040643710643053055, + 0.022153088822960854, + 0.0380491279065609, + 0.020094403997063637, + 0.05119108408689499, + 0.022748352959752083, + 0.05440197139978409, + 0.03155774250626564, + 0.045650895684957504, + 0.03918999433517456, + 0.02034544013440609, + 0.04308570176362991, + 0.02799524925649166, + 0.05155247822403908, + 0.032459575682878494, + 0.06241406500339508, + 0.07144802808761597, + 0.04771707206964493, + 0.02726258896291256, + 0.05042983219027519, + 0.03319954127073288, + 0.06471183896064758, + 0.03339804708957672, + 0.09100748598575592, + 0.01278496440500021, + 0.06295084953308105, + 0.010845393873751163, + 0.06175147742033005, + 0.014509993605315685, + 0.08670730143785477, + 0.008239218965172768, + 0.093605637550354 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00025232285732266205, + 0.00023011290450512519, + 0.00017599709815245326, + 0.00014078929682447445, + 0.00017294674415431897, + 8.593806812378746e-05, + 0.00010899508934467917, + 0.00010300707962332868, + 0.00011965785473729695, + 0.00010869806617994115, + 0.00013749191323610767, + 0.00016956747382310418, + 0.0001434597026135581, + 0.00012835358230432807, + 0.0001281471074043329, + 0.0001401386693332853, + 0.0001462422637670556, + 0.00016692405720379573, + 0.00022724862139714635, + 0.0002360166586077954, + 0.0006879903663218003, + 0.00048463703414726737, + 0.00041889129848543007, + 0.0003466009194771917, + 0.0001426437862720874, + 0.00011549999469420032, + 0.0001178542239817036, + 0.00012467994691411455, + 0.00018234592045258488, + 0.000172386131898392, + 0.00016970163654101184, + 0.00016042005282667672 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00015876364225635365, + "first_to_last_ratio": 1.071341360950546, + "imbalance_abs_log_ratio": 0.06891147172792389, + "last_quartile_mean": 0.0001481914616975964, + "mean": 0.00019517832564596991, + "normalized": [ + 1.2927811348292098, + 1.1789880036296776, + 0.9017246027189049, + 0.7213367383827718, + 0.8860960538621672, + 0.4403053865707856, + 0.5584384894375167, + 0.5277588035577843, + 0.6130693781764578, + 0.5569166853962382, + 0.7044425285495148, + 0.8687822956872745, + 0.7350186150985679, + 0.657622109829685, + 0.6565642316082596, + 0.7180032356025026, + 0.749275122035433, + 0.8552386985149978, + 1.1643127926476227, + 1.209236004185738, + 3.5249322077377196, + 2.483047400592732, + 2.1461978275459166, + 1.7758166452656439, + 0.7308382516346928, + 0.5917665002604001, + 0.6038284404359378, + 0.6388001664706819, + 0.9342529189605742, + 0.8832237459146963, + 0.8694696810179132, + 0.8219153038419823 + ], + "population_cv": 0.6379412607329027 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.43816590309143066, + "mean_weights": [ + 0.8293558955192566, + 0.17064407467842102 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6417856216430664, + "mean_weights": [ + 0.528282642364502, + 0.47171729803085327 + ], + "sources": 2 + }, + { + "entropy_mean": 0.46827006340026855, + "mean_weights": [ + 0.8155502080917358, + 0.18444976210594177 + ], + "sources": 2 + }, + { + "entropy_mean": 0.688706636428833, + "mean_weights": [ + 0.48138347268104553, + 0.5186165571212769 + ], + "sources": 2 + }, + { + "entropy_mean": 0.4676797389984131, + "mean_weights": [ + 0.8172187805175781, + 0.18278121948242188 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6879211068153381, + "mean_weights": [ + 0.5289583802223206, + 0.47104161977767944 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6218067407608032, + "mean_weights": [ + 0.6540642976760864, + 0.3459357023239136 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6757431030273438, + "mean_weights": [ + 0.5100307464599609, + 0.48996931314468384 + ], + "sources": 2 + }, + { + "entropy_mean": 0.984381914138794, + "mean_weights": [ + 0.2721431851387024, + 0.19659718871116638, + 0.5312596559524536 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0924484729766846, + "mean_weights": [ + 0.32214832305908203, + 0.30409926176071167, + 0.3737524151802063 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0103507041931152, + "mean_weights": [ + 0.2565372586250305, + 0.24575984477996826, + 0.497702956199646 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0870685577392578, + "mean_weights": [ + 0.34117013216018677, + 0.2724571228027344, + 0.38637274503707886 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0382781028747559, + "mean_weights": [ + 0.408648282289505, + 0.20773811638355255, + 0.38361358642578125 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0872046947479248, + "mean_weights": [ + 0.3424554467201233, + 0.27643316984176636, + 0.38111138343811035 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0455865859985352, + "mean_weights": [ + 0.3580952286720276, + 0.2454036921262741, + 0.3965010941028595 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0893900394439697, + "mean_weights": [ + 0.38099974393844604, + 0.2939714193344116, + 0.32502883672714233 + ], + "sources": 3 + }, + { + "entropy_mean": 1.352750301361084, + "mean_weights": [ + 0.2963322401046753, + 0.16735601425170898, + 0.25198984146118164, + 0.2843219041824341 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3803377151489258, + "mean_weights": [ + 0.27474844455718994, + 0.23236174881458282, + 0.22279876470565796, + 0.27009105682373047 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3632738590240479, + "mean_weights": [ + 0.29269301891326904, + 0.19119808077812195, + 0.24140125513076782, + 0.2747076749801636 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3646692037582397, + "mean_weights": [ + 0.28328514099121094, + 0.20917798578739166, + 0.19488075375556946, + 0.31265610456466675 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3278639316558838, + "mean_weights": [ + 0.3390665650367737, + 0.16696593165397644, + 0.18352128565311432, + 0.31044623255729675 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3690853118896484, + "mean_weights": [ + 0.2747570276260376, + 0.21201756596565247, + 0.20685924589633942, + 0.3063661456108093 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3082637786865234, + "mean_weights": [ + 0.39022335410118103, + 0.16710132360458374, + 0.17327755689620972, + 0.2693977952003479 + ], + "sources": 4 + }, + { + "entropy_mean": 1.37571120262146, + "mean_weights": [ + 0.2876010835170746, + 0.23559685051441193, + 0.20718392729759216, + 0.26961812376976013 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6047325134277344, + "mean_weights": [ + 0.22566823661327362, + 0.18626481294631958, + 0.19221937656402588, + 0.1927841305732727, + 0.2030634582042694 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5928101539611816, + "mean_weights": [ + 0.21172791719436646, + 0.21463404595851898, + 0.1510435938835144, + 0.17578744888305664, + 0.24680697917938232 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5724632740020752, + "mean_weights": [ + 0.20307239890098572, + 0.16472786664962769, + 0.15638282895088196, + 0.19397655129432678, + 0.28184035420417786 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5841104984283447, + "mean_weights": [ + 0.20893999934196472, + 0.19673532247543335, + 0.15304291248321533, + 0.1648825705051422, + 0.2763991951942444 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5809128284454346, + "mean_weights": [ + 0.2174575924873352, + 0.16172778606414795, + 0.1596592217683792, + 0.18877117335796356, + 0.2723841965198517 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5618667602539062, + "mean_weights": [ + 0.21157142519950867, + 0.2109021246433258, + 0.13861587643623352, + 0.14108233153820038, + 0.29782822728157043 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5745904445648193, + "mean_weights": [ + 0.1914936602115631, + 0.15772637724876404, + 0.15994736552238464, + 0.22539494931697845, + 0.26543763279914856 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5645475387573242, + "mean_weights": [ + 0.19709429144859314, + 0.20174247026443481, + 0.1345168948173523, + 0.16582883894443512, + 0.3008175194263458 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7812910079956055, + "mean_weights": [ + 0.17059893906116486, + 0.1558210253715515, + 0.15442928671836853, + 0.17894327640533447, + 0.1881815791130066, + 0.15202590823173523 + ], + "sources": 6 + }, + { + "entropy_mean": 1.747176170349121, + "mean_weights": [ + 0.15818384289741516, + 0.17246046662330627, + 0.11104817688465118, + 0.11147627979516983, + 0.19791880249977112, + 0.24891240894794464 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7318758964538574, + "mean_weights": [ + 0.14546990394592285, + 0.13619554042816162, + 0.11688403785228729, + 0.1495826244354248, + 0.16734179854393005, + 0.28452610969543457 + ], + "sources": 6 + }, + { + "entropy_mean": 1.764691948890686, + "mean_weights": [ + 0.16610369086265564, + 0.17969636619091034, + 0.1221495121717453, + 0.12015374004840851, + 0.18626880645751953, + 0.22562789916992188 + ], + "sources": 6 + }, + { + "entropy_mean": 1.693286657333374, + "mean_weights": [ + 0.14076460897922516, + 0.11972320079803467, + 0.11196848750114441, + 0.15171939134597778, + 0.14472678303718567, + 0.3310975134372711 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7657266855239868, + "mean_weights": [ + 0.16921313107013702, + 0.18652436137199402, + 0.129713773727417, + 0.11542271077632904, + 0.18010325729846954, + 0.21902278065681458 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7083728313446045, + "mean_weights": [ + 0.14149782061576843, + 0.1375911831855774, + 0.13043227791786194, + 0.15217545628547668, + 0.1272602081298828, + 0.31104305386543274 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7751741409301758, + "mean_weights": [ + 0.18067733943462372, + 0.2015353888273239, + 0.13959306478500366, + 0.1226978749036789, + 0.17863549292087555, + 0.17686083912849426 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7296388149261475, + "mean_weights": [ + 0.09262444078922272, + 0.09199242293834686, + 0.08533023297786713, + 0.1001829206943512, + 0.09970749914646149, + 0.13031859695911407, + 0.39984390139579773 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8746612071990967, + "mean_weights": [ + 0.13717350363731384, + 0.1495172083377838, + 0.10245971381664276, + 0.09319422394037247, + 0.12672355771064758, + 0.11817892640829086, + 0.2727528512477875 + ], + "sources": 7 + }, + { + "entropy_mean": 1.6826035976409912, + "mean_weights": [ + 0.09000763297080994, + 0.09147360175848007, + 0.08575387299060822, + 0.094002366065979, + 0.08878561109304428, + 0.11957557499408722, + 0.43040135502815247 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8978075981140137, + "mean_weights": [ + 0.13825683295726776, + 0.15689769387245178, + 0.11289732158184052, + 0.09766772389411926, + 0.13788357377052307, + 0.11564236879348755, + 0.24075448513031006 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7299697399139404, + "mean_weights": [ + 0.1026417464017868, + 0.09912316501140594, + 0.0906987264752388, + 0.09591005742549896, + 0.09112510085105896, + 0.11974775791168213, + 0.4007534384727478 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9012258052825928, + "mean_weights": [ + 0.14502407610416412, + 0.15984030067920685, + 0.1120377629995346, + 0.09669173508882523, + 0.13777585327625275, + 0.1154010072350502, + 0.23322926461696625 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7523971796035767, + "mean_weights": [ + 0.10635381937026978, + 0.11480027437210083, + 0.09922759979963303, + 0.09573516994714737, + 0.1001138836145401, + 0.10635073482990265, + 0.37741851806640625 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9197427034378052, + "mean_weights": [ + 0.14406782388687134, + 0.16876816749572754, + 0.12567192316055298, + 0.0996539443731308, + 0.15650184452533722, + 0.12792134284973145, + 0.17741495370864868 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0483996868133545, + "mean_weights": [ + 0.11221400648355484, + 0.14220482110977173, + 0.10712738335132599, + 0.09367531538009644, + 0.12853848934173584, + 0.10709765553474426, + 0.14095236361026764, + 0.16818997263908386 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0573606491088867, + "mean_weights": [ + 0.11874908953905106, + 0.14375966787338257, + 0.10706768929958344, + 0.0955226793885231, + 0.14080685377120972, + 0.10752296447753906, + 0.13675561547279358, + 0.1498154252767563 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0327858924865723, + "mean_weights": [ + 0.12014257907867432, + 0.12857738137245178, + 0.10980319231748581, + 0.09702245891094208, + 0.12268230319023132, + 0.09971651434898376, + 0.15491321682929993, + 0.1671423614025116 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0022356510162354, + "mean_weights": [ + 0.10498970746994019, + 0.12256842851638794, + 0.11012198030948639, + 0.10951010882854462, + 0.10492608696222305, + 0.08929784595966339, + 0.22632169723510742, + 0.1322641372680664 + ], + "sources": 8 + }, + { + "entropy_mean": 2.072866916656494, + "mean_weights": [ + 0.08920659124851227, + 0.11760489642620087, + 0.0819014310836792, + 0.08213429152965546, + 0.07646629214286804, + 0.07197368890047073, + 0.238392636179924, + 0.10916344821453094, + 0.13315674662590027 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0888843536376953, + "mean_weights": [ + 0.08150166273117065, + 0.10506386309862137, + 0.09466397017240524, + 0.1080104187130928, + 0.08821793645620346, + 0.08452457934617996, + 0.2252335548400879, + 0.08945520222187042, + 0.1233288124203682 + ], + "sources": 9 + }, + { + "entropy_mean": 2.028970718383789, + "mean_weights": [ + 0.0796959400177002, + 0.10641789436340332, + 0.08554358780384064, + 0.0815211609005928, + 0.08365342020988464, + 0.08185926079750061, + 0.25650620460510254, + 0.09857281297445297, + 0.12622973322868347 + ], + "sources": 9 + }, + { + "entropy_mean": 2.117379665374756, + "mean_weights": [ + 0.11046799272298813, + 0.1114221066236496, + 0.098695769906044, + 0.07872100174427032, + 0.10845552384853363, + 0.08489114046096802, + 0.14244475960731506, + 0.1177409291267395, + 0.14716076850891113 + ], + "sources": 9 + }, + { + "entropy_mean": 2.024315118789673, + "mean_weights": [ + 0.08912506699562073, + 0.12136160582304001, + 0.08845455944538116, + 0.08194804191589355, + 0.07905752956867218, + 0.06762532144784927, + 0.23523791134357452, + 0.10633023083209991, + 0.13085974752902985 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0895094871520996, + "mean_weights": [ + 0.08345537632703781, + 0.09734349697828293, + 0.09989890456199646, + 0.10864540189504623, + 0.08091968297958374, + 0.08874625712633133, + 0.24269162118434906, + 0.10028015822172165, + 0.0980190858244896 + ], + "sources": 9 + }, + { + "entropy_mean": 2.059999942779541, + "mean_weights": [ + 0.08754365146160126, + 0.12311948835849762, + 0.0878293514251709, + 0.09081916511058807, + 0.09198091179132462, + 0.09828788042068481, + 0.23319105803966522, + 0.10020989179611206, + 0.08701859414577484 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05599086731672287, + 0.04790570214390755, + 0.05566028878092766, + 0.06093958392739296, + 0.10357698053121567, + 0.0658627301454544, + 0.11313136667013168, + 0.09590502083301544, + 0.12319625169038773, + 0.07760516554117203, + 0.0942867249250412, + 0.08953271806240082, + 0.09126316756010056, + 0.08013118803501129, + 0.09393157064914703, + 0.08856885880231857, + 0.09513302892446518, + 0.06031104177236557, + 0.0725446492433548, + 0.06541399657726288, + 0.06651586294174194, + 0.05889610946178436, + 0.06827826797962189, + 0.05958769470453262, + 0.07393281161785126, + 0.05783972889184952, + 0.06153496354818344, + 0.05291128158569336, + 0.05987953767180443, + 0.05467431619763374, + 0.06490708887577057, + 0.061766184866428375, + 0.07276938110589981, + 0.05668729916214943, + 0.05836696922779083, + 0.053697023540735245, + 0.05993006005883217, + 0.055303920060396194, + 0.061307769268751144, + 0.058913011103868484, + 0.06396329402923584, + 0.034660086035728455, + 0.04545588418841362, + 0.039951033890247345, + 0.04860804229974747, + 0.04668503254652023, + 0.050228700041770935, + 0.053336501121520996, + 0.05336160585284233, + 0.04270453751087189, + 0.0441121906042099, + 0.0436425656080246, + 0.04693005234003067, + 0.04539760202169418, + 0.05381348356604576, + 0.048623085021972656, + 0.06201789155602455, + 0.05367467552423477, + 0.053286172449588776, + 0.058078594505786896, + 0.0645245611667633, + 0.0653827115893364, + 0.06236105412244797, + 0.06610161811113358 + ], + "loss_nats": 1.429879069328308, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.959059238433838, + "mean_weights": [ + 0.11137355864048004, + 0.07938310503959656, + 0.08366742730140686, + 0.06241578981280327, + 0.06857114285230637, + 0.05315379798412323, + 0.12024731189012527, + 0.13800561428070068, + 0.2831822633743286 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.04650525376200676, + 0.08216902613639832, + 0.16759145259857178, + 0.18183350563049316, + 0.20560869574546814, + 0.2180991768836975, + 0.22782908380031586, + 0.23346203565597534, + 0.07375111430883408, + 0.07513487339019775, + 0.08537013828754425, + 0.08731291443109512, + 0.09593576192855835, + 0.09794493764638901, + 0.10240181535482407, + 0.10548584163188934, + 0.029962820932269096, + 0.03381669893860817, + 0.04250640422105789, + 0.05014139786362648, + 0.06402412801980972, + 0.07702979445457458, + 0.09613197296857834, + 0.11409901082515717, + 0.04035414755344391, + 0.051963530480861664, + 0.06772714108228683, + 0.08275293558835983, + 0.09575163573026657, + 0.11449820548295975, + 0.133773535490036, + 0.15335743129253387, + 0.040643710643053055, + 0.049225207418203354, + 0.06838896125555038, + 0.07410992681980133, + 0.09598347544670105, + 0.10182089358568192, + 0.12257353216409683, + 0.1329580694437027, + 0.045650895684957504, + 0.06232793629169464, + 0.06614943593740463, + 0.08574102073907852, + 0.09268787503242493, + 0.11530966311693192, + 0.12428413331508636, + 0.15233835577964783, + 0.07144802808761597, + 0.09091266989707947, + 0.09853106737136841, + 0.12394227832555771, + 0.13304176926612854, + 0.16820983588695526, + 0.1741023063659668, + 0.227275088429451, + 0.01278496440500021, + 0.06458000093698502, + 0.06617290526628494, + 0.11642003804445267, + 0.11880602687597275, + 0.19070100784301758, + 0.19166752696037292, + 0.27311408519744873 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.833918450865895e-05, + 7.219606050057337e-05, + 6.801288691349328e-05, + 6.66896958136931e-05, + 6.77474235999398e-05, + 6.401057908078656e-05, + 5.945320299360901e-05, + 5.758965198765509e-05, + 5.835931369801983e-05, + 5.6850247347028926e-05, + 5.306725870468654e-05, + 4.8401478125015274e-05, + 5.1861345127690583e-05, + 4.9128801038023084e-05, + 4.750795051222667e-05, + 4.776344576384872e-05, + 6.393855437636375e-05, + 6.446982297347859e-05, + 5.932587737333961e-05, + 5.471019903779961e-05, + 0.00012503248581197113, + 9.990497346734628e-05, + 7.783094042679295e-05, + 5.9666639572242275e-05, + 2.9385437301243655e-05, + 2.7327721909387037e-05, + 2.5612416720832698e-05, + 2.4898467017919756e-05, + 4.457414979697205e-05, + 3.837289841612801e-05, + 3.2854379242053255e-05, + 3.16314326482825e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 6.550483567480114e-05, + "first_to_last_ratio": 2.0578224234892115, + "imbalance_abs_log_ratio": 0.7216483477265278, + "last_quartile_mean": 3.183211288160237e-05, + "mean": 5.6141091306471935e-05, + "normalized": [ + 1.2172756695376319, + 1.285975367070405, + 1.2114635702789192, + 1.1878945396632348, + 1.2067350673699833, + 1.1401734022474805, + 1.0589962113322022, + 1.02580214683173, + 1.0395115652355011, + 1.012631675374562, + 0.9452480788981199, + 0.8621399584271273, + 0.9237680266060666, + 0.8750952269493829, + 0.8462242077355194, + 0.8507751568830398, + 1.1388904791202896, + 1.1483535761985895, + 1.056728253633013, + 0.9745125676153829, + 2.2271117803789005, + 1.7795338697990228, + 1.3863453419869773, + 1.0627980002477064, + 0.5234211985803722, + 0.48676862656990605, + 0.45621515586534567, + 0.44349809450621536, + 0.7939665717156009, + 0.6835082383178467, + 0.5852109119629068, + 0.5634274630610188 + ], + "population_cv": 0.3657030811255725 + }, + "activation_output_rms_by_block": [ + 0.1163102313876152, + 0.2701985239982605, + 0.3088720142841339, + 0.32299426198005676, + 0.08081311732530594, + 0.09053189307451248, + 0.10288996994495392, + 0.1135161817073822, + 0.03170512244105339, + 0.05612620338797569, + 0.09513665735721588, + 0.15618830919265747, + 0.04777410998940468, + 0.08415272831916809, + 0.11939067393541336, + 0.16319428384304047, + 0.0474398173391819, + 0.07795747369527817, + 0.11981253325939178, + 0.1691688895225525, + 0.07734841853380203, + 0.10943150520324707, + 0.14846715331077576, + 0.19392108917236328, + 0.11690890789031982, + 0.16214139759540558, + 0.21733160316944122, + 0.2819598615169525, + 0.07869446277618408, + 0.13749846816062927, + 0.2228851318359375, + 0.3288308382034302 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.17576577421277761, + "first_to_last_ratio": 0.9093779035568548, + "imbalance_abs_log_ratio": 0.09499453571262267, + "last_quartile_mean": 0.19328133389353752, + "mean": 0.14529974479228258, + "normalized": [ + 0.8004847603407, + 1.859593933798924, + 2.125757445243216, + 2.2229513371947243, + 0.5561821009447377, + 0.623069869833116, + 0.7081221656104298, + 0.7812552036458323, + 0.21820494238567562, + 0.38627874720779803, + 0.6547613520809773, + 1.074938634035049, + 0.32879692980673525, + 0.5791663876593248, + 0.821685365697591, + 1.1231560253346593, + 0.32649621929481604, + 0.5365286346973597, + 0.8245887384776431, + 1.1642752006508528, + 0.5323369194101317, + 0.7531431342820871, + 1.0217991333916052, + 1.3346278718492501, + 0.8046050463299182, + 1.1159097204692237, + 1.4957466269478576, + 1.9405392756885866, + 0.5416008327384471, + 0.9463090823538207, + 1.5339678136019395, + 2.26312054899697 + ], + "population_cv": 0.5644369048717107 + }, + "bits_per_byte": 1.9385029651481003, + "branch_output_rms_by_sublayer": [ + 0.07099080085754395, + 0.08790170401334763, + 0.17927438020706177, + 0.030547721311450005, + 0.04922495782375336, + 0.03191060945391655, + 0.02927345596253872, + 0.015817100182175636, + 0.07919608801603317, + 0.004812270402908325, + 0.026761727407574654, + 0.0075308773666620255, + 0.03239119425415993, + 0.014680612832307816, + 0.02306862734258175, + 0.018474049866199493, + 0.02545703575015068, + 0.01858403906226158, + 0.026546726003289223, + 0.019265687093138695, + 0.03960845246911049, + 0.0333264097571373, + 0.05902634188532829, + 0.04816716909408569, + 0.033392176032066345, + 0.03104039467871189, + 0.03320883959531784, + 0.03645269572734833, + 0.027194170281291008, + 0.04122060537338257, + 0.04199736937880516, + 0.044514384120702744, + 0.03488221764564514, + 0.028846094384789467, + 0.04176390543580055, + 0.026438944041728973, + 0.06473326683044434, + 0.035805050283670425, + 0.06922738254070282, + 0.05365557596087456, + 0.05274078994989395, + 0.052817150950431824, + 0.025974830612540245, + 0.0607898123562336, + 0.03874174878001213, + 0.072703517973423, + 0.04229002073407173, + 0.08672743290662766, + 0.09003424644470215, + 0.06601041555404663, + 0.046459127217531204, + 0.06716953217983246, + 0.05384040251374245, + 0.08366808295249939, + 0.03662457689642906, + 0.11518727988004684, + 0.016323253512382507, + 0.07687225937843323, + 0.013699524104595184, + 0.07620421051979065, + 0.018941720947623253, + 0.10569602251052856, + 0.014551514759659767, + 0.12411575019359589 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00026752465120024417, + 0.00021587771930107027, + 0.00015967907038499888, + 0.00011915784921511235, + 0.00012792202034245557, + 8.085766134734438e-05, + 0.00011164158065215218, + 0.00010801640297096873, + 0.00010521030070510286, + 0.0001003818297034207, + 0.00014964229201663417, + 0.00020753984439651047, + 0.00013620377795560672, + 0.00015422662481767726, + 0.000155829882545591, + 0.00016830037981681867, + 0.00017009311637874628, + 0.0002092582843383427, + 0.00029696301612371784, + 0.0003081641929652966, + 0.0008358810797326418, + 0.0006164889297019506, + 0.00056361455432508, + 0.0004734322977496837, + 0.00020108658721555077, + 0.00015678575755442788, + 0.0001580917126092831, + 0.0001539830212360982, + 0.0002471882066412232, + 0.00022016980961935781, + 0.00021114048015340558, + 0.0001950714703944639 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0001488346194267933, + "first_to_last_ratio": 0.7714051224406252, + "imbalance_abs_log_ratio": 0.2595415927833044, + "last_quartile_mean": 0.0001929396306779763, + "mean": 0.0002307945126284681, + "normalized": [ + 1.1591464985604059, + 0.9353676430279311, + 0.6918668410546203, + 0.5162941174729405, + 0.5542680321363785, + 0.35034481724229144, + 0.483727188227704, + 0.4680198057605166, + 0.4558613612901183, + 0.4349402789528835, + 0.6483788991011568, + 0.8992408096400764, + 0.5901517172328395, + 0.6682421651243959, + 0.675188854236085, + 0.7292217561851128, + 0.7369894303014101, + 0.9066865670034612, + 1.2866987726080277, + 1.3352318885561107, + 3.6217545651886414, + 2.6711593905803626, + 2.4420621959603697, + 2.051315225643223, + 0.8712797584544785, + 0.6793305255347246, + 0.6849890441883182, + 0.667186665239218, + 1.071031558879156, + 0.9539646636823906, + 0.9148418554183674, + 0.8452171075162823 + ], + "population_cv": 0.7120860223640003 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.23803412914276123, + "mean_weights": [ + 0.9308671951293945, + 0.06913280487060547 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5923433303833008, + "mean_weights": [ + 0.6193963289260864, + 0.38060373067855835 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5038111209869385, + "mean_weights": [ + 0.7891626358032227, + 0.21083739399909973 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6824523210525513, + "mean_weights": [ + 0.43170660734176636, + 0.5682933926582336 + ], + "sources": 2 + }, + { + "entropy_mean": 0.4536406695842743, + "mean_weights": [ + 0.8259638547897339, + 0.1740362048149109 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6898715496063232, + "mean_weights": [ + 0.5186047554016113, + 0.4813952147960663 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6382434368133545, + "mean_weights": [ + 0.6198925375938416, + 0.38010740280151367 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6744688153266907, + "mean_weights": [ + 0.49289023876190186, + 0.5071097612380981 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6433025002479553, + "mean_weights": [ + 0.1175871342420578, + 0.10191814601421356, + 0.780494749546051 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0916191339492798, + "mean_weights": [ + 0.3107808232307434, + 0.3119170665740967, + 0.3773021399974823 + ], + "sources": 3 + }, + { + "entropy_mean": 0.92546546459198, + "mean_weights": [ + 0.18145589530467987, + 0.22260317206382751, + 0.5959409475326538 + ], + "sources": 3 + }, + { + "entropy_mean": 1.089654803276062, + "mean_weights": [ + 0.3537278175354004, + 0.2771497368812561, + 0.36912238597869873 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0277583599090576, + "mean_weights": [ + 0.3776150643825531, + 0.1953553855419159, + 0.427029550075531 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0870227813720703, + "mean_weights": [ + 0.35670050978660583, + 0.2779994606971741, + 0.3652999997138977 + ], + "sources": 3 + }, + { + "entropy_mean": 1.028363823890686, + "mean_weights": [ + 0.33548229932785034, + 0.22723403573036194, + 0.4372836947441101 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0881397724151611, + "mean_weights": [ + 0.3873855471611023, + 0.2860027551651001, + 0.32661163806915283 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3423279523849487, + "mean_weights": [ + 0.2821422815322876, + 0.1502475142478943, + 0.2720600962638855, + 0.295550137758255 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3765307664871216, + "mean_weights": [ + 0.2765946388244629, + 0.23256731033325195, + 0.2202550768852234, + 0.27058297395706177 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3583530187606812, + "mean_weights": [ + 0.27538156509399414, + 0.20539003610610962, + 0.26003462076187134, + 0.2591937780380249 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3578999042510986, + "mean_weights": [ + 0.3066127300262451, + 0.2021423578262329, + 0.18748149275779724, + 0.30376338958740234 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3263497352600098, + "mean_weights": [ + 0.3233564496040344, + 0.16292965412139893, + 0.2008475363254547, + 0.31286633014678955 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3654263019561768, + "mean_weights": [ + 0.2859629988670349, + 0.20946303009986877, + 0.20187759399414062, + 0.3026963770389557 + ], + "sources": 4 + }, + { + "entropy_mean": 1.2946724891662598, + "mean_weights": [ + 0.40006059408187866, + 0.1658974140882492, + 0.17814326286315918, + 0.25589871406555176 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3788061141967773, + "mean_weights": [ + 0.28424757719039917, + 0.24038995802402496, + 0.21404734253883362, + 0.26131510734558105 + ], + "sources": 4 + }, + { + "entropy_mean": 1.600654125213623, + "mean_weights": [ + 0.22448194026947021, + 0.2024102360010147, + 0.2070591151714325, + 0.1772775650024414, + 0.18877115845680237 + ], + "sources": 5 + }, + { + "entropy_mean": 1.582503080368042, + "mean_weights": [ + 0.21139678359031677, + 0.21127468347549438, + 0.1495627760887146, + 0.15729400515556335, + 0.2704717516899109 + ], + "sources": 5 + }, + { + "entropy_mean": 1.56495201587677, + "mean_weights": [ + 0.20842036604881287, + 0.1619974970817566, + 0.1597384214401245, + 0.19928060472011566, + 0.2705630958080292 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5598231554031372, + "mean_weights": [ + 0.19806599617004395, + 0.20232272148132324, + 0.14740434288978577, + 0.13874071836471558, + 0.31346622109413147 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5643353462219238, + "mean_weights": [ + 0.21911504864692688, + 0.1470745950937271, + 0.15926693379878998, + 0.18593734502792358, + 0.28860607743263245 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5489161014556885, + "mean_weights": [ + 0.2020987868309021, + 0.20591995120048523, + 0.1417527198791504, + 0.12904959917068481, + 0.32117897272109985 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5741775035858154, + "mean_weights": [ + 0.19899265468120575, + 0.16279086470603943, + 0.1725691556930542, + 0.21424663066864014, + 0.2514007091522217 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5700864791870117, + "mean_weights": [ + 0.19777736067771912, + 0.22011584043502808, + 0.14352375268936157, + 0.1510990709066391, + 0.28748399019241333 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7797424793243408, + "mean_weights": [ + 0.16638055443763733, + 0.14286619424819946, + 0.16056445240974426, + 0.17359018325805664, + 0.17503789067268372, + 0.1815607100725174 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7449073791503906, + "mean_weights": [ + 0.1564365178346634, + 0.17917592823505402, + 0.12025889754295349, + 0.10482373088598251, + 0.17921267449855804, + 0.26009225845336914 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6872644424438477, + "mean_weights": [ + 0.13885441422462463, + 0.11083191633224487, + 0.11442253738641739, + 0.14626961946487427, + 0.15138331055641174, + 0.3382381796836853 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7640838623046875, + "mean_weights": [ + 0.16653618216514587, + 0.19283242523670197, + 0.13476425409317017, + 0.11015628278255463, + 0.17283815145492554, + 0.22287270426750183 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6483933925628662, + "mean_weights": [ + 0.1412433385848999, + 0.10675234347581863, + 0.10247255861759186, + 0.14859884977340698, + 0.12817418575286865, + 0.37275874614715576 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7659040689468384, + "mean_weights": [ + 0.17153221368789673, + 0.19901537895202637, + 0.14121603965759277, + 0.10897990316152573, + 0.1691284328699112, + 0.2101280391216278 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6645760536193848, + "mean_weights": [ + 0.14680790901184082, + 0.12548011541366577, + 0.11959286034107208, + 0.14119240641593933, + 0.11379042267799377, + 0.3531363010406494 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7701325416564941, + "mean_weights": [ + 0.18254882097244263, + 0.22029608488082886, + 0.1568000316619873, + 0.11600597202777863, + 0.16151979565620422, + 0.16282927989959717 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7035170793533325, + "mean_weights": [ + 0.09249584376811981, + 0.08449239283800125, + 0.08839155733585358, + 0.09361176192760468, + 0.09255413711071014, + 0.13066577911376953, + 0.4177885055541992 + ], + "sources": 7 + }, + { + "entropy_mean": 1.866309642791748, + "mean_weights": [ + 0.13266775012016296, + 0.14844295382499695, + 0.11146600544452667, + 0.08739326149225235, + 0.12395921349525452, + 0.11387947201728821, + 0.28219133615493774 + ], + "sources": 7 + }, + { + "entropy_mean": 1.6761362552642822, + "mean_weights": [ + 0.09199865162372589, + 0.0861373022198677, + 0.09127336740493774, + 0.0915021225810051, + 0.0842541754245758, + 0.121028833091259, + 0.43380552530288696 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8849153518676758, + "mean_weights": [ + 0.13904348015785217, + 0.16547125577926636, + 0.1168784573674202, + 0.08359336853027344, + 0.13188466429710388, + 0.11393873393535614, + 0.24919000267982483 + ], + "sources": 7 + }, + { + "entropy_mean": 1.6970231533050537, + "mean_weights": [ + 0.10218608379364014, + 0.0924035906791687, + 0.09211519360542297, + 0.09030843526124954, + 0.08260206878185272, + 0.11970463395118713, + 0.4206800162792206 + ], + "sources": 7 + }, + { + "entropy_mean": 1.888169288635254, + "mean_weights": [ + 0.14510993659496307, + 0.1727357804775238, + 0.11756978929042816, + 0.08477480709552765, + 0.1326131820678711, + 0.10765296220779419, + 0.23954354226589203 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7295632362365723, + "mean_weights": [ + 0.10637101531028748, + 0.1087622195482254, + 0.10079064220190048, + 0.09328937530517578, + 0.0935361385345459, + 0.10445909202098846, + 0.3927915394306183 + ], + "sources": 7 + }, + { + "entropy_mean": 1.910825490951538, + "mean_weights": [ + 0.14582416415214539, + 0.17717620730400085, + 0.12502005696296692, + 0.08654762804508209, + 0.1576351821422577, + 0.123397096991539, + 0.18439966440200806 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0270450115203857, + "mean_weights": [ + 0.11430318653583527, + 0.14596056938171387, + 0.10318497568368912, + 0.07560490071773529, + 0.13337919116020203, + 0.09910126030445099, + 0.14202386140823364, + 0.1864420473575592 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.045854330062866, + "mean_weights": [ + 0.12201360613107681, + 0.15195448696613312, + 0.10882705450057983, + 0.07983958721160889, + 0.14160361886024475, + 0.10041823983192444, + 0.13708916306495667, + 0.1582542061805725 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.028752565383911, + "mean_weights": [ + 0.1180800050497055, + 0.13710573315620422, + 0.10759006440639496, + 0.09344697743654251, + 0.1392270028591156, + 0.09335210174322128, + 0.1529756784439087, + 0.15822245180606842 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 1.9969491958618164, + "mean_weights": [ + 0.09964598715305328, + 0.12386604398488998, + 0.12082550674676895, + 0.11058294773101807, + 0.11714793741703033, + 0.08647093921899796, + 0.219528928399086, + 0.12193168699741364 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0260696411132812, + "mean_weights": [ + 0.09158119559288025, + 0.10912381857633591, + 0.08129943907260895, + 0.07584778219461441, + 0.06846241652965546, + 0.06350254267454147, + 0.23163248598575592, + 0.11807465553283691, + 0.16047564148902893 + ], + "sources": 9 + }, + { + "entropy_mean": 2.092313289642334, + "mean_weights": [ + 0.07984863966703415, + 0.10204550623893738, + 0.113859161734581, + 0.11145438998937607, + 0.08977242559194565, + 0.0832073986530304, + 0.19726860523223877, + 0.08469775319099426, + 0.13784611225128174 + ], + "sources": 9 + }, + { + "entropy_mean": 1.99648916721344, + "mean_weights": [ + 0.07903002202510834, + 0.10009041428565979, + 0.08397360891103745, + 0.0773920863866806, + 0.07411383837461472, + 0.07082854956388474, + 0.24133984744548798, + 0.10359439253807068, + 0.16963723301887512 + ], + "sources": 9 + }, + { + "entropy_mean": 2.103548049926758, + "mean_weights": [ + 0.10561108589172363, + 0.11771437525749207, + 0.1068120226264, + 0.08028098940849304, + 0.11458904296159744, + 0.07470817863941193, + 0.13048933446407318, + 0.10960578918457031, + 0.1601891815662384 + ], + "sources": 9 + }, + { + "entropy_mean": 1.9802570343017578, + "mean_weights": [ + 0.09105579555034637, + 0.10425136983394623, + 0.08135946094989777, + 0.08036179840564728, + 0.07204489409923553, + 0.0610441118478775, + 0.22369657456874847, + 0.11430901288986206, + 0.17187699675559998 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0620784759521484, + "mean_weights": [ + 0.07946689426898956, + 0.10020706057548523, + 0.11354327946901321, + 0.12152859568595886, + 0.07952263951301575, + 0.08558133244514465, + 0.23295356333255768, + 0.09282220900058746, + 0.09437443315982819 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0503835678100586, + "mean_weights": [ + 0.08896902203559875, + 0.11803600192070007, + 0.09268700331449509, + 0.09350242465734482, + 0.08361929655075073, + 0.08848940581083298, + 0.220588818192482, + 0.10773170739412308, + 0.10637632012367249 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05911032855510712, + 0.055656228214502335, + 0.065603107213974, + 0.07895281165838242, + 0.1604311317205429, + 0.07835578173398972, + 0.15839993953704834, + 0.1353825330734253, + 0.17102932929992676, + 0.0870983824133873, + 0.12603406608104706, + 0.11119890958070755, + 0.11658843606710434, + 0.09777002781629562, + 0.11838720738887787, + 0.1088881716132164, + 0.11887550354003906, + 0.07130847126245499, + 0.09263374656438828, + 0.08642365783452988, + 0.08127535134553909, + 0.07197881489992142, + 0.08292870968580246, + 0.07007469236850739, + 0.09098084270954132, + 0.07604493200778961, + 0.07490143924951553, + 0.06412458419799805, + 0.07441893965005875, + 0.06271818280220032, + 0.07921985536813736, + 0.07260891050100327, + 0.0885709822177887, + 0.06254163384437561, + 0.0683855339884758, + 0.05454307422041893, + 0.07198251038789749, + 0.06029154360294342, + 0.07434829324483871, + 0.0690549835562706, + 0.08035052567720413, + 0.04126834124326706, + 0.05602646246552467, + 0.05029817298054695, + 0.06173199787735939, + 0.06093628704547882, + 0.06490133702754974, + 0.0703008845448494, + 0.06714639812707901, + 0.05234331637620926, + 0.054746728390455246, + 0.05367941036820412, + 0.05878523364663124, + 0.056092433631420135, + 0.06590934842824936, + 0.05957506224513054, + 0.07437775284051895, + 0.06695930659770966, + 0.06292693316936493, + 0.07170594483613968, + 0.07526534050703049, + 0.08271408826112747, + 0.07486039400100708, + 0.08155126124620438 + ], + "loss_nats": 1.3436678647994995, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.8903517723083496, + "mean_weights": [ + 0.10662682354450226, + 0.08120357990264893, + 0.07890123128890991, + 0.05622042343020439, + 0.0614643432199955, + 0.04690651595592499, + 0.1040859967470169, + 0.13219518959522247, + 0.33239588141441345 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.07099080085754395, + 0.1163102313876152, + 0.2592604458332062, + 0.2701985239982605, + 0.2966780364513397, + 0.3088720142841339, + 0.31859055161476135, + 0.32299426198005676, + 0.07919608801603317, + 0.08081311732530594, + 0.08872856944799423, + 0.09053189307451248, + 0.10108768939971924, + 0.10288996994495392, + 0.11029504984617233, + 0.1135161817073822, + 0.02545703575015068, + 0.03170512244105339, + 0.04651467502117157, + 0.05612620338797569, + 0.0768914520740509, + 0.09513665735721588, + 0.12738752365112305, + 0.15618830919265747, + 0.033392176032066345, + 0.04777410998940468, + 0.06362340599298477, + 0.08415272831916809, + 0.09378327429294586, + 0.11939067393541336, + 0.13502155244350433, + 0.16319428384304047, + 0.03488221764564514, + 0.0474398173391819, + 0.0678921490907669, + 0.07795747369527817, + 0.10834971070289612, + 0.11981253325939178, + 0.14848022162914276, + 0.1691688895225525, + 0.05274078994989395, + 0.07734841853380203, + 0.08190343528985977, + 0.10943150520324707, + 0.11846308410167694, + 0.14846715331077576, + 0.15885582566261292, + 0.19392108917236328, + 0.09003424644470215, + 0.11690890789031982, + 0.13141533732414246, + 0.16214139759540558, + 0.17780831456184387, + 0.21733160316944122, + 0.2236742079257965, + 0.2819598615169525, + 0.016323253512382507, + 0.07869446277618408, + 0.08025418221950531, + 0.13749846816062927, + 0.1396377980709076, + 0.2228851318359375, + 0.2245890498161316, + 0.3288308382034302 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00010971482697641477, + 0.00011380675277905539, + 0.00010946515976684168, + 0.00010794371337397024, + 0.00011168202036060393, + 0.0001070898724719882, + 0.0001006142920232378, + 9.775875514606014e-05, + 9.952406253432855e-05, + 9.70153632806614e-05, + 9.11346505745314e-05, + 8.317636820720509e-05, + 8.949485345510766e-05, + 8.352094300789759e-05, + 7.946522237034515e-05, + 7.939719944261014e-05, + 0.00011623578029684722, + 0.00011387227277737111, + 0.00010114193719346076, + 9.247743582818657e-05, + 0.00020436056365724653, + 0.0001567857980262488, + 0.00011874260235344991, + 8.899630483938381e-05, + 5.045254874858074e-05, + 4.603715206030756e-05, + 4.257175532984547e-05, + 4.093975803698413e-05, + 8.148024062393233e-05, + 6.529188976855949e-05, + 4.8964033339871094e-05, + 4.510274084168486e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00010725942411227152, + "first_to_last_ratio": 2.0389581569536372, + "imbalance_abs_log_ratio": 0.712438970025027, + "last_quartile_mean": 5.260501484372071e-05, + "mean": 9.294552717165061e-05, + "normalized": [ + 1.1804207293783469, + 1.2244457182848458, + 1.1777345623601965, + 1.161365335790943, + 1.201585743382261, + 1.1521788700409004, + 1.0825081645663768, + 1.0517854717798363, + 1.070778396366818, + 1.043787326113011, + 0.9805167967494025, + 0.8948937161182166, + 0.9628742358933498, + 0.8986009929628155, + 0.8549655350664673, + 0.8542336770652816, + 1.2505796011268457, + 1.2251506472933684, + 1.0881850936912019, + 0.9949638099034118, + 2.1987132665333755, + 1.6868568375183757, + 1.2775504746361555, + 0.9575103563082339, + 0.542818468880225, + 0.49531325994081177, + 0.45802909107422135, + 0.4404704484743748, + 0.8766450963633319, + 0.7024747909383445, + 0.5268035464411853, + 0.48525993895746805 + ], + "population_cv": 0.3561112389732315 + }, + "activation_output_rms_by_block": [ + 0.10425147414207458, + 0.2724364697933197, + 0.3075210154056549, + 0.31877824664115906, + 0.0578029565513134, + 0.06583662331104279, + 0.07787100225687027, + 0.08668279647827148, + 0.027938950806856155, + 0.04738941788673401, + 0.08426596224308014, + 0.15165171027183533, + 0.040915511548519135, + 0.07188191264867783, + 0.10232502222061157, + 0.1377868354320526, + 0.03995412588119507, + 0.06463814526796341, + 0.09971515089273453, + 0.14364886283874512, + 0.06747645884752274, + 0.09698512405157089, + 0.13247931003570557, + 0.17650066316127777, + 0.10708064585924149, + 0.15039627254009247, + 0.19441545009613037, + 0.24103640019893646, + 0.06552792340517044, + 0.11088091880083084, + 0.17807358503341675, + 0.2581630349159241 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.16139757307246327, + "first_to_last_ratio": 0.9889752371562447, + "imbalance_abs_log_ratio": 0.01108598593755416, + "last_quartile_mean": 0.16319677885621786, + "mean": 0.1275721243582666, + "normalized": [ + 0.8171963480775819, + 2.135548586054928, + 2.410565920671115, + 2.498808013464757, + 0.45310021168090553, + 0.5160737397940541, + 0.6104076627130673, + 0.6794807009314687, + 0.2190051388373367, + 0.3714715744142361, + 0.6605358550464536, + 1.1887527234864017, + 0.3207245450707887, + 0.5634609677487896, + 0.8020954635296937, + 1.0800700868247641, + 0.31318852831038607, + 0.5066792263052481, + 0.7816374571979372, + 1.1260207789229055, + 0.5289279236114743, + 0.7602375874779914, + 1.0384659713250357, + 1.3835362862313316, + 0.8393733850392116, + 1.178911719912477, + 1.523964981175234, + 1.8894127648295875, + 0.513653932901089, + 0.8691625961282805, + 1.3958659537041542, + 2.023663368581314 + ], + "population_cv": 0.6120788655113323 + }, + "bits_per_byte": 1.843277375243919, + "branch_output_rms_by_sublayer": [ + 0.06610569357872009, + 0.0759635791182518, + 0.19478636980056763, + 0.02243208698928356, + 0.04480339214205742, + 0.02177800051867962, + 0.02529100514948368, + 0.01183365099132061, + 0.056083619594573975, + 0.005018157884478569, + 0.02013784646987915, + 0.006211217027157545, + 0.025619102641940117, + 0.011712386272847652, + 0.020297767594456673, + 0.01479379553347826, + 0.022190503776073456, + 0.01601109653711319, + 0.02057056874036789, + 0.015698160976171494, + 0.03349706903100014, + 0.029429391026496887, + 0.06081850454211235, + 0.045232925564050674, + 0.027958564460277557, + 0.02670948952436447, + 0.026062071323394775, + 0.03323914110660553, + 0.023885980248451233, + 0.038425978273153305, + 0.02981531247496605, + 0.04100729525089264, + 0.027651365846395493, + 0.025877196341753006, + 0.033546917140483856, + 0.024878641590476036, + 0.0527394600212574, + 0.03716423362493515, + 0.05634446069598198, + 0.054621174931526184, + 0.04336739704012871, + 0.048741038888692856, + 0.02310824953019619, + 0.0576188825070858, + 0.03250083699822426, + 0.07075626403093338, + 0.040191881358623505, + 0.08655129373073578, + 0.08113188296556473, + 0.06302639842033386, + 0.046195607632398605, + 0.06448276340961456, + 0.049264587461948395, + 0.07652177661657333, + 0.031975120306015015, + 0.10123477131128311, + 0.014968980103731155, + 0.06357698142528534, + 0.01199111994355917, + 0.06442654132843018, + 0.015882622450590134, + 0.08845939487218857, + 0.016016943380236626, + 0.09924950450658798 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.000360602339330905, + 0.00033878647410351273, + 0.00021687779231241842, + 0.0001579762203485444, + 0.00016425300465921556, + 0.00010846771894125893, + 0.00015794962539006875, + 0.0001703343524197592, + 0.00016087175880611854, + 0.00015224544824107882, + 0.00021883132978410406, + 0.0003212618233148454, + 0.0002048804512873905, + 0.0002555575839308094, + 0.00025751871370367904, + 0.00026900058827935755, + 0.00030863142143831173, + 0.000389505273068397, + 0.0005182477573570475, + 0.0005451795831773235, + 0.001274502598642574, + 0.0009770302920026213, + 0.0008957893162545617, + 0.0007383571746218594, + 0.00035280542140376825, + 0.0002753100463696835, + 0.00026454173894889613, + 0.00024714563943750364, + 0.0004159041412638299, + 0.0003511532692263952, + 0.00032245113080163186, + 0.0002749032683287633 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00020940594093821037, + "first_to_last_ratio": 0.6689712176385174, + "imbalance_abs_log_ratio": 0.40201424274128017, + "last_quartile_mean": 0.000313026831972559, + "mean": 0.0003645897905373823, + "normalized": [ + 0.9890631846805144, + 0.9292264426937542, + 0.5948542661953157, + 0.4332985301527435, + 0.45051454791731005, + 0.2975061884793438, + 0.4332255852728651, + 0.4671945206383786, + 0.44124043783289635, + 0.41758011933542805, + 0.6002124455036454, + 0.881159680421456, + 0.5619478564811419, + 0.700945529917702, + 0.706324533454743, + 0.7378171173769503, + 0.846516905981949, + 1.068338398873679, + 1.4214543873902312, + 1.4953232299065842, + 3.4957166429812463, + 2.679807052640011, + 2.4569786085732814, + 2.025172382182089, + 0.9676777314135849, + 0.7551227530641873, + 0.7255873472457315, + 0.6778731765177148, + 1.1407454406521187, + 0.9631461942716977, + 0.8844217231819883, + 0.7540070387697178 + ], + "population_cv": 0.7107013693076124 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.23959854245185852, + "mean_weights": [ + 0.9301985502243042, + 0.06980141997337341 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5819015502929688, + "mean_weights": [ + 0.6415372490882874, + 0.35846275091171265 + ], + "sources": 2 + }, + { + "entropy_mean": 0.590268075466156, + "mean_weights": [ + 0.7070860862731934, + 0.29291391372680664 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6727287173271179, + "mean_weights": [ + 0.400887668132782, + 0.5991123914718628 + ], + "sources": 2 + }, + { + "entropy_mean": 0.531847357749939, + "mean_weights": [ + 0.7682218551635742, + 0.2317781299352646 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6911904811859131, + "mean_weights": [ + 0.4993182420730591, + 0.5006817579269409 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6616230607032776, + "mean_weights": [ + 0.5687706470489502, + 0.4312293529510498 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6796640157699585, + "mean_weights": [ + 0.48530682921409607, + 0.5146931409835815 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5444364547729492, + "mean_weights": [ + 0.08854205906391144, + 0.08413637429475784, + 0.8273215889930725 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0843924283981323, + "mean_weights": [ + 0.28266382217407227, + 0.3129391670227051, + 0.40439704060554504 + ], + "sources": 3 + }, + { + "entropy_mean": 0.896973729133606, + "mean_weights": [ + 0.16012360155582428, + 0.22331804037094116, + 0.6165583729743958 + ], + "sources": 3 + }, + { + "entropy_mean": 1.087345004081726, + "mean_weights": [ + 0.33752816915512085, + 0.27441325783729553, + 0.388058602809906 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0189108848571777, + "mean_weights": [ + 0.3314552903175354, + 0.19555091857910156, + 0.47299379110336304 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0849429368972778, + "mean_weights": [ + 0.3367761969566345, + 0.27767041325569153, + 0.38555338978767395 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0189003944396973, + "mean_weights": [ + 0.29674339294433594, + 0.22407832741737366, + 0.4791783094406128 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0905323028564453, + "mean_weights": [ + 0.37416836619377136, + 0.2851541042327881, + 0.34067749977111816 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3338582515716553, + "mean_weights": [ + 0.24551478028297424, + 0.14209024608135223, + 0.28473812341690063, + 0.32765689492225647 + ], + "sources": 4 + }, + { + "entropy_mean": 1.372963547706604, + "mean_weights": [ + 0.26745423674583435, + 0.22373241186141968, + 0.2112794667482376, + 0.29753389954566956 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3568017482757568, + "mean_weights": [ + 0.2517681121826172, + 0.20431649684906006, + 0.28384432196617126, + 0.2600710391998291 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3636871576309204, + "mean_weights": [ + 0.28646039962768555, + 0.2041320949792862, + 0.1982249766588211, + 0.31118252873420715 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3323019742965698, + "mean_weights": [ + 0.29170483350753784, + 0.1652611941099167, + 0.20543275773525238, + 0.3376012444496155 + ], + "sources": 4 + }, + { + "entropy_mean": 1.368016004562378, + "mean_weights": [ + 0.27519717812538147, + 0.20842969417572021, + 0.2092188596725464, + 0.30715426802635193 + ], + "sources": 4 + }, + { + "entropy_mean": 1.320062279701233, + "mean_weights": [ + 0.35898226499557495, + 0.17266947031021118, + 0.1879681944847107, + 0.2803800702095032 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3811825513839722, + "mean_weights": [ + 0.2767588496208191, + 0.23899877071380615, + 0.2247239500284195, + 0.2595183849334717 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6024467945098877, + "mean_weights": [ + 0.2064993977546692, + 0.19417789578437805, + 0.21868155896663666, + 0.18105898797512054, + 0.19958214461803436 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5831859111785889, + "mean_weights": [ + 0.20024192333221436, + 0.20617860555648804, + 0.15963420271873474, + 0.15645988285541534, + 0.27748537063598633 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5467031002044678, + "mean_weights": [ + 0.18913209438323975, + 0.15018588304519653, + 0.15624433755874634, + 0.19588792324066162, + 0.30854976177215576 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5686678886413574, + "mean_weights": [ + 0.19300052523612976, + 0.19660072028636932, + 0.15589365363121033, + 0.1477280855178833, + 0.3067770004272461 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5379416942596436, + "mean_weights": [ + 0.19705462455749512, + 0.1281079351902008, + 0.1508508324623108, + 0.1935005784034729, + 0.330485999584198 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5535097122192383, + "mean_weights": [ + 0.19593553245067596, + 0.2033739686012268, + 0.14922086894512177, + 0.1313421130180359, + 0.3201275169849396 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5602667331695557, + "mean_weights": [ + 0.18746352195739746, + 0.14858682453632355, + 0.17098458111286163, + 0.2107636034488678, + 0.28220146894454956 + ], + "sources": 5 + }, + { + "entropy_mean": 1.57486891746521, + "mean_weights": [ + 0.1976916640996933, + 0.22637659311294556, + 0.14915411174297333, + 0.14752629399299622, + 0.2792513370513916 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7746952772140503, + "mean_weights": [ + 0.162063330411911, + 0.13096609711647034, + 0.16074687242507935, + 0.17189636826515198, + 0.19467724859714508, + 0.17965009808540344 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7447539567947388, + "mean_weights": [ + 0.15188820660114288, + 0.18123823404312134, + 0.12486378848552704, + 0.104622021317482, + 0.17568540573120117, + 0.261702299118042 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6408610343933105, + "mean_weights": [ + 0.12882237136363983, + 0.09651497006416321, + 0.10367561876773834, + 0.14341750741004944, + 0.14768700301647186, + 0.3798825442790985 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7601447105407715, + "mean_weights": [ + 0.16279910504817963, + 0.19839249551296234, + 0.13761474192142487, + 0.10563923418521881, + 0.1673559546470642, + 0.22819846868515015 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6253314018249512, + "mean_weights": [ + 0.13548624515533447, + 0.10138410329818726, + 0.10108494758605957, + 0.151775524020195, + 0.1173190176486969, + 0.3929501473903656 + ], + "sources": 6 + }, + { + "entropy_mean": 1.762973427772522, + "mean_weights": [ + 0.1679578423500061, + 0.20785927772521973, + 0.14729946851730347, + 0.10495810210704803, + 0.16044309735298157, + 0.2114821970462799 + ], + "sources": 6 + }, + { + "entropy_mean": 1.653536319732666, + "mean_weights": [ + 0.14313188195228577, + 0.11954564601182938, + 0.12179657816886902, + 0.14351026713848114, + 0.10692499577999115, + 0.36509066820144653 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7673754692077637, + "mean_weights": [ + 0.1787394881248474, + 0.22618836164474487, + 0.16314007341861725, + 0.11237268149852753, + 0.15502649545669556, + 0.16453291475772858 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7190462350845337, + "mean_weights": [ + 0.0917370468378067, + 0.08665085583925247, + 0.09239379316568375, + 0.09709209203720093, + 0.09174054861068726, + 0.13269288837909698, + 0.4076927900314331 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8681623935699463, + "mean_weights": [ + 0.13027030229568481, + 0.1511368602514267, + 0.11763797700405121, + 0.0868469625711441, + 0.12043553590774536, + 0.11395217478275299, + 0.2797201871871948 + ], + "sources": 7 + }, + { + "entropy_mean": 1.6680206060409546, + "mean_weights": [ + 0.08869771659374237, + 0.08410435169935226, + 0.0903121829032898, + 0.09474394470453262, + 0.082368403673172, + 0.1202085092663765, + 0.43956488370895386 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8877370357513428, + "mean_weights": [ + 0.13624337315559387, + 0.17379914224147797, + 0.1252659261226654, + 0.08305396139621735, + 0.12757685780525208, + 0.11187835037708282, + 0.2421824038028717 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7159790992736816, + "mean_weights": [ + 0.10048972070217133, + 0.09172730147838593, + 0.09713296592235565, + 0.099998340010643, + 0.08090002834796906, + 0.12119938433170319, + 0.40855225920677185 + ], + "sources": 7 + }, + { + "entropy_mean": 1.889694333076477, + "mean_weights": [ + 0.1430073082447052, + 0.17592844367027283, + 0.12485204637050629, + 0.08317119628190994, + 0.13093045353889465, + 0.10659666359424591, + 0.23551388084888458 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7411407232284546, + "mean_weights": [ + 0.10461093485355377, + 0.10913486778736115, + 0.10598915815353394, + 0.10049256682395935, + 0.08886121213436127, + 0.10488255321979523, + 0.3860287070274353 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9110310077667236, + "mean_weights": [ + 0.14464938640594482, + 0.17999428510665894, + 0.1309017539024353, + 0.08414396643638611, + 0.1571645885705948, + 0.12025413662195206, + 0.1828918755054474 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0209193229675293, + "mean_weights": [ + 0.11301086843013763, + 0.14642637968063354, + 0.10255339741706848, + 0.06968183815479279, + 0.13323475420475006, + 0.09878873080015182, + 0.13675615191459656, + 0.19954785704612732 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0430846214294434, + "mean_weights": [ + 0.12173163145780563, + 0.15324601531028748, + 0.11323399096727371, + 0.07730218023061752, + 0.14335370063781738, + 0.09678509831428528, + 0.12900850176811218, + 0.16533884406089783 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0260653495788574, + "mean_weights": [ + 0.11786322295665741, + 0.14184105396270752, + 0.11150431632995605, + 0.08538752049207687, + 0.1505657434463501, + 0.09038135409355164, + 0.1409604698419571, + 0.16149631142616272 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0095877647399902, + "mean_weights": [ + 0.10195446014404297, + 0.12748080492019653, + 0.12588977813720703, + 0.1024467796087265, + 0.13601306080818176, + 0.08495569974184036, + 0.19784167408943176, + 0.12341775000095367 + ], + "sources": 8 + }, + { + "entropy_mean": 1.972090721130371, + "mean_weights": [ + 0.08354265987873077, + 0.09572072327136993, + 0.07414155453443527, + 0.0712641179561615, + 0.05835077911615372, + 0.057575780898332596, + 0.2046716809272766, + 0.11982528865337372, + 0.2349073886871338 + ], + "sources": 9 + }, + { + "entropy_mean": 2.106501579284668, + "mean_weights": [ + 0.07960617542266846, + 0.10406342893838882, + 0.1223711222410202, + 0.11687587201595306, + 0.10722068697214127, + 0.08320080488920212, + 0.1620112657546997, + 0.07835458219051361, + 0.14629608392715454 + ], + "sources": 9 + }, + { + "entropy_mean": 1.9735889434814453, + "mean_weights": [ + 0.07542510330677032, + 0.09434033930301666, + 0.07931336760520935, + 0.07251034677028656, + 0.06785320490598679, + 0.07022887468338013, + 0.22358202934265137, + 0.10872143507003784, + 0.20802530646324158 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1063718795776367, + "mean_weights": [ + 0.10431566834449768, + 0.12426742166280746, + 0.11089345812797546, + 0.07139710336923599, + 0.12711739540100098, + 0.07425706088542938, + 0.11523042619228363, + 0.10410453379154205, + 0.16841691732406616 + ], + "sources": 9 + }, + { + "entropy_mean": 1.9621572494506836, + "mean_weights": [ + 0.08836635947227478, + 0.10066446661949158, + 0.08412620425224304, + 0.07810831069946289, + 0.06651157140731812, + 0.05520293116569519, + 0.1949501782655716, + 0.11975784599781036, + 0.21231214702129364 + ], + "sources": 9 + }, + { + "entropy_mean": 2.087174892425537, + "mean_weights": [ + 0.07850535213947296, + 0.09793666750192642, + 0.13354092836380005, + 0.12271111458539963, + 0.08401210606098175, + 0.08235839754343033, + 0.20115184783935547, + 0.09428006410598755, + 0.10550353676080704 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0476691722869873, + "mean_weights": [ + 0.08722333610057831, + 0.11430556327104568, + 0.09186933189630508, + 0.09071213006973267, + 0.080611452460289, + 0.09060157090425491, + 0.20002081990242004, + 0.11261770129203796, + 0.13203807175159454 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06186225637793541, + 0.05811265483498573, + 0.06161171570420265, + 0.09540000557899475, + 0.1696767657995224, + 0.09032459557056427, + 0.16320227086544037, + 0.14649438858032227, + 0.17013853788375854, + 0.06792896240949631, + 0.11981337517499924, + 0.0998220294713974, + 0.10833755135536194, + 0.08974390476942062, + 0.11051425337791443, + 0.09890568256378174, + 0.1115480363368988, + 0.06200679391622543, + 0.08331575244665146, + 0.07978809624910355, + 0.07551561295986176, + 0.06349637359380722, + 0.07360780239105225, + 0.06073619797825813, + 0.07854541391134262, + 0.06404656171798706, + 0.06529200077056885, + 0.050598304718732834, + 0.06376136839389801, + 0.04906720668077469, + 0.06848077476024628, + 0.05809255316853523, + 0.07706216722726822, + 0.05020545423030853, + 0.059271879494190216, + 0.04196532815694809, + 0.06368854641914368, + 0.048060089349746704, + 0.06657783687114716, + 0.055918145924806595, + 0.07142683863639832, + 0.034014686942100525, + 0.048677653074264526, + 0.042518727481365204, + 0.05560046434402466, + 0.05150723084807396, + 0.05758795142173767, + 0.060814108699560165, + 0.058777425438165665, + 0.043337807059288025, + 0.04652208089828491, + 0.04449950158596039, + 0.049873340874910355, + 0.046631574630737305, + 0.05556628108024597, + 0.04940993711352348, + 0.06154909357428551, + 0.05375954881310463, + 0.050031352788209915, + 0.06056031584739685, + 0.060143765062093735, + 0.06915795803070068, + 0.060869500041007996, + 0.06915941089391708 + ], + "loss_nats": 1.2776625156402588, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.7981228828430176, + "mean_weights": [ + 0.10267478227615356, + 0.07433073222637177, + 0.07372724264860153, + 0.052466146647930145, + 0.05574408173561096, + 0.043878890573978424, + 0.0824076309800148, + 0.12793417274951935, + 0.38683632016181946 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.06610569357872009, + 0.10425147414207458, + 0.26576411724090576, + 0.2724364697933197, + 0.30077776312828064, + 0.3075210154056549, + 0.31614044308662415, + 0.31877824664115906, + 0.056083619594573975, + 0.0578029565513134, + 0.06440960615873337, + 0.06583662331104279, + 0.07627152651548386, + 0.07787100225687027, + 0.08397338539361954, + 0.08668279647827148, + 0.022190503776073456, + 0.027938950806856155, + 0.03897896036505699, + 0.04738941788673401, + 0.06716341525316238, + 0.08426596224308014, + 0.12356331944465637, + 0.15165171027183533, + 0.027958564460277557, + 0.040915511548519135, + 0.05308084934949875, + 0.07188191264867783, + 0.07899810373783112, + 0.10232502222061157, + 0.11167287826538086, + 0.1377868354320526, + 0.027651365846395493, + 0.03995412588119507, + 0.0548207089304924, + 0.06463814526796341, + 0.08721444010734558, + 0.09971515089273453, + 0.12212793529033661, + 0.14364886283874512, + 0.04336739704012871, + 0.06747645884752274, + 0.071417897939682, + 0.09698512405157089, + 0.10353606194257736, + 0.13247931003570557, + 0.14223140478134155, + 0.17650066316127777, + 0.08113188296556473, + 0.10708064585924149, + 0.12346197664737701, + 0.15039627254009247, + 0.1633904129266739, + 0.19441545009613037, + 0.19889990985393524, + 0.24103640019893646, + 0.014968980103731155, + 0.06552792340517044, + 0.0669957846403122, + 0.11088091880083084, + 0.1122601106762886, + 0.17807358503341675, + 0.17922791838645935, + 0.2581630349159241 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 8.034013546791773, + "cross_entropy_nats": 5.5687538385391235, + "step": 0 + }, + { + "bits_per_byte": 5.132748133473664, + "cross_entropy_nats": 3.5577498972415924, + "step": 100 + }, + { + "bits_per_byte": 3.07979965727421, + "cross_entropy_nats": 2.1347544491291046, + "step": 500 + }, + { + "bits_per_byte": 1.958522584123954, + "cross_entropy_nats": 1.357544407248497, + "step": 2000 + }, + { + "bits_per_byte": 1.7997651411385192, + "cross_entropy_nats": 1.2475021332502365, + "step": 4000 + }, + { + "bits_per_byte": 1.7110799224596625, + "cross_entropy_nats": 1.1860302239656448, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 49, + 51, + 53, + 55 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.49.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "49": true, + "51": true, + "53": true, + "55": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_group_7_mlp_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "ff656dc8b36c38c5815fac06c58dd341706b597d414fb7cc5a38a4369275af03", + "final_model_state": "ec21b161e5ab67a54face7a7442bb7c7715d2a42454428153f25c3bc00fc3ed7", + "final_optimizer_state": "bc97fed9ac5fdc62624cc56c09d8b67650e581ad16341d266893437f7ae651d5", + "final_public_parameters": "5abe870fff342cb2f3083f9f68e4ae085f982b6f12f3b07d04298cb3e5553c16", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "bbf2cfc5c0019fb54bf6927274ab9b185b564a77b4b8ef2dc6f5d81575f835c2" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "add6249175ac3bb3fa911069fbe8f25d17d9ab05df34b66ee89996983522ce83", + "1": "619b5ee4f70e975312ead9adf5df4f9d8584ce2839369c749442e429923405cf", + "7999": "fe22994f7750c2f8323fdb5ac51c9be1e208733abaf6a337f5b1ae1df1cd2730" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "a773a79b4e87b74534335be7cc8dfb5c7ea11281cb6ec4f3b06fb7ace614a268", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073002, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 211.07075032580775, + "measured_steps": 7980, + "median_ms": 217.80450250662398, + "p95_ms": 220.9848367550876, + "peak_allocated_bytes": 12208699392, + "peak_reserved_bytes": 12712935424, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 8.018870646365954, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.558257579803467, + "step": 1, + "unclipped_grad_norm": 26.834362030029297 + }, + { + "bits_per_byte": 7.214330990115511, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.000593185424805, + "step": 10, + "unclipped_grad_norm": 12.741909980773926 + }, + { + "bits_per_byte": 6.684460697974439, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.633315086364746, + "step": 20, + "unclipped_grad_norm": 3.4544384479522705 + }, + { + "bits_per_byte": 6.5412727583802885, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.534064769744873, + "step": 30, + "unclipped_grad_norm": 3.060544013977051 + }, + { + "bits_per_byte": 6.411191272643091, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.443899154663086, + "step": 40, + "unclipped_grad_norm": 3.0001461505889893 + }, + { + "bits_per_byte": 6.085831742018415, + "learning_rate": 3.75e-05, + "loss_nats": 4.218377113342285, + "step": 50, + "unclipped_grad_norm": 2.9763824939727783 + }, + { + "bits_per_byte": 5.879738740114081, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.07552433013916, + "step": 60, + "unclipped_grad_norm": 2.709944725036621 + }, + { + "bits_per_byte": 5.710436954812922, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.9581732749938965, + "step": 70, + "unclipped_grad_norm": 2.5604801177978516 + }, + { + "bits_per_byte": 5.512404435789225, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.8209075927734375, + "step": 80, + "unclipped_grad_norm": 2.5550060272216797 + }, + { + "bits_per_byte": 5.3416407979557325, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.702543258666992, + "step": 90, + "unclipped_grad_norm": 2.3835558891296387 + }, + { + "bits_per_byte": 5.16998628999077, + "learning_rate": 7.5e-05, + "loss_nats": 3.583561420440674, + "step": 100, + "unclipped_grad_norm": 2.2476909160614014 + }, + { + "bits_per_byte": 5.001322904289613, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.4666528701782227, + "step": 110, + "unclipped_grad_norm": 2.1934914588928223 + }, + { + "bits_per_byte": 4.7341629906475164, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.2814717292785645, + "step": 120, + "unclipped_grad_norm": 1.8824740648269653 + }, + { + "bits_per_byte": 4.5855438389391185, + "learning_rate": 9.75e-05, + "loss_nats": 3.1784567832946777, + "step": 130, + "unclipped_grad_norm": 1.6651767492294312 + }, + { + "bits_per_byte": 4.392234651438718, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.0444650650024414, + "step": 140, + "unclipped_grad_norm": 1.4581118822097778 + }, + { + "bits_per_byte": 4.260238654832116, + "learning_rate": 0.0001125, + "loss_nats": 2.952972412109375, + "step": 150, + "unclipped_grad_norm": 1.2330926656723022 + }, + { + "bits_per_byte": 4.1533686355859665, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.8788957595825195, + "step": 160, + "unclipped_grad_norm": 1.0453938245773315 + }, + { + "bits_per_byte": 3.997261359259002, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.770690441131592, + "step": 170, + "unclipped_grad_norm": 0.8844733834266663 + }, + { + "bits_per_byte": 3.8612058841614343, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.6763839721679688, + "step": 180, + "unclipped_grad_norm": 1.075565218925476 + }, + { + "bits_per_byte": 3.873992450287782, + "learning_rate": 0.0001425, + "loss_nats": 2.6852469444274902, + "step": 190, + "unclipped_grad_norm": 0.8373674750328064 + }, + { + "bits_per_byte": 3.7486549339919844, + "learning_rate": 0.00015, + "loss_nats": 2.598369598388672, + "step": 200, + "unclipped_grad_norm": 0.7607800364494324 + }, + { + "bits_per_byte": 3.6957014757913256, + "learning_rate": 0.0001575, + "loss_nats": 2.5616650581359863, + "step": 210, + "unclipped_grad_norm": 0.8777501583099365 + }, + { + "bits_per_byte": 3.663469863227825, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.5393238067626953, + "step": 220, + "unclipped_grad_norm": 0.8002967238426208 + }, + { + "bits_per_byte": 3.6762247845464144, + "learning_rate": 0.0001725, + "loss_nats": 2.5481648445129395, + "step": 230, + "unclipped_grad_norm": 0.7535240650177002 + }, + { + "bits_per_byte": 3.664650696108649, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.540142297744751, + "step": 240, + "unclipped_grad_norm": 1.0026183128356934 + }, + { + "bits_per_byte": 3.5179955862731047, + "learning_rate": 0.0001875, + "loss_nats": 2.438488721847534, + "step": 250, + "unclipped_grad_norm": 0.8005338311195374 + }, + { + "bits_per_byte": 3.5215215745810937, + "learning_rate": 0.000195, + "loss_nats": 2.4409327507019043, + "step": 260, + "unclipped_grad_norm": 0.8481922745704651 + }, + { + "bits_per_byte": 3.4870218548180145, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.4170193672180176, + "step": 270, + "unclipped_grad_norm": 0.7716256976127625 + }, + { + "bits_per_byte": 3.5032387868984722, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.428260087966919, + "step": 280, + "unclipped_grad_norm": 1.026376485824585 + }, + { + "bits_per_byte": 3.3964330891637453, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.3542280197143555, + "step": 290, + "unclipped_grad_norm": 0.9510752558708191 + }, + { + "bits_per_byte": 3.485784955593009, + "learning_rate": 0.000225, + "loss_nats": 2.4161620140075684, + "step": 300, + "unclipped_grad_norm": 1.2242447137832642 + }, + { + "bits_per_byte": 3.396928055232929, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.3545711040496826, + "step": 310, + "unclipped_grad_norm": 1.2847355604171753 + }, + { + "bits_per_byte": 3.3619691417920454, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.3303394317626953, + "step": 320, + "unclipped_grad_norm": 0.9468339085578918 + }, + { + "bits_per_byte": 3.349740143419795, + "learning_rate": 0.0002475, + "loss_nats": 2.3218629360198975, + "step": 330, + "unclipped_grad_norm": 1.235195517539978 + }, + { + "bits_per_byte": 3.351167943387258, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.322852611541748, + "step": 340, + "unclipped_grad_norm": 0.9404550194740295 + }, + { + "bits_per_byte": 3.3171339526427146, + "learning_rate": 0.0002625, + "loss_nats": 2.299262046813965, + "step": 350, + "unclipped_grad_norm": 0.8211237788200378 + }, + { + "bits_per_byte": 3.3042951037905044, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.29036283493042, + "step": 360, + "unclipped_grad_norm": 1.1608790159225464 + }, + { + "bits_per_byte": 3.311648050045493, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.295459508895874, + "step": 370, + "unclipped_grad_norm": 1.3759337663650513 + }, + { + "bits_per_byte": 3.3618312117060536, + "learning_rate": 0.000285, + "loss_nats": 2.3302438259124756, + "step": 380, + "unclipped_grad_norm": 1.1013025045394897 + }, + { + "bits_per_byte": 3.2947854950936963, + "learning_rate": 0.0002925, + "loss_nats": 2.283771276473999, + "step": 390, + "unclipped_grad_norm": 1.1621015071868896 + }, + { + "bits_per_byte": 3.238234847767591, + "learning_rate": 0.0003, + "loss_nats": 2.2445733547210693, + "step": 400, + "unclipped_grad_norm": 1.0470796823501587 + }, + { + "bits_per_byte": 3.186815130847327, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.2089319229125977, + "step": 410, + "unclipped_grad_norm": 0.938030481338501 + }, + { + "bits_per_byte": 3.2022082660301514, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.219601631164551, + "step": 420, + "unclipped_grad_norm": 1.1575859785079956 + }, + { + "bits_per_byte": 3.193713354973434, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.2137134075164795, + "step": 430, + "unclipped_grad_norm": 1.0374184846878052 + }, + { + "bits_per_byte": 3.2054511708948206, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.2218494415283203, + "step": 440, + "unclipped_grad_norm": 1.1109672784805298 + }, + { + "bits_per_byte": 3.171924872935625, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.198610782623291, + "step": 450, + "unclipped_grad_norm": 0.925281822681427 + }, + { + "bits_per_byte": 3.122976546708259, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.164682388305664, + "step": 460, + "unclipped_grad_norm": 1.0907044410705566 + }, + { + "bits_per_byte": 3.1628847768756723, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.1923446655273438, + "step": 470, + "unclipped_grad_norm": 1.3378021717071533 + }, + { + "bits_per_byte": 3.117777167207375, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.161078453063965, + "step": 480, + "unclipped_grad_norm": 1.0543017387390137 + }, + { + "bits_per_byte": 3.088226076240222, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.1405951976776123, + "step": 490, + "unclipped_grad_norm": 1.3364945650100708 + }, + { + "bits_per_byte": 3.1093066776870795, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.1552071571350098, + "step": 500, + "unclipped_grad_norm": 0.9324634671211243 + }, + { + "bits_per_byte": 3.080656431845146, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.135348320007324, + "step": 510, + "unclipped_grad_norm": 0.890409529209137 + }, + { + "bits_per_byte": 2.976364776500732, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.063058853149414, + "step": 520, + "unclipped_grad_norm": 1.3360276222229004 + }, + { + "bits_per_byte": 3.0682681775381964, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.1267614364624023, + "step": 530, + "unclipped_grad_norm": 1.5751396417617798 + }, + { + "bits_per_byte": 2.999836280759885, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.0793280601501465, + "step": 540, + "unclipped_grad_norm": 0.8673555850982666 + }, + { + "bits_per_byte": 2.9665070749185776, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.0562260150909424, + "step": 550, + "unclipped_grad_norm": 1.0737544298171997 + }, + { + "bits_per_byte": 2.9570813937553906, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.0496926307678223, + "step": 560, + "unclipped_grad_norm": 0.8876364827156067 + }, + { + "bits_per_byte": 2.957950594072851, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.050295114517212, + "step": 570, + "unclipped_grad_norm": 1.0454248189926147 + }, + { + "bits_per_byte": 2.861897595635369, + "learning_rate": 0.000299626473885547, + "loss_nats": 1.9837162494659424, + "step": 580, + "unclipped_grad_norm": 1.0878263711929321 + }, + { + "bits_per_byte": 2.9034998549137807, + "learning_rate": 0.00029958384005397226, + "loss_nats": 2.0125527381896973, + "step": 590, + "unclipped_grad_norm": 1.1924595832824707 + }, + { + "bits_per_byte": 2.8071611932932155, + "learning_rate": 0.0002995389065559004, + "loss_nats": 1.9457758665084839, + "step": 600, + "unclipped_grad_norm": 0.826246976852417 + }, + { + "bits_per_byte": 2.871335487566767, + "learning_rate": 0.000299491674159121, + "loss_nats": 1.9902580976486206, + "step": 610, + "unclipped_grad_norm": 0.8543432950973511 + }, + { + "bits_per_byte": 2.7675332948467086, + "learning_rate": 0.00029944214367070535, + "loss_nats": 1.918307900428772, + "step": 620, + "unclipped_grad_norm": 0.9187549352645874 + }, + { + "bits_per_byte": 2.773016273738866, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.9221084117889404, + "step": 630, + "unclipped_grad_norm": 0.956667423248291 + }, + { + "bits_per_byte": 2.7873481039832164, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.9320424795150757, + "step": 640, + "unclipped_grad_norm": 1.0171226263046265 + }, + { + "bits_per_byte": 2.6969407800256078, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.8693768978118896, + "step": 650, + "unclipped_grad_norm": 1.2362606525421143 + }, + { + "bits_per_byte": 2.6987483176861757, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.8706297874450684, + "step": 660, + "unclipped_grad_norm": 1.4490057229995728 + }, + { + "bits_per_byte": 2.6868882221000256, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.862408995628357, + "step": 670, + "unclipped_grad_norm": 0.9530034065246582 + }, + { + "bits_per_byte": 2.725945998058922, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.889481782913208, + "step": 680, + "unclipped_grad_norm": 1.1275639533996582 + }, + { + "bits_per_byte": 2.5997461607516983, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.802006721496582, + "step": 690, + "unclipped_grad_norm": 0.9060716032981873 + }, + { + "bits_per_byte": 2.5382821410730934, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.759403109550476, + "step": 700, + "unclipped_grad_norm": 0.8945295214653015 + }, + { + "bits_per_byte": 2.637368397523843, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.8280844688415527, + "step": 710, + "unclipped_grad_norm": 1.6186522245407104 + }, + { + "bits_per_byte": 2.681685230963473, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.8588025569915771, + "step": 720, + "unclipped_grad_norm": 0.8582053780555725 + }, + { + "bits_per_byte": 2.4926382894994057, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.7277652025222778, + "step": 730, + "unclipped_grad_norm": 0.9802802205085754 + }, + { + "bits_per_byte": 2.531444626822741, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.7546637058258057, + "step": 740, + "unclipped_grad_norm": 1.0117162466049194 + }, + { + "bits_per_byte": 2.538161237269537, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.7593193054199219, + "step": 750, + "unclipped_grad_norm": 1.0358431339263916 + }, + { + "bits_per_byte": 2.4960984084521147, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.73016357421875, + "step": 760, + "unclipped_grad_norm": 1.0911173820495605 + }, + { + "bits_per_byte": 2.4965827115969446, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.730499267578125, + "step": 770, + "unclipped_grad_norm": 1.42961847782135 + }, + { + "bits_per_byte": 2.4269605228920113, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.6822408437728882, + "step": 780, + "unclipped_grad_norm": 1.0549261569976807 + }, + { + "bits_per_byte": 2.486838862529661, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.723745346069336, + "step": 790, + "unclipped_grad_norm": 1.0945781469345093 + }, + { + "bits_per_byte": 2.4452661842671977, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.6949293613433838, + "step": 800, + "unclipped_grad_norm": 1.151052713394165 + }, + { + "bits_per_byte": 2.481617297266815, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7201260328292847, + "step": 810, + "unclipped_grad_norm": 1.0710535049438477 + }, + { + "bits_per_byte": 2.452502870250203, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.6999454498291016, + "step": 820, + "unclipped_grad_norm": 0.9736834764480591 + }, + { + "bits_per_byte": 2.4376989476292343, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.6896841526031494, + "step": 830, + "unclipped_grad_norm": 1.0553193092346191 + }, + { + "bits_per_byte": 2.4151981134515514, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.6740877628326416, + "step": 840, + "unclipped_grad_norm": 1.0642609596252441 + }, + { + "bits_per_byte": 2.347674629035097, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.627284049987793, + "step": 850, + "unclipped_grad_norm": 1.0160051584243774 + }, + { + "bits_per_byte": 2.3946565056324975, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.6598494052886963, + "step": 860, + "unclipped_grad_norm": 1.0633609294891357 + }, + { + "bits_per_byte": 2.3470348535738874, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.626840591430664, + "step": 870, + "unclipped_grad_norm": 1.049605369567871 + }, + { + "bits_per_byte": 2.4050552646342664, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.6670572757720947, + "step": 880, + "unclipped_grad_norm": 1.0622506141662598 + }, + { + "bits_per_byte": 2.3750367248220683, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.6462500095367432, + "step": 890, + "unclipped_grad_norm": 0.9530835747718811 + }, + { + "bits_per_byte": 2.316455994285178, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.605644941329956, + "step": 900, + "unclipped_grad_norm": 0.9274279475212097 + }, + { + "bits_per_byte": 2.334761139712412, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.618333101272583, + "step": 910, + "unclipped_grad_norm": 1.019445538520813 + }, + { + "bits_per_byte": 2.3194103122616996, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.6076927185058594, + "step": 920, + "unclipped_grad_norm": 1.74617600440979 + }, + { + "bits_per_byte": 2.3170859667352888, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.6060816049575806, + "step": 930, + "unclipped_grad_norm": 1.1208544969558716 + }, + { + "bits_per_byte": 2.2560890519364274, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.5638017654418945, + "step": 940, + "unclipped_grad_norm": 1.1008156538009644 + }, + { + "bits_per_byte": 2.2529283547788705, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.5616109371185303, + "step": 950, + "unclipped_grad_norm": 0.9597600698471069 + }, + { + "bits_per_byte": 2.2565269197655238, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.5641052722930908, + "step": 960, + "unclipped_grad_norm": 1.3756563663482666 + }, + { + "bits_per_byte": 2.2534508380721916, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5619730949401855, + "step": 970, + "unclipped_grad_norm": 0.9507015943527222 + }, + { + "bits_per_byte": 2.2444729997318515, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.5557501316070557, + "step": 980, + "unclipped_grad_norm": 1.013108491897583 + }, + { + "bits_per_byte": 2.168576024010189, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.5031423568725586, + "step": 990, + "unclipped_grad_norm": 0.9590882062911987 + }, + { + "bits_per_byte": 2.22506802525235, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.5422996282577515, + "step": 1000, + "unclipped_grad_norm": 1.016539216041565 + }, + { + "bits_per_byte": 2.2112954825886053, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.5327532291412354, + "step": 1010, + "unclipped_grad_norm": 0.896763265132904 + }, + { + "bits_per_byte": 2.22288539343025, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.5407867431640625, + "step": 1020, + "unclipped_grad_norm": 0.8642303943634033 + }, + { + "bits_per_byte": 2.147789856861608, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.488734483718872, + "step": 1030, + "unclipped_grad_norm": 1.0084913969039917 + }, + { + "bits_per_byte": 2.2230642553871474, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.5409107208251953, + "step": 1040, + "unclipped_grad_norm": 0.8961561322212219 + }, + { + "bits_per_byte": 2.20677560654128, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5296202898025513, + "step": 1050, + "unclipped_grad_norm": 1.0740177631378174 + }, + { + "bits_per_byte": 2.1613443255240585, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.4981297254562378, + "step": 1060, + "unclipped_grad_norm": 1.0554207563400269 + }, + { + "bits_per_byte": 2.276886741922616, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.578217625617981, + "step": 1070, + "unclipped_grad_norm": 0.9966774582862854 + }, + { + "bits_per_byte": 2.1995875916484686, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.5246379375457764, + "step": 1080, + "unclipped_grad_norm": 1.0713834762573242 + }, + { + "bits_per_byte": 2.1015079496532194, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.4566543102264404, + "step": 1090, + "unclipped_grad_norm": 0.8892715573310852 + }, + { + "bits_per_byte": 2.2096420413832103, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5316071510314941, + "step": 1100, + "unclipped_grad_norm": 0.9605683088302612 + }, + { + "bits_per_byte": 2.234101070023617, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.5485608577728271, + "step": 1110, + "unclipped_grad_norm": 1.1295127868652344 + }, + { + "bits_per_byte": 2.1145297880459477, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.4656803607940674, + "step": 1120, + "unclipped_grad_norm": 0.896943211555481 + }, + { + "bits_per_byte": 2.115503897780435, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.466355562210083, + "step": 1130, + "unclipped_grad_norm": 0.889281690120697 + }, + { + "bits_per_byte": 2.1257258586171166, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.4734408855438232, + "step": 1140, + "unclipped_grad_norm": 0.9178597331047058 + }, + { + "bits_per_byte": 2.1134622917320423, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.4649404287338257, + "step": 1150, + "unclipped_grad_norm": 0.8654142022132874 + }, + { + "bits_per_byte": 2.065822065547139, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.431918740272522, + "step": 1160, + "unclipped_grad_norm": 0.9470662474632263 + }, + { + "bits_per_byte": 2.169789705577328, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.5039836168289185, + "step": 1170, + "unclipped_grad_norm": 0.8727414608001709 + }, + { + "bits_per_byte": 2.0843000535384903, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.4447267055511475, + "step": 1180, + "unclipped_grad_norm": 0.9810739755630493 + }, + { + "bits_per_byte": 2.1008148595702423, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.4561738967895508, + "step": 1190, + "unclipped_grad_norm": 0.987103283405304 + }, + { + "bits_per_byte": 2.0765615221802673, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.4393627643585205, + "step": 1200, + "unclipped_grad_norm": 0.8685299158096313 + }, + { + "bits_per_byte": 2.2026082949282233, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.5267317295074463, + "step": 1210, + "unclipped_grad_norm": 1.0136322975158691 + }, + { + "bits_per_byte": 2.0694029163207537, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4344007968902588, + "step": 1220, + "unclipped_grad_norm": 0.937686562538147 + }, + { + "bits_per_byte": 2.0099445583165108, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.393187403678894, + "step": 1230, + "unclipped_grad_norm": 0.9196194410324097 + }, + { + "bits_per_byte": 2.0167119036453114, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.3978781700134277, + "step": 1240, + "unclipped_grad_norm": 0.9567685127258301 + }, + { + "bits_per_byte": 2.176413617375309, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.5085749626159668, + "step": 1250, + "unclipped_grad_norm": 0.9349751472473145 + }, + { + "bits_per_byte": 2.1139615573674972, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.4652864933013916, + "step": 1260, + "unclipped_grad_norm": 0.9187795519828796 + }, + { + "bits_per_byte": 1.9787374464042, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.371556282043457, + "step": 1270, + "unclipped_grad_norm": 0.9491552114486694 + }, + { + "bits_per_byte": 2.1365107186700723, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.4809163808822632, + "step": 1280, + "unclipped_grad_norm": 0.9274782538414001 + }, + { + "bits_per_byte": 2.0288569744839413, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4062964916229248, + "step": 1290, + "unclipped_grad_norm": 0.8205472826957703 + }, + { + "bits_per_byte": 2.067187263829688, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.432865023612976, + "step": 1300, + "unclipped_grad_norm": 0.9139857888221741 + }, + { + "bits_per_byte": 2.154780091705927, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.4935797452926636, + "step": 1310, + "unclipped_grad_norm": 0.9574426412582397 + }, + { + "bits_per_byte": 2.06550131790328, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.4316964149475098, + "step": 1320, + "unclipped_grad_norm": 0.7867470979690552 + }, + { + "bits_per_byte": 1.9719267614473819, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.3668354749679565, + "step": 1330, + "unclipped_grad_norm": 0.9349241256713867 + }, + { + "bits_per_byte": 1.9901371444339904, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.379457950592041, + "step": 1340, + "unclipped_grad_norm": 0.9121277928352356 + }, + { + "bits_per_byte": 2.159952469930626, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.4971649646759033, + "step": 1350, + "unclipped_grad_norm": 0.9149333238601685 + }, + { + "bits_per_byte": 1.9883719145055345, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.3782343864440918, + "step": 1360, + "unclipped_grad_norm": 0.7868726849555969 + }, + { + "bits_per_byte": 2.0840362321520667, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4445438385009766, + "step": 1370, + "unclipped_grad_norm": 0.9473254680633545 + }, + { + "bits_per_byte": 2.060261350496792, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.4280643463134766, + "step": 1380, + "unclipped_grad_norm": 0.7921129465103149 + }, + { + "bits_per_byte": 2.0742519671618305, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.437761902809143, + "step": 1390, + "unclipped_grad_norm": 0.8285639882087708 + }, + { + "bits_per_byte": 2.0780302540186355, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.4403808116912842, + "step": 1400, + "unclipped_grad_norm": 0.8623148202896118 + }, + { + "bits_per_byte": 1.998834995018726, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.3854868412017822, + "step": 1410, + "unclipped_grad_norm": 0.8480837345123291 + }, + { + "bits_per_byte": 2.0342795874656456, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.410055160522461, + "step": 1420, + "unclipped_grad_norm": 0.8806518316268921 + }, + { + "bits_per_byte": 2.036224539264253, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.4114032983779907, + "step": 1430, + "unclipped_grad_norm": 1.0133235454559326 + }, + { + "bits_per_byte": 1.9960375252197915, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.3835477828979492, + "step": 1440, + "unclipped_grad_norm": 0.8298848271369934 + }, + { + "bits_per_byte": 2.046835008909234, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.4187579154968262, + "step": 1450, + "unclipped_grad_norm": 0.8863471746444702 + }, + { + "bits_per_byte": 1.9914307979337804, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.380354642868042, + "step": 1460, + "unclipped_grad_norm": 0.8512238264083862 + }, + { + "bits_per_byte": 2.002845630436847, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.3882668018341064, + "step": 1470, + "unclipped_grad_norm": 0.8603302240371704 + }, + { + "bits_per_byte": 1.9083834715198456, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.3227906227111816, + "step": 1480, + "unclipped_grad_norm": 0.8295111060142517 + }, + { + "bits_per_byte": 2.0046046689898716, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.3894860744476318, + "step": 1490, + "unclipped_grad_norm": 0.8453484177589417 + }, + { + "bits_per_byte": 1.8994852611468547, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.3166228532791138, + "step": 1500, + "unclipped_grad_norm": 0.7648711204528809 + }, + { + "bits_per_byte": 1.9955669806870309, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.3832216262817383, + "step": 1510, + "unclipped_grad_norm": 0.8958491086959839 + }, + { + "bits_per_byte": 1.9782583027388962, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.3712241649627686, + "step": 1520, + "unclipped_grad_norm": 0.8666713237762451 + }, + { + "bits_per_byte": 1.9281963888471938, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.3365238904953003, + "step": 1530, + "unclipped_grad_norm": 0.7709778547286987 + }, + { + "bits_per_byte": 2.0123708895548833, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3948692083358765, + "step": 1540, + "unclipped_grad_norm": 0.8703699111938477 + }, + { + "bits_per_byte": 1.9897157869393765, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.3791658878326416, + "step": 1550, + "unclipped_grad_norm": 0.7820594906806946 + }, + { + "bits_per_byte": 2.0623966870899046, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.429544448852539, + "step": 1560, + "unclipped_grad_norm": 0.8584202527999878 + }, + { + "bits_per_byte": 2.0130125568252524, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.3953139781951904, + "step": 1570, + "unclipped_grad_norm": 0.847956657409668 + }, + { + "bits_per_byte": 1.9921182125892791, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.3808311223983765, + "step": 1580, + "unclipped_grad_norm": 0.815151035785675 + }, + { + "bits_per_byte": 2.0011388746096843, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3870837688446045, + "step": 1590, + "unclipped_grad_norm": 0.917992889881134 + }, + { + "bits_per_byte": 2.0110268451383906, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.393937587738037, + "step": 1600, + "unclipped_grad_norm": 0.904597282409668 + }, + { + "bits_per_byte": 2.021122570709339, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.400935411453247, + "step": 1610, + "unclipped_grad_norm": 0.9223417639732361 + }, + { + "bits_per_byte": 2.028781990048165, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.4062445163726807, + "step": 1620, + "unclipped_grad_norm": 0.817783772945404 + }, + { + "bits_per_byte": 1.891059487115784, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.310782551765442, + "step": 1630, + "unclipped_grad_norm": 0.7642184495925903 + }, + { + "bits_per_byte": 2.022145867481973, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.4016447067260742, + "step": 1640, + "unclipped_grad_norm": 0.795114278793335 + }, + { + "bits_per_byte": 1.9590929000920465, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.3579397201538086, + "step": 1650, + "unclipped_grad_norm": 0.8510079383850098 + }, + { + "bits_per_byte": 1.9486305075094583, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.3506877422332764, + "step": 1660, + "unclipped_grad_norm": 0.8046203255653381 + }, + { + "bits_per_byte": 2.0784091317984865, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.4406434297561646, + "step": 1670, + "unclipped_grad_norm": 0.8546750545501709 + }, + { + "bits_per_byte": 2.026701343938027, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.4048023223876953, + "step": 1680, + "unclipped_grad_norm": 0.8223488330841064 + }, + { + "bits_per_byte": 1.8905978856808683, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.3104625940322876, + "step": 1690, + "unclipped_grad_norm": 0.8010519742965698 + }, + { + "bits_per_byte": 1.9240964944332752, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.3336820602416992, + "step": 1700, + "unclipped_grad_norm": 0.8066733479499817 + }, + { + "bits_per_byte": 1.9952945601680643, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3830327987670898, + "step": 1710, + "unclipped_grad_norm": 0.8274087309837341 + }, + { + "bits_per_byte": 1.9917063141404625, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.3805456161499023, + "step": 1720, + "unclipped_grad_norm": 0.8306881189346313 + }, + { + "bits_per_byte": 1.9012329488449227, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.3178342580795288, + "step": 1730, + "unclipped_grad_norm": 0.8526703715324402 + }, + { + "bits_per_byte": 1.940441037640673, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.3450112342834473, + "step": 1740, + "unclipped_grad_norm": 0.7994658946990967 + }, + { + "bits_per_byte": 2.049937919896101, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.4209086894989014, + "step": 1750, + "unclipped_grad_norm": 0.8551579713821411 + }, + { + "bits_per_byte": 1.9089099104141367, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.3231555223464966, + "step": 1760, + "unclipped_grad_norm": 0.809868335723877 + }, + { + "bits_per_byte": 1.9790287850147614, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.371758222579956, + "step": 1770, + "unclipped_grad_norm": 0.8407905697822571 + }, + { + "bits_per_byte": 1.9161674062978964, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.32818603515625, + "step": 1780, + "unclipped_grad_norm": 0.9082315564155579 + }, + { + "bits_per_byte": 1.8882346720753624, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3088245391845703, + "step": 1790, + "unclipped_grad_norm": 0.8160743117332458 + }, + { + "bits_per_byte": 1.9751423210705639, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.3690643310546875, + "step": 1800, + "unclipped_grad_norm": 0.8683345913887024 + }, + { + "bits_per_byte": 1.958146995512301, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.3572840690612793, + "step": 1810, + "unclipped_grad_norm": 0.8294835090637207 + }, + { + "bits_per_byte": 2.022228419154387, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.4017019271850586, + "step": 1820, + "unclipped_grad_norm": 0.8760423064231873 + }, + { + "bits_per_byte": 1.874588536660002, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.299365758895874, + "step": 1830, + "unclipped_grad_norm": 0.8317356705665588 + }, + { + "bits_per_byte": 1.9516531025983723, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.3527828454971313, + "step": 1840, + "unclipped_grad_norm": 0.76496821641922 + }, + { + "bits_per_byte": 1.8401930383833454, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.275524616241455, + "step": 1850, + "unclipped_grad_norm": 0.7755637764930725 + }, + { + "bits_per_byte": 1.9873211005087628, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.3775060176849365, + "step": 1860, + "unclipped_grad_norm": 0.7884970307350159 + }, + { + "bits_per_byte": 2.0034092175837244, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.3886574506759644, + "step": 1870, + "unclipped_grad_norm": 0.7917197942733765 + }, + { + "bits_per_byte": 1.862635570442386, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.2910805940628052, + "step": 1880, + "unclipped_grad_norm": 0.7773393392562866 + }, + { + "bits_per_byte": 1.9500682824706714, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.351684331893921, + "step": 1890, + "unclipped_grad_norm": 0.7382581830024719 + }, + { + "bits_per_byte": 1.8599259837780424, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.2892024517059326, + "step": 1900, + "unclipped_grad_norm": 0.7946091890335083 + }, + { + "bits_per_byte": 1.8613878363103762, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.2902157306671143, + "step": 1910, + "unclipped_grad_norm": 0.8258906602859497 + }, + { + "bits_per_byte": 1.9289986879134688, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.3370800018310547, + "step": 1920, + "unclipped_grad_norm": 0.8169587850570679 + }, + { + "bits_per_byte": 1.8860725061887151, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.307325839996338, + "step": 1930, + "unclipped_grad_norm": 0.8183650374412537 + }, + { + "bits_per_byte": 1.8587399914176925, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.2883803844451904, + "step": 1940, + "unclipped_grad_norm": 0.7905799150466919 + }, + { + "bits_per_byte": 1.9175102468358332, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.3291168212890625, + "step": 1950, + "unclipped_grad_norm": 0.878126323223114 + }, + { + "bits_per_byte": 1.8772296742293022, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.3011964559555054, + "step": 1960, + "unclipped_grad_norm": 0.7538776397705078 + }, + { + "bits_per_byte": 1.8378239773677103, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.273882508277893, + "step": 1970, + "unclipped_grad_norm": 0.7208675742149353 + }, + { + "bits_per_byte": 1.8145084613728901, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.2577214241027832, + "step": 1980, + "unclipped_grad_norm": 0.711968719959259 + }, + { + "bits_per_byte": 1.9286554105423466, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.3368420600891113, + "step": 1990, + "unclipped_grad_norm": 0.8378818035125732 + }, + { + "bits_per_byte": 1.8636444206723473, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.29177987575531, + "step": 2000, + "unclipped_grad_norm": 0.7902836799621582 + }, + { + "bits_per_byte": 1.926812100490399, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.335564374923706, + "step": 2010, + "unclipped_grad_norm": 0.753839373588562 + }, + { + "bits_per_byte": 1.9158666086415372, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.3279775381088257, + "step": 2020, + "unclipped_grad_norm": 0.8420804142951965 + }, + { + "bits_per_byte": 1.8991772402191596, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.3164093494415283, + "step": 2030, + "unclipped_grad_norm": 0.7370197176933289 + }, + { + "bits_per_byte": 1.918894191227326, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.3300760984420776, + "step": 2040, + "unclipped_grad_norm": 0.7840562462806702 + }, + { + "bits_per_byte": 1.8649888090541418, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.2927117347717285, + "step": 2050, + "unclipped_grad_norm": 0.8036782145500183 + }, + { + "bits_per_byte": 1.7600694170340785, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.219987154006958, + "step": 2060, + "unclipped_grad_norm": 0.7625702619552612 + }, + { + "bits_per_byte": 1.858101247852388, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.2879376411437988, + "step": 2070, + "unclipped_grad_norm": 0.7532007098197937 + }, + { + "bits_per_byte": 1.9669870757493002, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.363411545753479, + "step": 2080, + "unclipped_grad_norm": 0.7592506408691406 + }, + { + "bits_per_byte": 1.8238256214833828, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.2641795873641968, + "step": 2090, + "unclipped_grad_norm": 0.7544317245483398 + }, + { + "bits_per_byte": 1.9098694016233002, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.3238205909729004, + "step": 2100, + "unclipped_grad_norm": 0.748927116394043 + }, + { + "bits_per_byte": 1.8327945167258783, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.27039635181427, + "step": 2110, + "unclipped_grad_norm": 0.7777599096298218 + }, + { + "bits_per_byte": 1.875041194997073, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.2996795177459717, + "step": 2120, + "unclipped_grad_norm": 0.780890166759491 + }, + { + "bits_per_byte": 1.8952474366469443, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.313685417175293, + "step": 2130, + "unclipped_grad_norm": 0.7705488204956055 + }, + { + "bits_per_byte": 1.776157362126389, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.2311384677886963, + "step": 2140, + "unclipped_grad_norm": 0.7004607915878296 + }, + { + "bits_per_byte": 1.8291610392610997, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2678778171539307, + "step": 2150, + "unclipped_grad_norm": 0.7346043586730957 + }, + { + "bits_per_byte": 1.8790525183457971, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.302459955215454, + "step": 2160, + "unclipped_grad_norm": 0.7597460150718689 + }, + { + "bits_per_byte": 1.847169686598246, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.280360460281372, + "step": 2170, + "unclipped_grad_norm": 0.7274605631828308 + }, + { + "bits_per_byte": 1.7688768205674137, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.226091980934143, + "step": 2180, + "unclipped_grad_norm": 0.7541468739509583 + }, + { + "bits_per_byte": 1.8965374785110662, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.3145796060562134, + "step": 2190, + "unclipped_grad_norm": 0.7791039347648621 + }, + { + "bits_per_byte": 1.939485845997781, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.3443491458892822, + "step": 2200, + "unclipped_grad_norm": 0.7533175945281982 + }, + { + "bits_per_byte": 1.973650199591678, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.368030071258545, + "step": 2210, + "unclipped_grad_norm": 0.7727386951446533 + }, + { + "bits_per_byte": 1.8509264756236934, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2829644680023193, + "step": 2220, + "unclipped_grad_norm": 0.7623061537742615 + }, + { + "bits_per_byte": 1.8639873540781675, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.2920175790786743, + "step": 2230, + "unclipped_grad_norm": 0.8112621307373047 + }, + { + "bits_per_byte": 1.8222339220496475, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.2630763053894043, + "step": 2240, + "unclipped_grad_norm": 0.8195337653160095 + }, + { + "bits_per_byte": 1.834141312864785, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2713298797607422, + "step": 2250, + "unclipped_grad_norm": 0.8321537971496582 + }, + { + "bits_per_byte": 1.8567556555920375, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.2870049476623535, + "step": 2260, + "unclipped_grad_norm": 0.7240771055221558 + }, + { + "bits_per_byte": 1.9165105116863677, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.3284238576889038, + "step": 2270, + "unclipped_grad_norm": 0.7986313700675964 + }, + { + "bits_per_byte": 1.7964360084722737, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.2451945543289185, + "step": 2280, + "unclipped_grad_norm": 0.7366197109222412 + }, + { + "bits_per_byte": 1.8201660026556732, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.2616429328918457, + "step": 2290, + "unclipped_grad_norm": 0.7449442148208618 + }, + { + "bits_per_byte": 1.885300476068992, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.3067907094955444, + "step": 2300, + "unclipped_grad_norm": 0.8084490895271301 + }, + { + "bits_per_byte": 1.837178698461673, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.2734352350234985, + "step": 2310, + "unclipped_grad_norm": 0.7587404251098633 + }, + { + "bits_per_byte": 1.8443285332059927, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.2783911228179932, + "step": 2320, + "unclipped_grad_norm": 0.7079181671142578 + }, + { + "bits_per_byte": 1.7907211969667534, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.2412333488464355, + "step": 2330, + "unclipped_grad_norm": 0.7142359018325806 + }, + { + "bits_per_byte": 1.8162145292694492, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.258903980255127, + "step": 2340, + "unclipped_grad_norm": 0.7731177806854248 + }, + { + "bits_per_byte": 1.7505346988702444, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2133781909942627, + "step": 2350, + "unclipped_grad_norm": 0.7694292664527893 + }, + { + "bits_per_byte": 1.8460242821434998, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.2795665264129639, + "step": 2360, + "unclipped_grad_norm": 0.7893185019493103 + }, + { + "bits_per_byte": 1.8782056757729486, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.301872968673706, + "step": 2370, + "unclipped_grad_norm": 0.8278341293334961 + }, + { + "bits_per_byte": 1.780296812550006, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.234007716178894, + "step": 2380, + "unclipped_grad_norm": 0.731339693069458 + }, + { + "bits_per_byte": 1.7783178081865276, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.2326359748840332, + "step": 2390, + "unclipped_grad_norm": 0.7836362719535828 + }, + { + "bits_per_byte": 1.867772864206309, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.2946414947509766, + "step": 2400, + "unclipped_grad_norm": 0.7047775983810425 + }, + { + "bits_per_byte": 1.8310593837613234, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2691936492919922, + "step": 2410, + "unclipped_grad_norm": 0.7564196586608887 + }, + { + "bits_per_byte": 1.9510344810032187, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.3523540496826172, + "step": 2420, + "unclipped_grad_norm": 0.8589839339256287 + }, + { + "bits_per_byte": 1.8319258323563703, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.269794225692749, + "step": 2430, + "unclipped_grad_norm": 0.7789909243583679 + }, + { + "bits_per_byte": 1.770855824930892, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.227463722229004, + "step": 2440, + "unclipped_grad_norm": 0.6986477375030518 + }, + { + "bits_per_byte": 1.8565011212687603, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.2868285179138184, + "step": 2450, + "unclipped_grad_norm": 0.7666138410568237 + }, + { + "bits_per_byte": 1.8812803816050738, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.304004192352295, + "step": 2460, + "unclipped_grad_norm": 0.7829411625862122 + }, + { + "bits_per_byte": 1.8056468833045332, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.2515790462493896, + "step": 2470, + "unclipped_grad_norm": 0.6850736737251282 + }, + { + "bits_per_byte": 1.8303631979906307, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.2687110900878906, + "step": 2480, + "unclipped_grad_norm": 0.7524204254150391 + }, + { + "bits_per_byte": 1.7488334464879094, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.2121989727020264, + "step": 2490, + "unclipped_grad_norm": 0.7187771201133728 + }, + { + "bits_per_byte": 1.8121872115341948, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2561124563217163, + "step": 2500, + "unclipped_grad_norm": 0.7521882653236389 + }, + { + "bits_per_byte": 1.7965249235027698, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.2452561855316162, + "step": 2510, + "unclipped_grad_norm": 0.7439985275268555 + }, + { + "bits_per_byte": 1.780631490788585, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.2342396974563599, + "step": 2520, + "unclipped_grad_norm": 0.7779805064201355 + }, + { + "bits_per_byte": 1.7827133407772793, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.235682725906372, + "step": 2530, + "unclipped_grad_norm": 0.696090817451477 + }, + { + "bits_per_byte": 1.8046607347844856, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.2508955001831055, + "step": 2540, + "unclipped_grad_norm": 0.7803798913955688 + }, + { + "bits_per_byte": 1.8508544148929817, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.2829145193099976, + "step": 2550, + "unclipped_grad_norm": 0.8144960999488831 + }, + { + "bits_per_byte": 1.794123701731423, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.2435917854309082, + "step": 2560, + "unclipped_grad_norm": 0.7290325164794922 + }, + { + "bits_per_byte": 1.7777488895774733, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2322416305541992, + "step": 2570, + "unclipped_grad_norm": 0.7364250421524048 + }, + { + "bits_per_byte": 1.8159068523070556, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.2586907148361206, + "step": 2580, + "unclipped_grad_norm": 0.8186172842979431 + }, + { + "bits_per_byte": 1.7866744451919514, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.2384283542633057, + "step": 2590, + "unclipped_grad_norm": 0.7418432235717773 + }, + { + "bits_per_byte": 1.8222543879851, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.2630904912948608, + "step": 2600, + "unclipped_grad_norm": 0.7122811079025269 + }, + { + "bits_per_byte": 1.7320125113376215, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.2005395889282227, + "step": 2610, + "unclipped_grad_norm": 0.7474760413169861 + }, + { + "bits_per_byte": 1.7809893866850304, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.234487771987915, + "step": 2620, + "unclipped_grad_norm": 0.7634249329566956 + }, + { + "bits_per_byte": 1.799025551246315, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.2469894886016846, + "step": 2630, + "unclipped_grad_norm": 0.7181291580200195 + }, + { + "bits_per_byte": 1.8165791324892784, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.2591567039489746, + "step": 2640, + "unclipped_grad_norm": 0.7574933767318726 + }, + { + "bits_per_byte": 1.752634951002581, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2148339748382568, + "step": 2650, + "unclipped_grad_norm": 0.7240418195724487 + }, + { + "bits_per_byte": 1.7611136956901174, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.2207109928131104, + "step": 2660, + "unclipped_grad_norm": 0.716282844543457 + }, + { + "bits_per_byte": 1.7822940470744757, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.2353920936584473, + "step": 2670, + "unclipped_grad_norm": 0.7130604386329651 + }, + { + "bits_per_byte": 1.774190740513773, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.229775309562683, + "step": 2680, + "unclipped_grad_norm": 0.7784094214439392 + }, + { + "bits_per_byte": 1.8471057090521252, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.2803161144256592, + "step": 2690, + "unclipped_grad_norm": 0.7257634401321411 + }, + { + "bits_per_byte": 1.82973116174871, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.2682729959487915, + "step": 2700, + "unclipped_grad_norm": 0.8259572386741638 + }, + { + "bits_per_byte": 1.7027938628345864, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.1802867650985718, + "step": 2710, + "unclipped_grad_norm": 0.6848697662353516 + }, + { + "bits_per_byte": 1.826944182891478, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.266341209411621, + "step": 2720, + "unclipped_grad_norm": 0.7445659637451172 + }, + { + "bits_per_byte": 1.7494976434855416, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.2126593589782715, + "step": 2730, + "unclipped_grad_norm": 0.7294603586196899 + }, + { + "bits_per_byte": 1.770034779755673, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.2268946170806885, + "step": 2740, + "unclipped_grad_norm": 0.7387851476669312 + }, + { + "bits_per_byte": 1.7631071465962682, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.2220927476882935, + "step": 2750, + "unclipped_grad_norm": 0.7266920208930969 + }, + { + "bits_per_byte": 1.8704963814653723, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.2965292930603027, + "step": 2760, + "unclipped_grad_norm": 0.7621487975120544 + }, + { + "bits_per_byte": 1.7869714592299915, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.2386342287063599, + "step": 2770, + "unclipped_grad_norm": 0.707604169845581 + }, + { + "bits_per_byte": 1.7804779102813646, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.234133243560791, + "step": 2780, + "unclipped_grad_norm": 0.7276896834373474 + }, + { + "bits_per_byte": 1.7664747388828128, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2244269847869873, + "step": 2790, + "unclipped_grad_norm": 0.7669316530227661 + }, + { + "bits_per_byte": 1.7171017874904668, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.1902042627334595, + "step": 2800, + "unclipped_grad_norm": 0.7080867886543274 + }, + { + "bits_per_byte": 1.7840544614887075, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.236612319946289, + "step": 2810, + "unclipped_grad_norm": 0.7356947660446167 + }, + { + "bits_per_byte": 1.7979927954278838, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2462736368179321, + "step": 2820, + "unclipped_grad_norm": 0.7632336020469666 + }, + { + "bits_per_byte": 1.7178665943388538, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.1907343864440918, + "step": 2830, + "unclipped_grad_norm": 0.7156579494476318 + }, + { + "bits_per_byte": 1.7938433700105167, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.2433974742889404, + "step": 2840, + "unclipped_grad_norm": 0.7389410138130188 + }, + { + "bits_per_byte": 1.727502782266697, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.197413682937622, + "step": 2850, + "unclipped_grad_norm": 0.7383334040641785 + }, + { + "bits_per_byte": 1.7011584798075319, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.1791532039642334, + "step": 2860, + "unclipped_grad_norm": 0.7167931199073792 + }, + { + "bits_per_byte": 1.8165842919688042, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.2591602802276611, + "step": 2870, + "unclipped_grad_norm": 0.7482909560203552 + }, + { + "bits_per_byte": 1.7367072937408745, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2037937641143799, + "step": 2880, + "unclipped_grad_norm": 0.708979070186615 + }, + { + "bits_per_byte": 1.8043439427415964, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.250675916671753, + "step": 2890, + "unclipped_grad_norm": 0.8066174387931824 + }, + { + "bits_per_byte": 1.6990647630159281, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.1777019500732422, + "step": 2900, + "unclipped_grad_norm": 0.7267720699310303 + }, + { + "bits_per_byte": 1.740481281031408, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.2064096927642822, + "step": 2910, + "unclipped_grad_norm": 0.7180272340774536 + }, + { + "bits_per_byte": 1.828367167344717, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.2673275470733643, + "step": 2920, + "unclipped_grad_norm": 0.7484069466590881 + }, + { + "bits_per_byte": 1.7733461337153855, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.2291898727416992, + "step": 2930, + "unclipped_grad_norm": 0.7961791157722473 + }, + { + "bits_per_byte": 1.8172037734772122, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.259589672088623, + "step": 2940, + "unclipped_grad_norm": 0.7607107758522034 + }, + { + "bits_per_byte": 1.7670010057944532, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2247917652130127, + "step": 2950, + "unclipped_grad_norm": 0.728738009929657 + }, + { + "bits_per_byte": 1.7551761666517298, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.2165954113006592, + "step": 2960, + "unclipped_grad_norm": 0.728874921798706 + }, + { + "bits_per_byte": 1.7978065382169994, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.2461445331573486, + "step": 2970, + "unclipped_grad_norm": 0.7364332675933838 + }, + { + "bits_per_byte": 1.744931504105134, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.2094943523406982, + "step": 2980, + "unclipped_grad_norm": 0.7082704901695251 + }, + { + "bits_per_byte": 1.8601177444337544, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.2893353700637817, + "step": 2990, + "unclipped_grad_norm": 0.7957470417022705 + }, + { + "bits_per_byte": 1.7943259533288378, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.24373197555542, + "step": 3000, + "unclipped_grad_norm": 0.7535686492919922 + }, + { + "bits_per_byte": 1.7581122544672596, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.2186305522918701, + "step": 3010, + "unclipped_grad_norm": 0.7599316835403442 + }, + { + "bits_per_byte": 1.7231794823893076, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.1944169998168945, + "step": 3020, + "unclipped_grad_norm": 0.7523236274719238 + }, + { + "bits_per_byte": 1.7522151413518248, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.2145429849624634, + "step": 3030, + "unclipped_grad_norm": 0.7453410029411316 + }, + { + "bits_per_byte": 1.7299488915099188, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.1991091966629028, + "step": 3040, + "unclipped_grad_norm": 0.7408472299575806 + }, + { + "bits_per_byte": 1.7554700850020544, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.2167991399765015, + "step": 3050, + "unclipped_grad_norm": 0.7746976017951965 + }, + { + "bits_per_byte": 1.7505058057848994, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.2133581638336182, + "step": 3060, + "unclipped_grad_norm": 0.7900155186653137 + }, + { + "bits_per_byte": 1.7217955379978147, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.1934577226638794, + "step": 3070, + "unclipped_grad_norm": 0.7483907341957092 + }, + { + "bits_per_byte": 1.7383316698782738, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.204919695854187, + "step": 3080, + "unclipped_grad_norm": 0.7793744206428528 + }, + { + "bits_per_byte": 1.7138002365418536, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.1879158020019531, + "step": 3090, + "unclipped_grad_norm": 0.75126051902771 + }, + { + "bits_per_byte": 1.7998675783049392, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.2475731372833252, + "step": 3100, + "unclipped_grad_norm": 0.777847409248352 + }, + { + "bits_per_byte": 1.7818432805465643, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.2350796461105347, + "step": 3110, + "unclipped_grad_norm": 0.7636353373527527 + }, + { + "bits_per_byte": 1.6608486701503542, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.1512125730514526, + "step": 3120, + "unclipped_grad_norm": 0.6783154606819153 + }, + { + "bits_per_byte": 1.7679592931250607, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.2254559993743896, + "step": 3130, + "unclipped_grad_norm": 0.7460579872131348 + }, + { + "bits_per_byte": 1.709205376058752, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.184730887413025, + "step": 3140, + "unclipped_grad_norm": 0.7275160551071167 + }, + { + "bits_per_byte": 1.7098668213339703, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.1851893663406372, + "step": 3150, + "unclipped_grad_norm": 0.7651799917221069 + }, + { + "bits_per_byte": 1.7606921662128527, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.2204188108444214, + "step": 3160, + "unclipped_grad_norm": 0.7503833770751953 + }, + { + "bits_per_byte": 1.8047271200877189, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.250941514968872, + "step": 3170, + "unclipped_grad_norm": 0.8046275973320007 + }, + { + "bits_per_byte": 1.6806866969447285, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.1649632453918457, + "step": 3180, + "unclipped_grad_norm": 0.7404168248176575 + }, + { + "bits_per_byte": 1.7647865571819437, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.2232568264007568, + "step": 3190, + "unclipped_grad_norm": 0.7237268090248108 + }, + { + "bits_per_byte": 1.6832741759269594, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.1667567491531372, + "step": 3200, + "unclipped_grad_norm": 0.693095862865448 + }, + { + "bits_per_byte": 1.6982182644083814, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.1771152019500732, + "step": 3210, + "unclipped_grad_norm": 0.7053117156028748 + }, + { + "bits_per_byte": 1.7076264033411805, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.1836364269256592, + "step": 3220, + "unclipped_grad_norm": 0.6680545210838318 + }, + { + "bits_per_byte": 1.677434161051611, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.1627087593078613, + "step": 3230, + "unclipped_grad_norm": 0.751581609249115 + }, + { + "bits_per_byte": 1.7218875487160263, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.193521499633789, + "step": 3240, + "unclipped_grad_norm": 0.7485718131065369 + }, + { + "bits_per_byte": 1.7463651514827263, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.2104880809783936, + "step": 3250, + "unclipped_grad_norm": 0.7665358185768127 + }, + { + "bits_per_byte": 1.8398908648657795, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.2753151655197144, + "step": 3260, + "unclipped_grad_norm": 0.7642648816108704 + }, + { + "bits_per_byte": 1.6846292272331074, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.1676959991455078, + "step": 3270, + "unclipped_grad_norm": 0.7008034586906433 + }, + { + "bits_per_byte": 1.6254121608880252, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.1266498565673828, + "step": 3280, + "unclipped_grad_norm": 0.745010495185852 + }, + { + "bits_per_byte": 1.6834800391600422, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.1668994426727295, + "step": 3290, + "unclipped_grad_norm": 0.7105581760406494 + }, + { + "bits_per_byte": 1.7515800094221885, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.2141027450561523, + "step": 3300, + "unclipped_grad_norm": 0.75157630443573 + }, + { + "bits_per_byte": 1.7509660313586082, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.213677167892456, + "step": 3310, + "unclipped_grad_norm": 0.7542247176170349 + }, + { + "bits_per_byte": 1.760063569623949, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.2199831008911133, + "step": 3320, + "unclipped_grad_norm": 0.743175745010376 + }, + { + "bits_per_byte": 1.7151736899916439, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.1888678073883057, + "step": 3330, + "unclipped_grad_norm": 0.7453376054763794 + }, + { + "bits_per_byte": 1.7197749138328275, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.1920571327209473, + "step": 3340, + "unclipped_grad_norm": 0.7782391309738159 + }, + { + "bits_per_byte": 1.709533518956598, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.1849583387374878, + "step": 3350, + "unclipped_grad_norm": 0.7033495306968689 + }, + { + "bits_per_byte": 1.715980976554794, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.189427375793457, + "step": 3360, + "unclipped_grad_norm": 0.7803524136543274 + }, + { + "bits_per_byte": 1.648752958332522, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.1428284645080566, + "step": 3370, + "unclipped_grad_norm": 0.7324966788291931 + }, + { + "bits_per_byte": 1.7519470203991296, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.2143571376800537, + "step": 3380, + "unclipped_grad_norm": 0.8328567743301392 + }, + { + "bits_per_byte": 1.719849554303302, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.1921088695526123, + "step": 3390, + "unclipped_grad_norm": 0.777250349521637 + }, + { + "bits_per_byte": 1.7191795098955405, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.1916444301605225, + "step": 3400, + "unclipped_grad_norm": 0.736153244972229 + }, + { + "bits_per_byte": 1.7452570672632173, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.2097200155258179, + "step": 3410, + "unclipped_grad_norm": 0.757843017578125 + }, + { + "bits_per_byte": 1.729867027768108, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.1990524530410767, + "step": 3420, + "unclipped_grad_norm": 0.7703743577003479 + }, + { + "bits_per_byte": 1.6696000073395083, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1572785377502441, + "step": 3430, + "unclipped_grad_norm": 0.7217070460319519 + }, + { + "bits_per_byte": 1.668839500057393, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.1567513942718506, + "step": 3440, + "unclipped_grad_norm": 0.6863085627555847 + }, + { + "bits_per_byte": 1.6783556440949339, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.1633474826812744, + "step": 3450, + "unclipped_grad_norm": 0.7539138197898865 + }, + { + "bits_per_byte": 1.6634559271374345, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.1530197858810425, + "step": 3460, + "unclipped_grad_norm": 0.6869522929191589 + }, + { + "bits_per_byte": 1.7221530899289585, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.1937055587768555, + "step": 3470, + "unclipped_grad_norm": 0.7498345375061035 + }, + { + "bits_per_byte": 1.7732057958722816, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.2290925979614258, + "step": 3480, + "unclipped_grad_norm": 0.7591171860694885 + }, + { + "bits_per_byte": 1.7216890807369307, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.1933839321136475, + "step": 3490, + "unclipped_grad_norm": 0.725121021270752 + }, + { + "bits_per_byte": 1.662817355554781, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1525771617889404, + "step": 3500, + "unclipped_grad_norm": 0.7168233394622803 + }, + { + "bits_per_byte": 1.6775707152763961, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.1628034114837646, + "step": 3510, + "unclipped_grad_norm": 0.7341516017913818 + }, + { + "bits_per_byte": 1.7128661987650176, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.1872683763504028, + "step": 3520, + "unclipped_grad_norm": 0.7342941761016846 + }, + { + "bits_per_byte": 1.6782663850991362, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.1632856130599976, + "step": 3530, + "unclipped_grad_norm": 0.7482740879058838 + }, + { + "bits_per_byte": 1.6925467925308786, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.1731840372085571, + "step": 3540, + "unclipped_grad_norm": 0.7696171402931213 + }, + { + "bits_per_byte": 1.7371637356962644, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.2041101455688477, + "step": 3550, + "unclipped_grad_norm": 0.7772866487503052 + }, + { + "bits_per_byte": 1.6716954439576208, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.1587309837341309, + "step": 3560, + "unclipped_grad_norm": 0.8553851246833801 + }, + { + "bits_per_byte": 1.7126806294847368, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.1871397495269775, + "step": 3570, + "unclipped_grad_norm": 0.751911461353302 + }, + { + "bits_per_byte": 1.7349369043328928, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.2025666236877441, + "step": 3580, + "unclipped_grad_norm": 0.814304769039154 + }, + { + "bits_per_byte": 1.7016218010689563, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.1794743537902832, + "step": 3590, + "unclipped_grad_norm": 0.7460026741027832 + }, + { + "bits_per_byte": 1.7573214782385924, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.2180824279785156, + "step": 3600, + "unclipped_grad_norm": 0.8624771237373352 + }, + { + "bits_per_byte": 1.7394060454982136, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.2056643962860107, + "step": 3610, + "unclipped_grad_norm": 0.8098861575126648 + }, + { + "bits_per_byte": 1.668054743221506, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.1562074422836304, + "step": 3620, + "unclipped_grad_norm": 0.719837486743927 + }, + { + "bits_per_byte": 1.6744146616504125, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.1606158018112183, + "step": 3630, + "unclipped_grad_norm": 0.7894346117973328 + }, + { + "bits_per_byte": 1.6143411217040335, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.1189759969711304, + "step": 3640, + "unclipped_grad_norm": 0.7176504135131836 + }, + { + "bits_per_byte": 1.6741062967574156, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.1604020595550537, + "step": 3650, + "unclipped_grad_norm": 0.7702606916427612 + }, + { + "bits_per_byte": 1.6942870849749594, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.1743903160095215, + "step": 3660, + "unclipped_grad_norm": 0.7334628701210022 + }, + { + "bits_per_byte": 1.703832466063147, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.181006669998169, + "step": 3670, + "unclipped_grad_norm": 0.7452021241188049 + }, + { + "bits_per_byte": 1.6656108697527454, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.1545134782791138, + "step": 3680, + "unclipped_grad_norm": 0.7041264772415161 + }, + { + "bits_per_byte": 1.7435893514978007, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.208564043045044, + "step": 3690, + "unclipped_grad_norm": 0.7965982556343079 + }, + { + "bits_per_byte": 1.6336704238171555, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.1323740482330322, + "step": 3700, + "unclipped_grad_norm": 0.7122047543525696 + }, + { + "bits_per_byte": 1.68351546758612, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.166923999786377, + "step": 3710, + "unclipped_grad_norm": 0.7666597366333008 + }, + { + "bits_per_byte": 1.7233669434787482, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.1945469379425049, + "step": 3720, + "unclipped_grad_norm": 0.7551341652870178 + }, + { + "bits_per_byte": 1.7011268349997732, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.179131269454956, + "step": 3730, + "unclipped_grad_norm": 0.7796503305435181 + }, + { + "bits_per_byte": 1.6595391942466848, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.150304913520813, + "step": 3740, + "unclipped_grad_norm": 0.7257726192474365 + }, + { + "bits_per_byte": 1.679273171537287, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1639834642410278, + "step": 3750, + "unclipped_grad_norm": 0.7345721125602722 + }, + { + "bits_per_byte": 1.7288196534243536, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.1983264684677124, + "step": 3760, + "unclipped_grad_norm": 0.8156569600105286 + }, + { + "bits_per_byte": 1.7209237579405912, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.1928534507751465, + "step": 3770, + "unclipped_grad_norm": 0.7233296632766724 + }, + { + "bits_per_byte": 1.7669177661914355, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.2247340679168701, + "step": 3780, + "unclipped_grad_norm": 0.7918567657470703 + }, + { + "bits_per_byte": 1.664034304792286, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.1534206867218018, + "step": 3790, + "unclipped_grad_norm": 0.7050100564956665 + }, + { + "bits_per_byte": 1.6915401780753785, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.1724863052368164, + "step": 3800, + "unclipped_grad_norm": 0.7477746605873108 + }, + { + "bits_per_byte": 1.6259160700550532, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.1269991397857666, + "step": 3810, + "unclipped_grad_norm": 0.7570709586143494 + }, + { + "bits_per_byte": 1.6249213224024628, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1263096332550049, + "step": 3820, + "unclipped_grad_norm": 0.7578487396240234 + }, + { + "bits_per_byte": 1.6969046609210912, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.1762046813964844, + "step": 3830, + "unclipped_grad_norm": 0.7486853003501892 + }, + { + "bits_per_byte": 1.6482965163771321, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.1425120830535889, + "step": 3840, + "unclipped_grad_norm": 0.7303411960601807 + }, + { + "bits_per_byte": 1.6143512686804344, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1189830303192139, + "step": 3850, + "unclipped_grad_norm": 0.726355254650116 + }, + { + "bits_per_byte": 1.5970336476344549, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.1069793701171875, + "step": 3860, + "unclipped_grad_norm": 0.7231200337409973 + }, + { + "bits_per_byte": 1.5622443090873224, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.0828652381896973, + "step": 3870, + "unclipped_grad_norm": 0.7307581901550293 + }, + { + "bits_per_byte": 1.6657586028498366, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.154615879058838, + "step": 3880, + "unclipped_grad_norm": 0.740444004535675 + }, + { + "bits_per_byte": 1.6546232421544222, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.1468974351882935, + "step": 3890, + "unclipped_grad_norm": 0.7575617432594299 + }, + { + "bits_per_byte": 1.6384637522793528, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.135696530342102, + "step": 3900, + "unclipped_grad_norm": 0.7584123015403748 + }, + { + "bits_per_byte": 1.6637415903205175, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1532177925109863, + "step": 3910, + "unclipped_grad_norm": 0.7546544075012207 + }, + { + "bits_per_byte": 1.6828328684448455, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.16645085811615, + "step": 3920, + "unclipped_grad_norm": 0.7836748957633972 + }, + { + "bits_per_byte": 1.6733954924613996, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.1599093675613403, + "step": 3930, + "unclipped_grad_norm": 0.7405208945274353 + }, + { + "bits_per_byte": 1.6163761924116933, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.1203866004943848, + "step": 3940, + "unclipped_grad_norm": 0.746957540512085 + }, + { + "bits_per_byte": 1.719537405791986, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1918925046920776, + "step": 3950, + "unclipped_grad_norm": 0.765501856803894 + }, + { + "bits_per_byte": 1.692838303124091, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.1733860969543457, + "step": 3960, + "unclipped_grad_norm": 0.7456875443458557 + }, + { + "bits_per_byte": 1.649052724092976, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1430362462997437, + "step": 3970, + "unclipped_grad_norm": 0.7683944702148438 + }, + { + "bits_per_byte": 1.6322942186449514, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.1314201354980469, + "step": 3980, + "unclipped_grad_norm": 0.776586651802063 + }, + { + "bits_per_byte": 1.6717153939451208, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1587448120117188, + "step": 3990, + "unclipped_grad_norm": 0.7844317555427551 + }, + { + "bits_per_byte": 1.5924922737557714, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.1038315296173096, + "step": 4000, + "unclipped_grad_norm": 0.7318036556243896 + }, + { + "bits_per_byte": 1.6676631387254914, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.1559360027313232, + "step": 4010, + "unclipped_grad_norm": 0.7986090779304504 + }, + { + "bits_per_byte": 1.6855106383187792, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.1683069467544556, + "step": 4020, + "unclipped_grad_norm": 0.8139289617538452 + }, + { + "bits_per_byte": 1.635063827254446, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.1333398818969727, + "step": 4030, + "unclipped_grad_norm": 0.770820140838623 + }, + { + "bits_per_byte": 1.6277132887565695, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1282448768615723, + "step": 4040, + "unclipped_grad_norm": 0.7391415238380432 + }, + { + "bits_per_byte": 1.6431915553515715, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.138973593711853, + "step": 4050, + "unclipped_grad_norm": 0.7868523001670837 + }, + { + "bits_per_byte": 1.6272043920926664, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1278921365737915, + "step": 4060, + "unclipped_grad_norm": 0.7677471041679382 + }, + { + "bits_per_byte": 1.750261934385976, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.2131891250610352, + "step": 4070, + "unclipped_grad_norm": 0.7784451842308044 + }, + { + "bits_per_byte": 1.6012461906846884, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.1098992824554443, + "step": 4080, + "unclipped_grad_norm": 0.7680949568748474 + }, + { + "bits_per_byte": 1.6206953647054618, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.123380422592163, + "step": 4090, + "unclipped_grad_norm": 0.7537407279014587 + }, + { + "bits_per_byte": 1.733303069149696, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.2014341354370117, + "step": 4100, + "unclipped_grad_norm": 0.7991123795509338 + }, + { + "bits_per_byte": 1.6275762185838318, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.1281498670578003, + "step": 4110, + "unclipped_grad_norm": 0.7494040131568909 + }, + { + "bits_per_byte": 1.652798506228768, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.1456326246261597, + "step": 4120, + "unclipped_grad_norm": 0.7923789620399475 + }, + { + "bits_per_byte": 1.6989118704393111, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.1775959730148315, + "step": 4130, + "unclipped_grad_norm": 0.8111186623573303 + }, + { + "bits_per_byte": 1.691416694532059, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.172400712966919, + "step": 4140, + "unclipped_grad_norm": 0.7716047763824463 + }, + { + "bits_per_byte": 1.6243336576844645, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1259022951126099, + "step": 4150, + "unclipped_grad_norm": 0.7682421207427979 + }, + { + "bits_per_byte": 1.6382596088727788, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.1355550289154053, + "step": 4160, + "unclipped_grad_norm": 0.7744424343109131 + }, + { + "bits_per_byte": 1.66260788068603, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.1524319648742676, + "step": 4170, + "unclipped_grad_norm": 0.8312137722969055 + }, + { + "bits_per_byte": 1.629352455401943, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.1293810606002808, + "step": 4180, + "unclipped_grad_norm": 0.7480774521827698 + }, + { + "bits_per_byte": 1.6720462885653808, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.1589741706848145, + "step": 4190, + "unclipped_grad_norm": 0.762869119644165 + }, + { + "bits_per_byte": 1.6650515821721397, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1541258096694946, + "step": 4200, + "unclipped_grad_norm": 0.8252105712890625 + }, + { + "bits_per_byte": 1.6775351148676674, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.1627787351608276, + "step": 4210, + "unclipped_grad_norm": 0.7442479729652405 + }, + { + "bits_per_byte": 1.6541256963454762, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.146552562713623, + "step": 4220, + "unclipped_grad_norm": 0.7466332316398621 + }, + { + "bits_per_byte": 1.6470231568301439, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.1416294574737549, + "step": 4230, + "unclipped_grad_norm": 0.7704961895942688 + }, + { + "bits_per_byte": 1.668409543430236, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.1564533710479736, + "step": 4240, + "unclipped_grad_norm": 0.7521082162857056 + }, + { + "bits_per_byte": 1.612399953523745, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1176304817199707, + "step": 4250, + "unclipped_grad_norm": 0.7584742307662964 + }, + { + "bits_per_byte": 1.645933818719579, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1408743858337402, + "step": 4260, + "unclipped_grad_norm": 0.7271707057952881 + }, + { + "bits_per_byte": 1.5618026576399067, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.0825591087341309, + "step": 4270, + "unclipped_grad_norm": 0.7376978993415833 + }, + { + "bits_per_byte": 1.6383371730483178, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1356087923049927, + "step": 4280, + "unclipped_grad_norm": 0.7671453356742859 + }, + { + "bits_per_byte": 1.6162143567372314, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.1202744245529175, + "step": 4290, + "unclipped_grad_norm": 0.772279679775238 + }, + { + "bits_per_byte": 1.5898874245258034, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.1020259857177734, + "step": 4300, + "unclipped_grad_norm": 0.8084535598754883 + }, + { + "bits_per_byte": 1.5736372997937291, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.0907622575759888, + "step": 4310, + "unclipped_grad_norm": 0.7517771124839783 + }, + { + "bits_per_byte": 1.7023198786488085, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.1799582242965698, + "step": 4320, + "unclipped_grad_norm": 0.8091748356819153 + }, + { + "bits_per_byte": 1.638529965599935, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.135742425918579, + "step": 4330, + "unclipped_grad_norm": 0.7803901433944702 + }, + { + "bits_per_byte": 1.590539926703177, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.102478265762329, + "step": 4340, + "unclipped_grad_norm": 0.7751718163490295 + }, + { + "bits_per_byte": 1.5582900839786848, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.0801243782043457, + "step": 4350, + "unclipped_grad_norm": 0.7860615253448486 + }, + { + "bits_per_byte": 1.6481088833050408, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.142382025718689, + "step": 4360, + "unclipped_grad_norm": 0.8128257393836975 + }, + { + "bits_per_byte": 1.6067252139758759, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.1136970520019531, + "step": 4370, + "unclipped_grad_norm": 0.7521242499351501 + }, + { + "bits_per_byte": 1.5483441552966377, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.0732303857803345, + "step": 4380, + "unclipped_grad_norm": 0.7293151021003723 + }, + { + "bits_per_byte": 1.7022356071498856, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.17989981174469, + "step": 4390, + "unclipped_grad_norm": 0.8060577511787415 + }, + { + "bits_per_byte": 1.6495908578075258, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.143409252166748, + "step": 4400, + "unclipped_grad_norm": 0.7656570076942444 + }, + { + "bits_per_byte": 1.6655498159116893, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.1544711589813232, + "step": 4410, + "unclipped_grad_norm": 0.7993553280830383 + }, + { + "bits_per_byte": 1.6698579813158025, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.1574573516845703, + "step": 4420, + "unclipped_grad_norm": 0.8027517795562744 + }, + { + "bits_per_byte": 1.621821851068613, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.1241612434387207, + "step": 4430, + "unclipped_grad_norm": 0.8277782797813416 + }, + { + "bits_per_byte": 1.6340456899613383, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.132634162902832, + "step": 4440, + "unclipped_grad_norm": 0.7808631062507629 + }, + { + "bits_per_byte": 1.660436599718887, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.150926947593689, + "step": 4450, + "unclipped_grad_norm": 0.8067575693130493 + }, + { + "bits_per_byte": 1.6490031930895275, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.143001914024353, + "step": 4460, + "unclipped_grad_norm": 0.8614035844802856 + }, + { + "bits_per_byte": 1.615151847920201, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1195379495620728, + "step": 4470, + "unclipped_grad_norm": 0.7758786678314209 + }, + { + "bits_per_byte": 1.6656062262211722, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.154510259628296, + "step": 4480, + "unclipped_grad_norm": 0.8008512258529663 + }, + { + "bits_per_byte": 1.6944590676258222, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.1745095252990723, + "step": 4490, + "unclipped_grad_norm": 0.8085695505142212 + }, + { + "bits_per_byte": 1.607275214493335, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.1140782833099365, + "step": 4500, + "unclipped_grad_norm": 0.8030744194984436 + }, + { + "bits_per_byte": 1.6025374364273663, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.1107943058013916, + "step": 4510, + "unclipped_grad_norm": 0.797730028629303 + }, + { + "bits_per_byte": 1.6747144274108665, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.1608235836029053, + "step": 4520, + "unclipped_grad_norm": 0.841271698474884 + }, + { + "bits_per_byte": 1.630286321196128, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.1300283670425415, + "step": 4530, + "unclipped_grad_norm": 0.7702062726020813 + }, + { + "bits_per_byte": 1.6464501106374692, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.1412322521209717, + "step": 4540, + "unclipped_grad_norm": 0.7924251556396484 + }, + { + "bits_per_byte": 1.6550134707892301, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.1471679210662842, + "step": 4550, + "unclipped_grad_norm": 0.8080877065658569 + }, + { + "bits_per_byte": 1.6483426077275634, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.1425440311431885, + "step": 4560, + "unclipped_grad_norm": 0.8648765087127686 + }, + { + "bits_per_byte": 1.6421085805990883, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.1382229328155518, + "step": 4570, + "unclipped_grad_norm": 0.8034600019454956 + }, + { + "bits_per_byte": 1.6129965613395882, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.1180440187454224, + "step": 4580, + "unclipped_grad_norm": 0.788112998008728 + }, + { + "bits_per_byte": 1.582351488730297, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.0968024730682373, + "step": 4590, + "unclipped_grad_norm": 0.7574982047080994 + }, + { + "bits_per_byte": 1.5894184278369006, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.1017009019851685, + "step": 4600, + "unclipped_grad_norm": 0.7745283246040344 + }, + { + "bits_per_byte": 1.5942464967945722, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.1050474643707275, + "step": 4610, + "unclipped_grad_norm": 0.7411626577377319 + }, + { + "bits_per_byte": 1.6584928517988355, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.149579644203186, + "step": 4620, + "unclipped_grad_norm": 0.8009671568870544 + }, + { + "bits_per_byte": 1.6380112659249328, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.135382890701294, + "step": 4630, + "unclipped_grad_norm": 0.8536442518234253 + }, + { + "bits_per_byte": 1.6558599693967768, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.1477546691894531, + "step": 4640, + "unclipped_grad_norm": 0.8481485247612 + }, + { + "bits_per_byte": 1.6834222529893523, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.1668593883514404, + "step": 4650, + "unclipped_grad_norm": 0.8236886858940125 + }, + { + "bits_per_byte": 1.6014303841037625, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.1100269556045532, + "step": 4660, + "unclipped_grad_norm": 0.742050051689148 + }, + { + "bits_per_byte": 1.6101280627058474, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.1160557270050049, + "step": 4670, + "unclipped_grad_norm": 0.7754783034324646 + }, + { + "bits_per_byte": 1.5943364437209735, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.1051098108291626, + "step": 4680, + "unclipped_grad_norm": 0.8107611536979675 + }, + { + "bits_per_byte": 1.608078889420817, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.1146353483200073, + "step": 4690, + "unclipped_grad_norm": 0.7870569229125977 + }, + { + "bits_per_byte": 1.6247268100243368, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.126174807548523, + "step": 4700, + "unclipped_grad_norm": 0.7982705235481262 + }, + { + "bits_per_byte": 1.6073101269714603, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.1141024827957153, + "step": 4710, + "unclipped_grad_norm": 0.8157671093940735 + }, + { + "bits_per_byte": 1.6428064861962897, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.1387066841125488, + "step": 4720, + "unclipped_grad_norm": 0.8460482358932495 + }, + { + "bits_per_byte": 1.6675788672265686, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.1558775901794434, + "step": 4730, + "unclipped_grad_norm": 0.8206400871276855 + }, + { + "bits_per_byte": 1.6163629497475767, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.1203774213790894, + "step": 4740, + "unclipped_grad_norm": 0.8464535474777222 + }, + { + "bits_per_byte": 1.574961394222722, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.0916800498962402, + "step": 4750, + "unclipped_grad_norm": 0.8082135319709778 + }, + { + "bits_per_byte": 1.5501243477157185, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.0744643211364746, + "step": 4760, + "unclipped_grad_norm": 0.775844156742096 + }, + { + "bits_per_byte": 1.5682231139619172, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.0870094299316406, + "step": 4770, + "unclipped_grad_norm": 0.7340955138206482 + }, + { + "bits_per_byte": 1.597474783133918, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.1072851419448853, + "step": 4780, + "unclipped_grad_norm": 0.8426451683044434 + }, + { + "bits_per_byte": 1.5987441870799364, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.1081650257110596, + "step": 4790, + "unclipped_grad_norm": 0.8340768814086914 + }, + { + "bits_per_byte": 1.6185092932303446, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.121865153312683, + "step": 4800, + "unclipped_grad_norm": 0.834192156791687 + }, + { + "bits_per_byte": 1.5930634281392868, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.1042274236679077, + "step": 4810, + "unclipped_grad_norm": 0.7817437648773193 + }, + { + "bits_per_byte": 1.5982127606887702, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.1077966690063477, + "step": 4820, + "unclipped_grad_norm": 0.8387192487716675 + }, + { + "bits_per_byte": 1.5966601013167807, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.1067204475402832, + "step": 4830, + "unclipped_grad_norm": 0.7935706973075867 + }, + { + "bits_per_byte": 1.5492523956758442, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.0738599300384521, + "step": 4840, + "unclipped_grad_norm": 0.7711616158485413 + }, + { + "bits_per_byte": 1.6020832302464378, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.110479474067688, + "step": 4850, + "unclipped_grad_norm": 0.7958681583404541 + }, + { + "bits_per_byte": 1.5893621895100685, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.1016619205474854, + "step": 4860, + "unclipped_grad_norm": 0.7912132143974304 + }, + { + "bits_per_byte": 1.6063435844736111, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.11343252658844, + "step": 4870, + "unclipped_grad_norm": 0.7937769293785095 + }, + { + "bits_per_byte": 1.5713693645768014, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.0891902446746826, + "step": 4880, + "unclipped_grad_norm": 0.7924904227256775 + }, + { + "bits_per_byte": 1.5329680463962483, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.0625724792480469, + "step": 4890, + "unclipped_grad_norm": 0.8004396557807922 + }, + { + "bits_per_byte": 1.5228372363645237, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.0555503368377686, + "step": 4900, + "unclipped_grad_norm": 0.7636532187461853 + }, + { + "bits_per_byte": 1.579435522884918, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.0947812795639038, + "step": 4910, + "unclipped_grad_norm": 0.825004518032074 + }, + { + "bits_per_byte": 1.5900162395312996, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.102115273475647, + "step": 4920, + "unclipped_grad_norm": 0.796214759349823 + }, + { + "bits_per_byte": 1.57045218109975, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.0885545015335083, + "step": 4930, + "unclipped_grad_norm": 0.8025288581848145 + }, + { + "bits_per_byte": 1.6592824241489468, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.1501269340515137, + "step": 4940, + "unclipped_grad_norm": 0.8388831615447998 + }, + { + "bits_per_byte": 1.5234206015162504, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.0559546947479248, + "step": 4950, + "unclipped_grad_norm": 0.773041307926178 + }, + { + "bits_per_byte": 1.5897278246258026, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.1019153594970703, + "step": 4960, + "unclipped_grad_norm": 0.8400413990020752 + }, + { + "bits_per_byte": 1.683121799298295, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.1666511297225952, + "step": 4970, + "unclipped_grad_norm": 0.9151731133460999 + }, + { + "bits_per_byte": 1.5563045442744736, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.078748106956482, + "step": 4980, + "unclipped_grad_norm": 0.865057110786438 + }, + { + "bits_per_byte": 1.602423927877797, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.110715627670288, + "step": 4990, + "unclipped_grad_norm": 0.8210780024528503 + }, + { + "bits_per_byte": 1.6118439336135055, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.117245078086853, + "step": 5000, + "unclipped_grad_norm": 0.8020146489143372 + }, + { + "bits_per_byte": 1.5925098159861595, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.1038436889648438, + "step": 5010, + "unclipped_grad_norm": 0.7995685338973999 + }, + { + "bits_per_byte": 1.5929222303829285, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.1041295528411865, + "step": 5020, + "unclipped_grad_norm": 0.8250970244407654 + }, + { + "bits_per_byte": 1.574421024733711, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.0913054943084717, + "step": 5030, + "unclipped_grad_norm": 0.8047043085098267 + }, + { + "bits_per_byte": 1.577845887242993, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.093679428100586, + "step": 5040, + "unclipped_grad_norm": 0.8365565538406372 + }, + { + "bits_per_byte": 1.597595170989522, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.1073685884475708, + "step": 5050, + "unclipped_grad_norm": 0.8280006051063538 + }, + { + "bits_per_byte": 1.5495189687846815, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.0740447044372559, + "step": 5060, + "unclipped_grad_norm": 0.8379175662994385 + }, + { + "bits_per_byte": 1.623328935038124, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.1252058744430542, + "step": 5070, + "unclipped_grad_norm": 0.8445026874542236 + }, + { + "bits_per_byte": 1.6295022522908444, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.1294848918914795, + "step": 5080, + "unclipped_grad_norm": 0.8759730458259583 + }, + { + "bits_per_byte": 1.6182176106544812, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.121662974357605, + "step": 5090, + "unclipped_grad_norm": 0.8883287310600281 + }, + { + "bits_per_byte": 1.547760790144911, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.0728260278701782, + "step": 5100, + "unclipped_grad_norm": 0.8373895287513733 + }, + { + "bits_per_byte": 1.5667347761013504, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.0859777927398682, + "step": 5110, + "unclipped_grad_norm": 0.8021323084831238 + }, + { + "bits_per_byte": 1.5556002753191904, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0782599449157715, + "step": 5120, + "unclipped_grad_norm": 0.841288685798645 + }, + { + "bits_per_byte": 1.6043090297139042, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.1120222806930542, + "step": 5130, + "unclipped_grad_norm": 0.8280785083770752 + }, + { + "bits_per_byte": 1.546351736286392, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.0718493461608887, + "step": 5140, + "unclipped_grad_norm": 0.830622673034668 + }, + { + "bits_per_byte": 1.5570578282852527, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.0792702436447144, + "step": 5150, + "unclipped_grad_norm": 0.8963741064071655 + }, + { + "bits_per_byte": 1.5727983684228204, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.09018075466156, + "step": 5160, + "unclipped_grad_norm": 0.8034247756004333 + }, + { + "bits_per_byte": 1.5900793571641663, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.102159023284912, + "step": 5170, + "unclipped_grad_norm": 0.8571594953536987 + }, + { + "bits_per_byte": 1.6204088416091242, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.1231818199157715, + "step": 5180, + "unclipped_grad_norm": 0.869792640209198 + }, + { + "bits_per_byte": 1.5668850889382044, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.0860819816589355, + "step": 5190, + "unclipped_grad_norm": 0.8282527923583984 + }, + { + "bits_per_byte": 1.5182714409494178, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.0523855686187744, + "step": 5200, + "unclipped_grad_norm": 0.8308689594268799 + }, + { + "bits_per_byte": 1.6154517856633055, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.1197458505630493, + "step": 5210, + "unclipped_grad_norm": 0.8774551153182983 + }, + { + "bits_per_byte": 1.5790219046095928, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.0944945812225342, + "step": 5220, + "unclipped_grad_norm": 0.8189566135406494 + }, + { + "bits_per_byte": 1.6211201618530928, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.1236748695373535, + "step": 5230, + "unclipped_grad_norm": 0.8472607731819153 + }, + { + "bits_per_byte": 1.575514146462595, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0920631885528564, + "step": 5240, + "unclipped_grad_norm": 0.8183190822601318 + }, + { + "bits_per_byte": 1.538835062547782, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0666391849517822, + "step": 5250, + "unclipped_grad_norm": 0.8243229985237122 + }, + { + "bits_per_byte": 1.5136850076162085, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.0492064952850342, + "step": 5260, + "unclipped_grad_norm": 0.7627201080322266 + }, + { + "bits_per_byte": 1.6076872849248023, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.1143639087677002, + "step": 5270, + "unclipped_grad_norm": 0.8696957230567932 + }, + { + "bits_per_byte": 1.616622127602427, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.1205570697784424, + "step": 5280, + "unclipped_grad_norm": 0.8094338178634644 + }, + { + "bits_per_byte": 1.5786113820219834, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.0942100286483765, + "step": 5290, + "unclipped_grad_norm": 0.8488825559616089 + }, + { + "bits_per_byte": 1.5238294042773513, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.056238055229187, + "step": 5300, + "unclipped_grad_norm": 0.8374587893486023 + }, + { + "bits_per_byte": 1.5889788401812952, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.1013962030410767, + "step": 5310, + "unclipped_grad_norm": 0.8429878950119019 + }, + { + "bits_per_byte": 1.5708193640593422, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0888090133666992, + "step": 5320, + "unclipped_grad_norm": 0.8318946361541748 + }, + { + "bits_per_byte": 1.6378248367313974, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.135253667831421, + "step": 5330, + "unclipped_grad_norm": 0.8648045063018799 + }, + { + "bits_per_byte": 1.5476381665148458, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.0727410316467285, + "step": 5340, + "unclipped_grad_norm": 0.855337917804718 + }, + { + "bits_per_byte": 1.5806956397677898, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.0956547260284424, + "step": 5350, + "unclipped_grad_norm": 0.8570436835289001 + }, + { + "bits_per_byte": 1.50149040577413, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.0407538414001465, + "step": 5360, + "unclipped_grad_norm": 0.7960495352745056 + }, + { + "bits_per_byte": 1.5424843624164402, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.0691686868667603, + "step": 5370, + "unclipped_grad_norm": 0.8350602388381958 + }, + { + "bits_per_byte": 1.601012294279515, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.1097371578216553, + "step": 5380, + "unclipped_grad_norm": 0.8993886709213257 + }, + { + "bits_per_byte": 1.5149456404470327, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.0500802993774414, + "step": 5390, + "unclipped_grad_norm": 0.8403478264808655 + }, + { + "bits_per_byte": 1.5811661843005504, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.0959808826446533, + "step": 5400, + "unclipped_grad_norm": 0.8424691557884216 + }, + { + "bits_per_byte": 1.5419367696560928, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.0687891244888306, + "step": 5410, + "unclipped_grad_norm": 0.8184977173805237 + }, + { + "bits_per_byte": 1.5026990998443939, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.0415916442871094, + "step": 5420, + "unclipped_grad_norm": 0.8618150353431702 + }, + { + "bits_per_byte": 1.5652149654156757, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.084924340248108, + "step": 5430, + "unclipped_grad_norm": 0.829234778881073 + }, + { + "bits_per_byte": 1.533254053544633, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.0627707242965698, + "step": 5440, + "unclipped_grad_norm": 0.8429527878761292 + }, + { + "bits_per_byte": 1.557993929853899, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.0799190998077393, + "step": 5450, + "unclipped_grad_norm": 0.8463035821914673 + }, + { + "bits_per_byte": 1.5290898376192918, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.0598843097686768, + "step": 5460, + "unclipped_grad_norm": 0.8170950412750244 + }, + { + "bits_per_byte": 1.5569541227467825, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0791983604431152, + "step": 5470, + "unclipped_grad_norm": 0.8432405591011047 + }, + { + "bits_per_byte": 1.5204594042336945, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.0539021492004395, + "step": 5480, + "unclipped_grad_norm": 0.8466464877128601 + }, + { + "bits_per_byte": 1.6004715808252024, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.1093623638153076, + "step": 5490, + "unclipped_grad_norm": 0.9681119322776794 + }, + { + "bits_per_byte": 1.5584029045976509, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.080202579498291, + "step": 5500, + "unclipped_grad_norm": 0.8270514607429504 + }, + { + "bits_per_byte": 1.5332996289471117, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.0628023147583008, + "step": 5510, + "unclipped_grad_norm": 0.8548083305358887 + }, + { + "bits_per_byte": 1.488530825101014, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.031770944595337, + "step": 5520, + "unclipped_grad_norm": 0.7991176247596741 + }, + { + "bits_per_byte": 1.5034859204720914, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.0421370267868042, + "step": 5530, + "unclipped_grad_norm": 0.8186874985694885 + }, + { + "bits_per_byte": 1.630913197958523, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.130462884902954, + "step": 5540, + "unclipped_grad_norm": 0.8672067523002625 + }, + { + "bits_per_byte": 1.572444084162043, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.0899351835250854, + "step": 5550, + "unclipped_grad_norm": 0.8589032292366028 + }, + { + "bits_per_byte": 1.608595353321358, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.1149933338165283, + "step": 5560, + "unclipped_grad_norm": 0.9433630108833313 + }, + { + "bits_per_byte": 1.5305042229399877, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.0608646869659424, + "step": 5570, + "unclipped_grad_norm": 0.846960723400116 + }, + { + "bits_per_byte": 1.567181587028292, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.086287498474121, + "step": 5580, + "unclipped_grad_norm": 0.8285986185073853 + }, + { + "bits_per_byte": 1.5776336606518282, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.0935323238372803, + "step": 5590, + "unclipped_grad_norm": 0.8785344958305359 + }, + { + "bits_per_byte": 1.550596268109686, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.074791431427002, + "step": 5600, + "unclipped_grad_norm": 0.8857266306877136 + }, + { + "bits_per_byte": 1.499423174310759, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.039320945739746, + "step": 5610, + "unclipped_grad_norm": 0.8646987676620483 + }, + { + "bits_per_byte": 1.555049242905826, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.0778779983520508, + "step": 5620, + "unclipped_grad_norm": 0.8930706977844238 + }, + { + "bits_per_byte": 1.4813082416953796, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0267646312713623, + "step": 5630, + "unclipped_grad_norm": 0.8334411978721619 + }, + { + "bits_per_byte": 1.4680377163895035, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.017566204071045, + "step": 5640, + "unclipped_grad_norm": 0.8495423793792725 + }, + { + "bits_per_byte": 1.5551015256316882, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.0779142379760742, + "step": 5650, + "unclipped_grad_norm": 0.897169828414917 + }, + { + "bits_per_byte": 1.4837543509386013, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.028460144996643, + "step": 5660, + "unclipped_grad_norm": 0.9015696048736572 + }, + { + "bits_per_byte": 1.5809952335455928, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.0958623886108398, + "step": 5670, + "unclipped_grad_norm": 0.9271894693374634 + }, + { + "bits_per_byte": 1.570284326032508, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0884381532669067, + "step": 5680, + "unclipped_grad_norm": 0.9326921701431274 + }, + { + "bits_per_byte": 1.5826878867953844, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.0970356464385986, + "step": 5690, + "unclipped_grad_norm": 0.8946327567100525 + }, + { + "bits_per_byte": 1.5807462026671435, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.0956897735595703, + "step": 5700, + "unclipped_grad_norm": 0.8683182001113892 + }, + { + "bits_per_byte": 1.56181727616523, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.0825692415237427, + "step": 5710, + "unclipped_grad_norm": 0.8785070776939392 + }, + { + "bits_per_byte": 1.5764026088369523, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0926790237426758, + "step": 5720, + "unclipped_grad_norm": 0.8550999760627747 + }, + { + "bits_per_byte": 1.515511979316324, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.0504728555679321, + "step": 5730, + "unclipped_grad_norm": 0.8619773387908936 + }, + { + "bits_per_byte": 1.5934247636887497, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.104477882385254, + "step": 5740, + "unclipped_grad_norm": 0.8648645877838135 + }, + { + "bits_per_byte": 1.4888108128566186, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.0319650173187256, + "step": 5750, + "unclipped_grad_norm": 0.8336424231529236 + }, + { + "bits_per_byte": 1.4953334268732417, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.0364861488342285, + "step": 5760, + "unclipped_grad_norm": 0.8543860912322998 + }, + { + "bits_per_byte": 1.5097660389509977, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.0464900732040405, + "step": 5770, + "unclipped_grad_norm": 0.8796142935752869 + }, + { + "bits_per_byte": 1.5944840048354136, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.1052120923995972, + "step": 5780, + "unclipped_grad_norm": 0.8971201777458191 + }, + { + "bits_per_byte": 1.4472469057093493, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 1.0031551122665405, + "step": 5790, + "unclipped_grad_norm": 0.8133521676063538 + }, + { + "bits_per_byte": 1.5669862147369118, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.0861520767211914, + "step": 5800, + "unclipped_grad_norm": 0.8655558824539185 + }, + { + "bits_per_byte": 1.473167958864741, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.0211222171783447, + "step": 5810, + "unclipped_grad_norm": 0.8204559087753296 + }, + { + "bits_per_byte": 1.488298820505, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.031610131263733, + "step": 5820, + "unclipped_grad_norm": 0.9307176470756531 + }, + { + "bits_per_byte": 1.562355065914478, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.082942008972168, + "step": 5830, + "unclipped_grad_norm": 0.9166134595870972 + }, + { + "bits_per_byte": 1.5226055757338115, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.0553897619247437, + "step": 5840, + "unclipped_grad_norm": 0.8916307091712952 + }, + { + "bits_per_byte": 1.4832148413628445, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.0280861854553223, + "step": 5850, + "unclipped_grad_norm": 0.8169286251068115 + }, + { + "bits_per_byte": 1.5023190181859871, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.0413281917572021, + "step": 5860, + "unclipped_grad_norm": 0.8052369952201843 + }, + { + "bits_per_byte": 1.5721078580796062, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0897021293640137, + "step": 5870, + "unclipped_grad_norm": 0.8723217248916626 + }, + { + "bits_per_byte": 1.5060283399997962, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.0438992977142334, + "step": 5880, + "unclipped_grad_norm": 0.8689809441566467 + }, + { + "bits_per_byte": 1.5592631618172665, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.080798864364624, + "step": 5890, + "unclipped_grad_norm": 0.9391161799430847 + }, + { + "bits_per_byte": 1.5736947419991174, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.0908020734786987, + "step": 5900, + "unclipped_grad_norm": 0.9239736199378967 + }, + { + "bits_per_byte": 1.5169265366196707, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.0514533519744873, + "step": 5910, + "unclipped_grad_norm": 0.8326829671859741 + }, + { + "bits_per_byte": 1.4908610180375543, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.0333861112594604, + "step": 5920, + "unclipped_grad_norm": 0.8875053524971008 + }, + { + "bits_per_byte": 1.5937785320015745, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.1047230958938599, + "step": 5930, + "unclipped_grad_norm": 0.9360204339027405 + }, + { + "bits_per_byte": 1.5267986847444976, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.0582962036132812, + "step": 5940, + "unclipped_grad_norm": 0.9076133370399475 + }, + { + "bits_per_byte": 1.5328184214899976, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.0624687671661377, + "step": 5950, + "unclipped_grad_norm": 0.9003602266311646 + }, + { + "bits_per_byte": 1.4793784243700479, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.025426983833313, + "step": 5960, + "unclipped_grad_norm": 0.8468102216720581 + }, + { + "bits_per_byte": 1.5094624895722248, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.0462796688079834, + "step": 5970, + "unclipped_grad_norm": 0.8730514049530029 + }, + { + "bits_per_byte": 1.500569266696109, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.0401153564453125, + "step": 5980, + "unclipped_grad_norm": 0.9051312208175659 + }, + { + "bits_per_byte": 1.5050642052590593, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0432310104370117, + "step": 5990, + "unclipped_grad_norm": 0.867213249206543 + }, + { + "bits_per_byte": 1.5758593156428766, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.0923024415969849, + "step": 6000, + "unclipped_grad_norm": 0.8903218507766724 + }, + { + "bits_per_byte": 1.550350676884254, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0746212005615234, + "step": 6010, + "unclipped_grad_norm": 0.9406603574752808 + }, + { + "bits_per_byte": 1.5041413463545295, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0425913333892822, + "step": 6020, + "unclipped_grad_norm": 0.8609697818756104 + }, + { + "bits_per_byte": 1.503684216468536, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.0422744750976562, + "step": 6030, + "unclipped_grad_norm": 0.9095016121864319 + }, + { + "bits_per_byte": 1.5255789837845786, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.057450771331787, + "step": 6040, + "unclipped_grad_norm": 0.8473249077796936 + }, + { + "bits_per_byte": 1.5408193983734373, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.0680146217346191, + "step": 6050, + "unclipped_grad_norm": 0.8645480275154114 + }, + { + "bits_per_byte": 1.508293695476961, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.0454695224761963, + "step": 6060, + "unclipped_grad_norm": 0.8711590766906738 + }, + { + "bits_per_byte": 1.4219986486841332, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 0.985654354095459, + "step": 6070, + "unclipped_grad_norm": 0.8984127640724182 + }, + { + "bits_per_byte": 1.4957408537731356, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.0367685556411743, + "step": 6080, + "unclipped_grad_norm": 0.9123033285140991 + }, + { + "bits_per_byte": 1.4926697595766785, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.034639835357666, + "step": 6090, + "unclipped_grad_norm": 0.9221343398094177 + }, + { + "bits_per_byte": 1.587376821788508, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.1002857685089111, + "step": 6100, + "unclipped_grad_norm": 0.9512685537338257 + }, + { + "bits_per_byte": 1.5251565943940595, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.0571579933166504, + "step": 6110, + "unclipped_grad_norm": 0.9248404502868652 + }, + { + "bits_per_byte": 1.4807453424791055, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0263744592666626, + "step": 6120, + "unclipped_grad_norm": 0.859676718711853 + }, + { + "bits_per_byte": 1.5412173662275337, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.0682904720306396, + "step": 6130, + "unclipped_grad_norm": 0.8665011525154114 + }, + { + "bits_per_byte": 1.502826882953985, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.0416802167892456, + "step": 6140, + "unclipped_grad_norm": 0.8677138090133667 + }, + { + "bits_per_byte": 1.4739276062336022, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.0216487646102905, + "step": 6150, + "unclipped_grad_norm": 0.9414398670196533 + }, + { + "bits_per_byte": 1.541046587455227, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 1.0681720972061157, + "step": 6160, + "unclipped_grad_norm": 0.95173180103302 + }, + { + "bits_per_byte": 1.4792609602195086, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 1.0253455638885498, + "step": 6170, + "unclipped_grad_norm": 0.8647065758705139 + }, + { + "bits_per_byte": 1.4816680294009845, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0270140171051025, + "step": 6180, + "unclipped_grad_norm": 0.8885964751243591 + }, + { + "bits_per_byte": 1.4940488884539473, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.0355957746505737, + "step": 6190, + "unclipped_grad_norm": 0.9163282513618469 + }, + { + "bits_per_byte": 1.4879209746210544, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.0313482284545898, + "step": 6200, + "unclipped_grad_norm": 0.8809823393821716 + }, + { + "bits_per_byte": 1.5068627998217825, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.0444777011871338, + "step": 6210, + "unclipped_grad_norm": 0.8927851915359497 + }, + { + "bits_per_byte": 1.5389652534144853, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0667294263839722, + "step": 6220, + "unclipped_grad_norm": 0.9621226191520691 + }, + { + "bits_per_byte": 1.489952261710395, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.0327562093734741, + "step": 6230, + "unclipped_grad_norm": 0.9074224829673767 + }, + { + "bits_per_byte": 1.53872499365123, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.0665628910064697, + "step": 6240, + "unclipped_grad_norm": 0.9124370217323303 + }, + { + "bits_per_byte": 1.482398267736548, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.0275201797485352, + "step": 6250, + "unclipped_grad_norm": 0.922465980052948 + }, + { + "bits_per_byte": 1.4465175272870403, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 1.0026495456695557, + "step": 6260, + "unclipped_grad_norm": 0.8908060193061829 + }, + { + "bits_per_byte": 1.5105397888972294, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.0470263957977295, + "step": 6270, + "unclipped_grad_norm": 0.9079294204711914 + }, + { + "bits_per_byte": 1.508445556157673, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.0455747842788696, + "step": 6280, + "unclipped_grad_norm": 0.8567790985107422 + }, + { + "bits_per_byte": 1.4956870232034154, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 1.036731243133545, + "step": 6290, + "unclipped_grad_norm": 0.9134734272956848 + }, + { + "bits_per_byte": 1.4635807859923937, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.0144768953323364, + "step": 6300, + "unclipped_grad_norm": 0.8859038949012756 + }, + { + "bits_per_byte": 1.539502183250479, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.0671015977859497, + "step": 6310, + "unclipped_grad_norm": 0.9397910237312317 + }, + { + "bits_per_byte": 1.4988112600389893, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 1.0388967990875244, + "step": 6320, + "unclipped_grad_norm": 0.907313883304596 + }, + { + "bits_per_byte": 1.5023854034892201, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 1.0413742065429688, + "step": 6330, + "unclipped_grad_norm": 0.8601198792457581 + }, + { + "bits_per_byte": 1.436187647336942, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 0.9954894185066223, + "step": 6340, + "unclipped_grad_norm": 0.8679407238960266 + }, + { + "bits_per_byte": 1.5283331139554956, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.0593597888946533, + "step": 6350, + "unclipped_grad_norm": 0.9039879441261292 + }, + { + "bits_per_byte": 1.5491884181297233, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.0738155841827393, + "step": 6360, + "unclipped_grad_norm": 0.9510418176651001 + }, + { + "bits_per_byte": 1.4693238026526556, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 1.0184576511383057, + "step": 6370, + "unclipped_grad_norm": 0.9187111258506775 + }, + { + "bits_per_byte": 1.4539937851026974, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.0078316926956177, + "step": 6380, + "unclipped_grad_norm": 0.8908602595329285 + }, + { + "bits_per_byte": 1.5747139111881303, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0915085077285767, + "step": 6390, + "unclipped_grad_norm": 0.9649614691734314 + }, + { + "bits_per_byte": 1.5357963010896871, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.0645328760147095, + "step": 6400, + "unclipped_grad_norm": 0.948226809501648 + }, + { + "bits_per_byte": 1.5501198761667963, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.0744612216949463, + "step": 6410, + "unclipped_grad_norm": 0.9891357421875 + }, + { + "bits_per_byte": 1.4775098328684235, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0241317749023438, + "step": 6420, + "unclipped_grad_norm": 0.9163603186607361 + }, + { + "bits_per_byte": 1.5326294125566995, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.0623377561569214, + "step": 6430, + "unclipped_grad_norm": 0.9027984738349915 + }, + { + "bits_per_byte": 1.4444766091692514, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.0012348890304565, + "step": 6440, + "unclipped_grad_norm": 0.9175200462341309 + }, + { + "bits_per_byte": 1.4695881399870316, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0186408758163452, + "step": 6450, + "unclipped_grad_norm": 0.9318752288818359 + }, + { + "bits_per_byte": 1.486625257329454, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.0304501056671143, + "step": 6460, + "unclipped_grad_norm": 0.9268568754196167 + }, + { + "bits_per_byte": 1.5084352371986212, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.0455676317214966, + "step": 6470, + "unclipped_grad_norm": 0.9912216663360596 + }, + { + "bits_per_byte": 1.461071043168353, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 1.0127372741699219, + "step": 6480, + "unclipped_grad_norm": 0.8796574473381042 + }, + { + "bits_per_byte": 1.5290545811758651, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0598598718643188, + "step": 6490, + "unclipped_grad_norm": 0.9140607118606567 + }, + { + "bits_per_byte": 1.5682970665017881, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.0870606899261475, + "step": 6500, + "unclipped_grad_norm": 0.9787759184837341 + }, + { + "bits_per_byte": 1.5381340612628653, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 1.0661532878875732, + "step": 6510, + "unclipped_grad_norm": 0.9421491026878357 + }, + { + "bits_per_byte": 1.5206855614195791, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 1.0540589094161987, + "step": 6520, + "unclipped_grad_norm": 0.926185667514801 + }, + { + "bits_per_byte": 1.4962134620977066, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.0370961427688599, + "step": 6530, + "unclipped_grad_norm": 0.9024524688720703 + }, + { + "bits_per_byte": 1.470436530403738, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.0192289352416992, + "step": 6540, + "unclipped_grad_norm": 0.9008923172950745 + }, + { + "bits_per_byte": 1.5157969545688037, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 1.0506703853607178, + "step": 6550, + "unclipped_grad_norm": 0.8769800662994385 + }, + { + "bits_per_byte": 1.5385136269733195, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.0664163827896118, + "step": 6560, + "unclipped_grad_norm": 0.9339950680732727 + }, + { + "bits_per_byte": 1.509519587812311, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 1.0463192462921143, + "step": 6570, + "unclipped_grad_norm": 0.9148213863372803 + }, + { + "bits_per_byte": 1.491253654429474, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.0336582660675049, + "step": 6580, + "unclipped_grad_norm": 0.9044763445854187 + }, + { + "bits_per_byte": 1.5132161829099564, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.0488815307617188, + "step": 6590, + "unclipped_grad_norm": 0.9223966002464294 + }, + { + "bits_per_byte": 1.4243855958954579, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9873088598251343, + "step": 6600, + "unclipped_grad_norm": 0.9176822304725647 + }, + { + "bits_per_byte": 1.4558451783392354, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.0091149806976318, + "step": 6610, + "unclipped_grad_norm": 0.895537257194519 + }, + { + "bits_per_byte": 1.5796489533546387, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 1.0949292182922363, + "step": 6620, + "unclipped_grad_norm": 0.9994770884513855 + }, + { + "bits_per_byte": 1.516109962993374, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.0508873462677002, + "step": 6630, + "unclipped_grad_norm": 0.913341760635376 + }, + { + "bits_per_byte": 1.4750147085697058, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.022402286529541, + "step": 6640, + "unclipped_grad_norm": 0.9091306924819946 + }, + { + "bits_per_byte": 1.4842281631417282, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.0287885665893555, + "step": 6650, + "unclipped_grad_norm": 0.9299202561378479 + }, + { + "bits_per_byte": 1.5221212725889819, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.0550540685653687, + "step": 6660, + "unclipped_grad_norm": 0.976958692073822 + }, + { + "bits_per_byte": 1.4811061620806156, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.0266245603561401, + "step": 6670, + "unclipped_grad_norm": 0.9014349579811096 + }, + { + "bits_per_byte": 1.492375669243703, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.0344359874725342, + "step": 6680, + "unclipped_grad_norm": 0.9197087287902832 + }, + { + "bits_per_byte": 1.491835299754692, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.0340614318847656, + "step": 6690, + "unclipped_grad_norm": 0.986449122428894 + }, + { + "bits_per_byte": 1.509621401541622, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.0463898181915283, + "step": 6700, + "unclipped_grad_norm": 0.9386336207389832 + }, + { + "bits_per_byte": 1.513041104571378, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.048760175704956, + "step": 6710, + "unclipped_grad_norm": 0.9825718998908997 + }, + { + "bits_per_byte": 1.572084468439089, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.0896859169006348, + "step": 6720, + "unclipped_grad_norm": 0.9967424273490906 + }, + { + "bits_per_byte": 1.447935524243404, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 1.0036324262619019, + "step": 6730, + "unclipped_grad_norm": 0.8940417766571045 + }, + { + "bits_per_byte": 1.4731951321235774, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 1.0211410522460938, + "step": 6740, + "unclipped_grad_norm": 0.9541178345680237 + }, + { + "bits_per_byte": 1.461154970701974, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 1.0127954483032227, + "step": 6750, + "unclipped_grad_norm": 0.9349017143249512 + }, + { + "bits_per_byte": 1.4533552135200438, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 1.0073890686035156, + "step": 6760, + "unclipped_grad_norm": 0.9357563257217407 + }, + { + "bits_per_byte": 1.4802046290247928, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 1.025999665260315, + "step": 6770, + "unclipped_grad_norm": 0.9544798135757446 + }, + { + "bits_per_byte": 1.4630825522528443, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 1.0141315460205078, + "step": 6780, + "unclipped_grad_norm": 0.9021315574645996 + }, + { + "bits_per_byte": 1.492706563863963, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.0346653461456299, + "step": 6790, + "unclipped_grad_norm": 0.9396920204162598 + }, + { + "bits_per_byte": 1.473484062977027, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.021341323852539, + "step": 6800, + "unclipped_grad_norm": 0.9342244863510132 + }, + { + "bits_per_byte": 1.4678134510127785, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0174107551574707, + "step": 6810, + "unclipped_grad_norm": 0.9041791558265686 + }, + { + "bits_per_byte": 1.4974722031193715, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 1.037968635559082, + "step": 6820, + "unclipped_grad_norm": 0.9535414576530457 + }, + { + "bits_per_byte": 1.4836444540247, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.0283839702606201, + "step": 6830, + "unclipped_grad_norm": 0.9237582087516785 + }, + { + "bits_per_byte": 1.4709160180343435, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 1.0195612907409668, + "step": 6840, + "unclipped_grad_norm": 0.9728788733482361 + }, + { + "bits_per_byte": 1.5421921638926241, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.0689661502838135, + "step": 6850, + "unclipped_grad_norm": 0.991144061088562 + }, + { + "bits_per_byte": 1.5002885910099009, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.0399208068847656, + "step": 6860, + "unclipped_grad_norm": 0.9714362025260925 + }, + { + "bits_per_byte": 1.425529366515021, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 0.9881016612052917, + "step": 6870, + "unclipped_grad_norm": 0.9117473363876343 + }, + { + "bits_per_byte": 1.465368201682811, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 1.0157158374786377, + "step": 6880, + "unclipped_grad_norm": 1.0009933710098267 + }, + { + "bits_per_byte": 1.4553990553428975, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 1.008805751800537, + "step": 6890, + "unclipped_grad_norm": 0.9414173364639282 + }, + { + "bits_per_byte": 1.5135267835774147, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 1.0490968227386475, + "step": 6900, + "unclipped_grad_norm": 0.9642160534858704 + }, + { + "bits_per_byte": 1.4983899025443754, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.038604736328125, + "step": 6910, + "unclipped_grad_norm": 0.9923614263534546 + }, + { + "bits_per_byte": 1.4505309144275746, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 1.0054314136505127, + "step": 6920, + "unclipped_grad_norm": 0.9207749366760254 + }, + { + "bits_per_byte": 1.4341909287604246, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.9941053986549377, + "step": 6930, + "unclipped_grad_norm": 0.931898295879364 + }, + { + "bits_per_byte": 1.5079796551564857, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 1.0452518463134766, + "step": 6940, + "unclipped_grad_norm": 0.9550427198410034 + }, + { + "bits_per_byte": 1.4279730680011307, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 0.9897955060005188, + "step": 6950, + "unclipped_grad_norm": 0.9696134328842163 + }, + { + "bits_per_byte": 1.4334182967014235, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 0.9935698509216309, + "step": 6960, + "unclipped_grad_norm": 0.9392643570899963 + }, + { + "bits_per_byte": 1.4066393940835282, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 0.9750081300735474, + "step": 6970, + "unclipped_grad_norm": 0.9234665632247925 + }, + { + "bits_per_byte": 1.5240345795798305, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.056380271911621, + "step": 6980, + "unclipped_grad_norm": 1.0241727828979492 + }, + { + "bits_per_byte": 1.4838109332307352, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.0284993648529053, + "step": 6990, + "unclipped_grad_norm": 0.9986793994903564 + }, + { + "bits_per_byte": 1.5010375754544083, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 1.0404399633407593, + "step": 7000, + "unclipped_grad_norm": 0.9659420847892761 + }, + { + "bits_per_byte": 1.3970830920970108, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 0.9683842062950134, + "step": 7010, + "unclipped_grad_norm": 0.9124282598495483 + }, + { + "bits_per_byte": 1.4958698407612827, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 1.0368579626083374, + "step": 7020, + "unclipped_grad_norm": 0.9753313064575195 + }, + { + "bits_per_byte": 1.463588181246381, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 1.014482021331787, + "step": 7030, + "unclipped_grad_norm": 0.9809290766716003 + }, + { + "bits_per_byte": 1.4501986439461079, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.0052011013031006, + "step": 7040, + "unclipped_grad_norm": 0.9492007493972778 + }, + { + "bits_per_byte": 1.4550148461008698, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.0085394382476807, + "step": 7050, + "unclipped_grad_norm": 0.9695240259170532 + }, + { + "bits_per_byte": 1.4262209947454658, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 0.9885810613632202, + "step": 7060, + "unclipped_grad_norm": 0.9257261157035828 + }, + { + "bits_per_byte": 1.471846788140813, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 1.0202064514160156, + "step": 7070, + "unclipped_grad_norm": 0.9551657438278198 + }, + { + "bits_per_byte": 1.4798092409104593, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 1.0257256031036377, + "step": 7080, + "unclipped_grad_norm": 0.9489098191261292 + }, + { + "bits_per_byte": 1.451192703668095, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 1.005890130996704, + "step": 7090, + "unclipped_grad_norm": 0.9999250173568726 + }, + { + "bits_per_byte": 1.434179835879444, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 0.9940977096557617, + "step": 7100, + "unclipped_grad_norm": 0.9532710313796997 + }, + { + "bits_per_byte": 1.4092086429047677, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 0.9767889976501465, + "step": 7110, + "unclipped_grad_norm": 0.9028475284576416 + }, + { + "bits_per_byte": 1.4770217461052748, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.0237934589385986, + "step": 7120, + "unclipped_grad_norm": 1.004341959953308 + }, + { + "bits_per_byte": 1.5147215470529585, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 1.0499249696731567, + "step": 7130, + "unclipped_grad_norm": 0.9853718876838684 + }, + { + "bits_per_byte": 1.4725594842459884, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.020700454711914, + "step": 7140, + "unclipped_grad_norm": 0.9728355407714844 + }, + { + "bits_per_byte": 1.5037610927134717, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 1.0423277616500854, + "step": 7150, + "unclipped_grad_norm": 0.9885263442993164 + }, + { + "bits_per_byte": 1.5015323695409406, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0407829284667969, + "step": 7160, + "unclipped_grad_norm": 0.9811927676200867 + }, + { + "bits_per_byte": 1.5161207979003783, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.050894856452942, + "step": 7170, + "unclipped_grad_norm": 1.032510757446289 + }, + { + "bits_per_byte": 1.4557008848951616, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 1.0090149641036987, + "step": 7180, + "unclipped_grad_norm": 0.9654735326766968 + }, + { + "bits_per_byte": 1.5117491708980966, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 1.0478646755218506, + "step": 7190, + "unclipped_grad_norm": 1.0547142028808594 + }, + { + "bits_per_byte": 1.4408587821257013, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 0.9987272024154663, + "step": 7200, + "unclipped_grad_norm": 0.9307000637054443 + }, + { + "bits_per_byte": 1.4254401935105487, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 0.9880398511886597, + "step": 7210, + "unclipped_grad_norm": 0.8944076299667358 + }, + { + "bits_per_byte": 1.4447617564043818, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 1.0014325380325317, + "step": 7220, + "unclipped_grad_norm": 0.9928653240203857 + }, + { + "bits_per_byte": 1.488266315783987, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 1.0315876007080078, + "step": 7230, + "unclipped_grad_norm": 0.9765942096710205 + }, + { + "bits_per_byte": 1.501120643074775, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 1.0404975414276123, + "step": 7240, + "unclipped_grad_norm": 0.9696548581123352 + }, + { + "bits_per_byte": 1.495059458510417, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 1.0362962484359741, + "step": 7250, + "unclipped_grad_norm": 1.003565788269043 + }, + { + "bits_per_byte": 1.4667055387759202, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 1.0166428089141846, + "step": 7260, + "unclipped_grad_norm": 0.9783204197883606 + }, + { + "bits_per_byte": 1.4746533730202431, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 1.0221518278121948, + "step": 7270, + "unclipped_grad_norm": 0.9960736632347107 + }, + { + "bits_per_byte": 1.4700418302200078, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.0189553499221802, + "step": 7280, + "unclipped_grad_norm": 0.9735279679298401 + }, + { + "bits_per_byte": 1.4600210890848353, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 1.0120095014572144, + "step": 7290, + "unclipped_grad_norm": 1.0104917287826538 + }, + { + "bits_per_byte": 1.4665927181569542, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 1.0165646076202393, + "step": 7300, + "unclipped_grad_norm": 0.9501184225082397 + }, + { + "bits_per_byte": 1.4552138300279183, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 1.008677363395691, + "step": 7310, + "unclipped_grad_norm": 1.0020359754562378 + }, + { + "bits_per_byte": 1.4254196415837705, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 0.9880256056785583, + "step": 7320, + "unclipped_grad_norm": 0.9911538362503052 + }, + { + "bits_per_byte": 1.4791333490925684, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 1.0252571105957031, + "step": 7330, + "unclipped_grad_norm": 1.0009909868240356 + }, + { + "bits_per_byte": 1.4387796838594207, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.9972860813140869, + "step": 7340, + "unclipped_grad_norm": 0.996161937713623 + }, + { + "bits_per_byte": 1.440092513424782, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 0.9981960654258728, + "step": 7350, + "unclipped_grad_norm": 1.004310965538025 + }, + { + "bits_per_byte": 1.450652162196433, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 1.005515456199646, + "step": 7360, + "unclipped_grad_norm": 0.9492549896240234 + }, + { + "bits_per_byte": 1.4093955020549302, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 0.9769185185432434, + "step": 7370, + "unclipped_grad_norm": 0.9890016913414001 + }, + { + "bits_per_byte": 1.4082163030092893, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 0.9761011600494385, + "step": 7380, + "unclipped_grad_norm": 0.9570937156677246 + }, + { + "bits_per_byte": 1.4221050199536918, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 0.9857280850410461, + "step": 7390, + "unclipped_grad_norm": 0.9834490418434143 + }, + { + "bits_per_byte": 1.428768229787395, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.9903466701507568, + "step": 7400, + "unclipped_grad_norm": 0.9708907604217529 + }, + { + "bits_per_byte": 1.4728477271688345, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 1.0209002494812012, + "step": 7410, + "unclipped_grad_norm": 1.0163742303848267 + }, + { + "bits_per_byte": 1.4585456499230833, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0109868049621582, + "step": 7420, + "unclipped_grad_norm": 0.9965204000473022 + }, + { + "bits_per_byte": 1.4139233752955207, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 0.9800570011138916, + "step": 7430, + "unclipped_grad_norm": 0.9576085805892944 + }, + { + "bits_per_byte": 1.3763924613109844, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 0.9540425539016724, + "step": 7440, + "unclipped_grad_norm": 0.9713699817657471 + }, + { + "bits_per_byte": 1.4982521444410344, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 1.0385092496871948, + "step": 7450, + "unclipped_grad_norm": 1.0151621103286743 + }, + { + "bits_per_byte": 1.4277210274262913, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 0.9896208047866821, + "step": 7460, + "unclipped_grad_norm": 0.9804997444152832 + }, + { + "bits_per_byte": 1.4026076047993514, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 0.9722135066986084, + "step": 7470, + "unclipped_grad_norm": 0.991990327835083 + }, + { + "bits_per_byte": 1.500402443524772, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 1.0399997234344482, + "step": 7480, + "unclipped_grad_norm": 0.9778653383255005 + }, + { + "bits_per_byte": 1.4553643148474231, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 1.0087816715240479, + "step": 7490, + "unclipped_grad_norm": 1.0049223899841309 + }, + { + "bits_per_byte": 1.4863592001685693, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.0302656888961792, + "step": 7500, + "unclipped_grad_norm": 1.0053783655166626 + }, + { + "bits_per_byte": 1.4650730794539304, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 1.0155112743377686, + "step": 7510, + "unclipped_grad_norm": 0.9855281710624695 + }, + { + "bits_per_byte": 1.4650512376572709, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 1.0154961347579956, + "step": 7520, + "unclipped_grad_norm": 0.985152542591095 + }, + { + "bits_per_byte": 1.4541783224870732, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.0079596042633057, + "step": 7530, + "unclipped_grad_norm": 0.9860301613807678 + }, + { + "bits_per_byte": 1.4750535766488009, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.0224292278289795, + "step": 7540, + "unclipped_grad_norm": 0.9990776181221008 + }, + { + "bits_per_byte": 1.457138487873724, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 1.0100114345550537, + "step": 7550, + "unclipped_grad_norm": 0.9592881798744202 + }, + { + "bits_per_byte": 1.373044647029289, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 0.9517220258712769, + "step": 7560, + "unclipped_grad_norm": 0.9685735106468201 + }, + { + "bits_per_byte": 1.419239531016341, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 0.9837418794631958, + "step": 7570, + "unclipped_grad_norm": 0.9625195860862732 + }, + { + "bits_per_byte": 1.407609290243069, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 0.975680410861969, + "step": 7580, + "unclipped_grad_norm": 0.9880757927894592 + }, + { + "bits_per_byte": 1.4710052770301412, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 1.0196231603622437, + "step": 7590, + "unclipped_grad_norm": 1.0152498483657837 + }, + { + "bits_per_byte": 1.4918872385152526, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 1.03409743309021, + "step": 7600, + "unclipped_grad_norm": 1.0182538032531738 + }, + { + "bits_per_byte": 1.4563219142474273, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 1.0094454288482666, + "step": 7610, + "unclipped_grad_norm": 0.9846963882446289 + }, + { + "bits_per_byte": 1.47490790734352, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 1.02232825756073, + "step": 7620, + "unclipped_grad_norm": 0.9959330558776855 + }, + { + "bits_per_byte": 1.4439807831868139, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 1.0008912086486816, + "step": 7630, + "unclipped_grad_norm": 0.978793203830719 + }, + { + "bits_per_byte": 1.4236575933343558, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.9868042469024658, + "step": 7640, + "unclipped_grad_norm": 1.0194649696350098 + }, + { + "bits_per_byte": 1.3929547345543745, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 0.9655226469039917, + "step": 7650, + "unclipped_grad_norm": 0.9518723487854004 + }, + { + "bits_per_byte": 1.4068598758419344, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 0.9751609563827515, + "step": 7660, + "unclipped_grad_norm": 1.033186674118042 + }, + { + "bits_per_byte": 1.4655642619047946, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 1.0158517360687256, + "step": 7670, + "unclipped_grad_norm": 0.9971597790718079 + }, + { + "bits_per_byte": 1.460744964062317, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 1.0125112533569336, + "step": 7680, + "unclipped_grad_norm": 1.0221179723739624 + }, + { + "bits_per_byte": 1.432434297964512, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 0.9928877949714661, + "step": 7690, + "unclipped_grad_norm": 0.980252206325531 + }, + { + "bits_per_byte": 1.4827501442402133, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 1.027764081954956, + "step": 7700, + "unclipped_grad_norm": 0.9810852408409119 + }, + { + "bits_per_byte": 1.4101831825958817, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 0.977464497089386, + "step": 7710, + "unclipped_grad_norm": 0.9668226838111877 + }, + { + "bits_per_byte": 1.4156680532971986, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 0.9812663197517395, + "step": 7720, + "unclipped_grad_norm": 0.9539591670036316 + }, + { + "bits_per_byte": 1.4228393858728758, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 0.986237108707428, + "step": 7730, + "unclipped_grad_norm": 0.9774504899978638 + }, + { + "bits_per_byte": 1.460200982937638, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 1.0121341943740845, + "step": 7740, + "unclipped_grad_norm": 1.0424498319625854 + }, + { + "bits_per_byte": 1.3995881053981531, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9701205492019653, + "step": 7750, + "unclipped_grad_norm": 0.9873414039611816 + }, + { + "bits_per_byte": 1.41283885269918, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 0.9793052673339844, + "step": 7760, + "unclipped_grad_norm": 1.0039513111114502 + }, + { + "bits_per_byte": 1.4655054438381996, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 1.0158109664916992, + "step": 7770, + "unclipped_grad_norm": 1.0126665830612183 + }, + { + "bits_per_byte": 1.4406716650015625, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 0.9985975027084351, + "step": 7780, + "unclipped_grad_norm": 1.0267153978347778 + }, + { + "bits_per_byte": 1.4657295372322738, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 1.0159662961959839, + "step": 7790, + "unclipped_grad_norm": 1.013239860534668 + }, + { + "bits_per_byte": 1.4565669895249067, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 1.0096153020858765, + "step": 7800, + "unclipped_grad_norm": 1.0022791624069214 + }, + { + "bits_per_byte": 1.4306827406567997, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9916737079620361, + "step": 7810, + "unclipped_grad_norm": 1.0330612659454346 + }, + { + "bits_per_byte": 1.4581898178184483, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 1.0107401609420776, + "step": 7820, + "unclipped_grad_norm": 1.0448929071426392 + }, + { + "bits_per_byte": 1.4612282353112414, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 1.0128462314605713, + "step": 7830, + "unclipped_grad_norm": 1.0258532762527466 + }, + { + "bits_per_byte": 1.4382826539984273, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.9969415664672852, + "step": 7840, + "unclipped_grad_norm": 1.0607435703277588 + }, + { + "bits_per_byte": 1.421009146502394, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.9849684834480286, + "step": 7850, + "unclipped_grad_norm": 0.991746723651886 + }, + { + "bits_per_byte": 1.4651265660583488, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 1.0155483484268188, + "step": 7860, + "unclipped_grad_norm": 1.0202583074569702 + }, + { + "bits_per_byte": 1.450212058592875, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0052103996276855, + "step": 7870, + "unclipped_grad_norm": 0.9972054958343506 + }, + { + "bits_per_byte": 1.4201922289107956, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9844022393226624, + "step": 7880, + "unclipped_grad_norm": 0.9974161386489868 + }, + { + "bits_per_byte": 1.439974877291592, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 0.9981145262718201, + "step": 7890, + "unclipped_grad_norm": 0.9990294575691223 + }, + { + "bits_per_byte": 1.402634434092886, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 0.9722321033477783, + "step": 7900, + "unclipped_grad_norm": 0.9622854590415955 + }, + { + "bits_per_byte": 1.4610956366874261, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.0127543210983276, + "step": 7910, + "unclipped_grad_norm": 1.093667984008789 + }, + { + "bits_per_byte": 1.477739945655278, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 1.0242912769317627, + "step": 7920, + "unclipped_grad_norm": 1.0083937644958496 + }, + { + "bits_per_byte": 1.434365577142376, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.9942264556884766, + "step": 7930, + "unclipped_grad_norm": 1.059904932975769 + }, + { + "bits_per_byte": 1.4476010179874759, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 1.0034005641937256, + "step": 7940, + "unclipped_grad_norm": 1.0224626064300537 + }, + { + "bits_per_byte": 1.3935098085600341, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 0.9659073948860168, + "step": 7950, + "unclipped_grad_norm": 0.9811362624168396 + }, + { + "bits_per_byte": 1.4619749839812877, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 1.0133638381958008, + "step": 7960, + "unclipped_grad_norm": 1.0705063343048096 + }, + { + "bits_per_byte": 1.428583692403019, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 0.9902187585830688, + "step": 7970, + "unclipped_grad_norm": 1.0157805681228638 + }, + { + "bits_per_byte": 1.465526253738954, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 1.0158253908157349, + "step": 7980, + "unclipped_grad_norm": 1.1353037357330322 + }, + { + "bits_per_byte": 1.4197623582749639, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 0.9841042757034302, + "step": 7990, + "unclipped_grad_norm": 1.1050702333450317 + }, + { + "bits_per_byte": 1.4607263899360237, + "learning_rate": 3e-05, + "loss_nats": 1.012498378753662, + "step": 8000, + "unclipped_grad_norm": 1.0229021310806274 + } + ], + "variant": "uniform_group_7_mlp_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073003.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073003.json new file mode 100644 index 0000000..cb4da64 --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073003.json @@ -0,0 +1,14029 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "011c035860ead7b26f58a90bd3d56fb4dce99c55111666daa7f9d1c0f47675f3", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.00027819047681987286, + 0.0002560806169640273, + 0.0002331122086616233, + 0.00022053472639527172, + 0.00020666928321588784, + 0.00019684624567162246, + 0.0001890837011160329, + 0.00018263237143401057, + 0.00017469045997131616, + 0.0001688334159553051, + 0.0001649101759539917, + 0.00016157960635609925, + 0.00015717606584075838, + 0.00015400603297166526, + 0.00015046694898046553, + 0.00014767753600608557, + 0.00014588078192900866, + 0.00014366162940859795, + 0.00014116008242126554, + 0.00013849612150806934, + 0.0001361419854220003, + 0.00013430001854430884, + 0.00013224364374764264, + 0.00013045054220128804, + 0.00012888714263681322, + 0.00012767451698891819, + 0.0001263424928765744, + 0.00012512138346210122, + 0.00012419422273524106, + 0.00012343283742666245, + 0.0001227961911354214, + 0.00012195061572128907 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00022039370378479362, + "first_to_last_ratio": 1.7624457042066763, + "imbalance_abs_log_ratio": 0.5667024491058967, + "last_quartile_mean": 0.00012504992537287762, + "mean": 0.0001607882525149762, + "normalized": [ + 1.7301666786506156, + 1.592657504255016, + 1.4498087081325215, + 1.3715848200709229, + 1.2853506396348091, + 1.224257634451794, + 1.175979576607571, + 1.1358564358860717, + 1.0864628307036615, + 1.050035766385231, + 1.0256357250890054, + 1.0049217142965674, + 0.9775345112735686, + 0.9578189361646354, + 0.9358080993289649, + 0.9184597363064851, + 0.9072850761620224, + 0.8934833681037547, + 0.8779253472396409, + 0.8613572157279928, + 0.8467159963027754, + 0.8352601414820391, + 0.8224708066612338, + 0.81131886291965, + 0.8015955184587156, + 0.7940537632065271, + 0.7857694259399116, + 0.7781749070905981, + 0.7724085609032495, + 0.7676732316943716, + 0.7637136993200662, + 0.75845476155001 + ], + "population_cv": 0.2498932502086342 + }, + "activation_output_rms_by_block": [ + 0.0034565660171210766, + 0.004928664769977331, + 0.0060537648387253284, + 0.006960673723369837, + 0.0036145171616226435, + 0.0051584807224571705, + 0.006370538379997015, + 0.007546419743448496, + 0.00395671371370554, + 0.005665866658091545, + 0.006755253300070763, + 0.007946331985294819, + 0.004383862018585205, + 0.006652849726378918, + 0.007932198233902454, + 0.009347881190478802, + 0.004880861379206181, + 0.006906712893396616, + 0.008132148534059525, + 0.009816600009799004, + 0.005453306715935469, + 0.007622495759278536, + 0.009448722936213017, + 0.010670131072402, + 0.0055701653473079205, + 0.007738742511719465, + 0.009509840048849583, + 0.010865328833460808, + 0.005987230688333511, + 0.00815181527286768, + 0.009874933399260044, + 0.011666664853692055 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.005511203169589862, + "first_to_last_ratio": 0.6356203088456116, + "imbalance_abs_log_ratio": 0.45315389258709987, + "last_quartile_mean": 0.008670590119436383, + "mean": 0.007157071326219011, + "normalized": [ + 0.4829581625738996, + 0.6886426787339398, + 0.8458438602600115, + 0.9725589429115095, + 0.5050274053272774, + 0.7207530129760953, + 0.8901040788372985, + 1.0544005221525679, + 0.5528397766850007, + 0.7916459680002588, + 0.9438571997073384, + 1.1102770425361632, + 0.6125217730505899, + 0.9295491721602742, + 1.1083022471557418, + 1.3061042379491232, + 0.6819634955048963, + 0.9650194302374437, + 1.1362396940587196, + 1.3715945478756304, + 0.7619466772614072, + 1.0650300118366156, + 1.3201940438400879, + 1.490851576861243, + 0.7782743937317399, + 1.0812722353862223, + 1.3287334463206348, + 1.5181249897087195, + 0.8365475786723071, + 1.138987569259587, + 1.3797450031110483, + 1.6300892253166077 + ], + "population_cv": 0.3025375276561286 + }, + "bits_per_byte": 8.06433666787865, + "branch_output_rms_by_sublayer": [ + 0.0021227982360869646, + 0.0027448106557130814, + 0.00224129855632782, + 0.0027243648655712605, + 0.0024903982412070036, + 0.0027103275060653687, + 0.0023722455371171236, + 0.0027234479784965515, + 0.0023577234242111444, + 0.0027595730498433113, + 0.0023993616923689842, + 0.0027272943407297134, + 0.002688874490559101, + 0.002728298306465149, + 0.0029664887115359306, + 0.0027007071767002344, + 0.0029547461308538914, + 0.002684502862393856, + 0.003052416956052184, + 0.002683063969016075, + 0.0028214550111442804, + 0.0027139110025018454, + 0.0032544767018407583, + 0.002666700165718794, + 0.00348988501355052, + 0.002679243916645646, + 0.003914494998753071, + 0.0026584581937640905, + 0.0035071265883743763, + 0.002684473292902112, + 0.004211232997477055, + 0.0026407993864268064, + 0.004072706215083599, + 0.002663099905475974, + 0.004050548188388348, + 0.0026283918414264917, + 0.003929397091269493, + 0.002597011858597398, + 0.004410331137478352, + 0.0026434636674821377, + 0.004824656527489424, + 0.0025708195753395557, + 0.00433384021744132, + 0.002691218862310052, + 0.004475373309105635, + 0.0026598076801747084, + 0.004755842965096235, + 0.0026596153620630503, + 0.004843301139771938, + 0.0026765079237520695, + 0.0050001065246760845, + 0.002582729561254382, + 0.005346097983419895, + 0.002614464145153761, + 0.004958339501172304, + 0.0025760054122656584, + 0.0054801348596811295, + 0.002634295728057623, + 0.005455824080854654, + 0.00259007909335196, + 0.005286100786179304, + 0.00257018837146461, + 0.005402065813541412, + 0.0025643527042120695 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.006159982634052032, + 0.005514663551817784, + 0.005191681274696802, + 0.00489831575692433, + 0.004666415075367917, + 0.004399813239791133, + 0.004483878567781538, + 0.004868281936291656, + 0.00443828064080646, + 0.004266304440803418, + 0.0039471188591477255, + 0.004494388758593749, + 0.0042334459038140095, + 0.00446434979681851, + 0.004322339302329385, + 0.004638471384339534, + 0.004165330760400444, + 0.004499937764754925, + 0.003981478974932938, + 0.004372386299982334, + 0.00458495381977442, + 0.00429501466057645, + 0.003963900484885007, + 0.004271473918508079, + 0.004041799650627023, + 0.004206875397956982, + 0.003955591472942362, + 0.003916464042690177, + 0.003989587235588314, + 0.003922309599766117, + 0.003679536678174494, + 0.003763244111925236 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.005022879004590398, + "first_to_last_ratio": 1.2766484804448084, + "imbalance_abs_log_ratio": 0.24423826934058726, + "last_quartile_mean": 0.003934426023708838, + "mean": 0.0043936754999019155, + "normalized": [ + 1.4020112851278042, + 1.255136741878387, + 1.1816260155791434, + 1.1148560600421402, + 1.0620754936207943, + 1.0013969488391563, + 1.0205302070855338, + 1.108020366183241, + 1.0101521245493756, + 0.9710103627130996, + 0.8983637638318627, + 1.0229223252136126, + 0.9635317637610054, + 1.0160854612313022, + 0.9837638902613262, + 1.0557155130011497, + 0.9480287655502622, + 1.0241852783291305, + 0.9061841219320409, + 0.995154580733134, + 1.0435349219296635, + 0.9775448051801577, + 0.9021832597727114, + 0.9721869351988865, + 0.9199131002545027, + 0.9574843199164109, + 0.9002921296829196, + 0.891386731400079, + 0.9080295610536958, + 0.892717179467095, + 0.8374620925593245, + 0.8565138941210488 + ], + "population_cv": 0.11556590700625036 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028097106143832207, + 0.014079716056585312, + 0.014136734418570995, + 0.014167090877890587, + 0.014224153943359852, + 0.014255538582801819, + 0.014337832108139992, + 0.014355218037962914, + 0.014408928342163563, + 0.009655896574258804, + 0.009704872034490108, + 0.009734091348946095, + 0.009773381985723972, + 0.009828771464526653, + 0.00987160298973322, + 0.009915035218000412, + 0.00994801800698042, + 0.007479202933609486, + 0.007515520788729191, + 0.007559895049780607, + 0.007602147292345762, + 0.0076358383521437645, + 0.007658426649868488, + 0.007706487085670233, + 0.007734050042927265, + 0.0062490664422512054, + 0.0062688966281712055, + 0.006315518636256456, + 0.006336068734526634, + 0.0063744354993104935, + 0.006399397272616625, + 0.006439714692533016, + 0.006462901830673218, + 0.0054116579703986645, + 0.005432929843664169, + 0.005481995176523924, + 0.005504137370735407, + 0.005541813559830189, + 0.005560056772083044, + 0.005614723544567823, + 0.005631812382489443, + 0.004862982779741287, + 0.004873365629464388, + 0.004923288244754076, + 0.004937449004501104, + 0.0049925679340958595, + 0.005005750339478254, + 0.005004179198294878, + 0.005022916477173567, + 0.004448203835636377, + 0.0044684382155537605, + 0.004490054678171873, + 0.004500201437622309, + 0.00455933902412653, + 0.004573327023535967, + 0.004625064320862293, + 0.00463611725717783, + 0.004160413984209299, + 0.004166214261204004, + 0.004242690280079842, + 0.0042496914975345135, + 0.004270957782864571, + 0.004283483140170574, + 0.004303553607314825 + ], + "loss_nats": 5.5897722244262695, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0021227982360869646, + 0.0034565660171210766, + 0.00413061399012804, + 0.004928664769977331, + 0.005416445434093475, + 0.0060537648387253284, + 0.006388718727976084, + 0.006960673723369837, + 0.0023577234242111444, + 0.0036145171616226435, + 0.004388594534248114, + 0.0051584807224571705, + 0.00574506726115942, + 0.006370538379997015, + 0.007066869642585516, + 0.007546419743448496, + 0.0029547461308538914, + 0.00395671371370554, + 0.004991721361875534, + 0.005665866658091545, + 0.006190058775246143, + 0.006755253300070763, + 0.0074937604367733, + 0.007946331985294819, + 0.00348988501355052, + 0.004383862018585205, + 0.006057556718587875, + 0.006652849726378918, + 0.0074848332442343235, + 0.007932198233902454, + 0.00897275935858488, + 0.009347881190478802, + 0.004072706215083599, + 0.004880861379206181, + 0.0063449000008404255, + 0.006906712893396616, + 0.007735534105449915, + 0.008132148534059525, + 0.00946879480034113, + 0.009816600009799004, + 0.004824656527489424, + 0.005453306715935469, + 0.007074959110468626, + 0.007622495759278536, + 0.009095527231693268, + 0.009448722936213017, + 0.010346688330173492, + 0.010670131072402, + 0.004843301139771938, + 0.0055701653473079205, + 0.007334613241255283, + 0.007738742511719465, + 0.009146629832684994, + 0.009509840048849583, + 0.010587968863546848, + 0.010865328833460808, + 0.0054801348596811295, + 0.005987230688333511, + 0.0077637480571866035, + 0.00815181527286768, + 0.009536821395158768, + 0.009874933399260044, + 0.011367839761078358, + 0.011666664853692055 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.855150422779843e-05, + 6.49694338790141e-05, + 6.157680036267266e-05, + 5.872502151760273e-05, + 5.402821261668578e-05, + 5.094641528557986e-05, + 4.8056641389848664e-05, + 4.5395248889690265e-05, + 4.1273018723586574e-05, + 3.8490463339257985e-05, + 3.60785452357959e-05, + 3.379531335667707e-05, + 3.1306641176342964e-05, + 2.9401047868304886e-05, + 2.7745165425585583e-05, + 2.6120722395717166e-05, + 2.4809520255075768e-05, + 2.3514592612627894e-05, + 2.246276562800631e-05, + 2.1602329070447013e-05, + 2.116804535035044e-05, + 2.05241194635164e-05, + 2.0005971236969344e-05, + 1.9540089851943776e-05, + 1.9181190509698354e-05, + 1.875746966106817e-05, + 1.837764102674555e-05, + 1.808313390938565e-05, + 1.7768801626516506e-05, + 1.7557515093358234e-05, + 1.7355283489450812e-05, + 1.7209338693646714e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.653115977111156e-05, + "first_to_last_ratio": 3.1342997152253345, + "imbalance_abs_log_ratio": 1.1424057728090198, + "last_quartile_mean": 1.803629675123375e-05, + "mean": 3.2324312599030236e-05, + "normalized": [ + 2.1207412846841187, + 2.0099246868737204, + 1.9049685952028574, + 1.8167446357193235, + 1.6714419665124352, + 1.5761020479398626, + 1.4867026558606669, + 1.404368577076939, + 1.276841343404927, + 1.1907589131659597, + 1.116142690591301, + 1.045507565029889, + 0.9685168425602405, + 0.9095645198403058, + 0.858337368832956, + 0.8080828421545712, + 0.7675188816179213, + 0.7274583965424637, + 0.694918586719775, + 0.6682997203502823, + 0.6548645167781697, + 0.6349437254276568, + 0.6189140503971473, + 0.6045013267360246, + 0.593398249411615, + 0.5802898237542387, + 0.5685392680955851, + 0.559428258651606, + 0.5497039286474908, + 0.5431674699831043, + 0.5369111388302652, + 0.5323961226065798 + ], + "population_cv": 0.4870702477047579 + }, + "activation_output_rms_by_block": [ + 0.005520366132259369, + 0.009551605209708214, + 0.013989881612360477, + 0.01876777783036232, + 0.006622526329010725, + 0.01178906112909317, + 0.017192937433719635, + 0.02232205867767334, + 0.0074642314575612545, + 0.013203916139900684, + 0.019116846844553947, + 0.02576720155775547, + 0.008968878537416458, + 0.016232330352067947, + 0.023723822087049484, + 0.032115157693624496, + 0.009892941452562809, + 0.01923578418791294, + 0.028606966137886047, + 0.03876862674951553, + 0.012688412331044674, + 0.022661328315734863, + 0.03460892662405968, + 0.04590345546603203, + 0.014741362072527409, + 0.02941947802901268, + 0.046154435724020004, + 0.062365636229515076, + 0.01962464675307274, + 0.03736087307333946, + 0.05551042780280113, + 0.07462256401777267 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013219526794273406, + "first_to_last_ratio": 0.31123129404396854, + "imbalance_abs_log_ratio": 1.1672189324663935, + "last_quartile_mean": 0.04247492796275765, + "mean": 0.02514107699971646, + "normalized": [ + 0.2195755628257941, + 0.3799202878148751, + 0.5564551436089334, + 0.7464985621170478, + 0.2634145835950231, + 0.46891631290203384, + 0.6838584295300291, + 0.8878720143104882, + 0.29689386248828703, + 0.5251929398271044, + 0.7603829718499946, + 1.024904444548898, + 0.35674201775515063, + 0.6456497608376528, + 0.9436279156742983, + 1.2773978495028948, + 0.39349712236569584, + 0.7651137693158444, + 1.137857623927912, + 1.5420431968746908, + 0.5046884956912455, + 0.901366648532616, + 1.3765888638919483, + 1.8258348863316287, + 0.5863456872867324, + 1.1701757259382513, + 1.8358177624825114, + 2.480627072190202, + 0.7805809891634342, + 1.486049029393642, + 2.207957431713334, + 2.968153035711806 + ], + "population_cv": 0.6640736464365783 + }, + "bits_per_byte": 5.183356909199448, + "branch_output_rms_by_sublayer": [ + 0.002719750627875328, + 0.00483035109937191, + 0.0025345177855342627, + 0.004947863984853029, + 0.002758872229605913, + 0.0054994672536849976, + 0.002652689116075635, + 0.005899749230593443, + 0.00279375072568655, + 0.006002360954880714, + 0.0025332141667604446, + 0.00605355529114604, + 0.0034121458884328604, + 0.006182286888360977, + 0.003361867507919669, + 0.0059405043721199036, + 0.0035713587421923876, + 0.006377940997481346, + 0.003841861616820097, + 0.0062105231918394566, + 0.003399067558348179, + 0.006491490174084902, + 0.0046233078464865685, + 0.006909551564604044, + 0.004881689324975014, + 0.007106001954525709, + 0.005147914867848158, + 0.006858175154775381, + 0.005394710227847099, + 0.006822903174906969, + 0.006120751146227121, + 0.007109703961759806, + 0.005600836127996445, + 0.007118321023881435, + 0.0063801109790802, + 0.00763789564371109, + 0.007226853631436825, + 0.007455042563378811, + 0.008725153282284737, + 0.0074478075839579105, + 0.00841668900102377, + 0.007471881806850433, + 0.0075513701885938644, + 0.007769893854856491, + 0.009496993385255337, + 0.007438631262630224, + 0.00888020172715187, + 0.0073490324430167675, + 0.009537886828184128, + 0.008383425883948803, + 0.011849360540509224, + 0.008326031267642975, + 0.014158839359879494, + 0.008139106445014477, + 0.013660875149071217, + 0.0077112894505262375, + 0.014504852704703808, + 0.00793725997209549, + 0.014560646377503872, + 0.007785677444189787, + 0.013708031736314297, + 0.008717497810721397, + 0.015193133614957333, + 0.008376051671802998 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0002371736215844889, + 0.00020531263376162147, + 0.00019745294772652, + 0.000181562973460587, + 0.00017047528803911574, + 0.00015977318440641035, + 0.00017390570631318786, + 0.00016044422785875442, + 0.00016456265374223354, + 0.00016169845768866385, + 0.0001472744603760807, + 0.0001699453955980444, + 0.00016281234661929636, + 0.00016597391227296724, + 0.00015358999603492499, + 0.0001599273984822134, + 0.00014080537479975856, + 0.00015399207586296376, + 0.00014519617781193664, + 0.00016343729223333324, + 0.0001543405808087071, + 0.00013581516151365598, + 0.0001517266913363975, + 0.0001437657019903694, + 0.00014417004239774308, + 0.00015356660210869174, + 0.00015699173013435664, + 0.00014441773261414864, + 0.00014426244360311467, + 0.00014104980980941462, + 0.00013593515775558497, + 0.00013462386475489167 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0001857625728938357, + "first_to_last_ratio": 1.2866478070328158, + "imbalance_abs_log_ratio": 0.25204023695468775, + "last_quartile_mean": 0.00014437717289724324, + "mean": 0.0001598744263593806, + "normalized": [ + 1.4834994375607515, + 1.2842118557479647, + 1.2350502342548952, + 1.1356598900467851, + 1.0663074259037875, + 0.999366740789783, + 1.087764380290982, + 1.0035640565683281, + 1.029324435993999, + 1.0114091501112443, + 0.9211883584496716, + 1.0629929968663365, + 1.0183764240899396, + 1.038151729782446, + 0.9606914597439813, + 1.000331335811731, + 0.880724816383348, + 0.9632064325085118, + 0.9081888899826366, + 1.0222854020814043, + 0.9653862992556796, + 0.8495114860231494, + 0.9490366582791181, + 0.8992413937873934, + 0.9017705062701165, + 0.9605451328625284, + 0.9819689972269614, + 0.9033197860520421, + 0.9023484674079653, + 0.8822537351430413, + 0.850262051605535, + 0.8420600331179399 + ], + "population_cv": 0.13317819095382705 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.692057192325592, + "mean_weights": [ + 0.5229623913764954, + 0.47703757882118225 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6922353506088257, + "mean_weights": [ + 0.5207095742225647, + 0.4792904853820801 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6930612325668335, + "mean_weights": [ + 0.5023279786109924, + 0.49767202138900757 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6924437284469604, + "mean_weights": [ + 0.5176364779472351, + 0.4823635220527649 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929886341094971, + "mean_weights": [ + 0.501381516456604, + 0.4986185133457184 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929053068161011, + "mean_weights": [ + 0.509856104850769, + 0.49014389514923096 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927419900894165, + "mean_weights": [ + 0.5127999782562256, + 0.4872000217437744 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928871870040894, + "mean_weights": [ + 0.5100691318511963, + 0.4899308383464813 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0981073379516602, + "mean_weights": [ + 0.33567744493484497, + 0.3208778500556946, + 0.3434446454048157 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985171794891357, + "mean_weights": [ + 0.33793938159942627, + 0.33097097277641296, + 0.33108964562416077 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984208583831787, + "mean_weights": [ + 0.34078651666641235, + 0.3312467932701111, + 0.32796669006347656 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985374450683594, + "mean_weights": [ + 0.33723682165145874, + 0.33111339807510376, + 0.3316497802734375 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0983846187591553, + "mean_weights": [ + 0.3410840332508087, + 0.3330618143081665, + 0.32585418224334717 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984747409820557, + "mean_weights": [ + 0.33782798051834106, + 0.3331233263015747, + 0.32904869318008423 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982093811035156, + "mean_weights": [ + 0.34277671575546265, + 0.33254319429397583, + 0.3246800899505615 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985099077224731, + "mean_weights": [ + 0.33662915229797363, + 0.33530691266059875, + 0.3280639350414276 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3860619068145752, + "mean_weights": [ + 0.2556872069835663, + 0.25280922651290894, + 0.24703474342823029, + 0.2444688230752945 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861664533615112, + "mean_weights": [ + 0.2540717124938965, + 0.2527104914188385, + 0.2465764880180359, + 0.24664132297039032 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859951496124268, + "mean_weights": [ + 0.25520774722099304, + 0.2541807293891907, + 0.24830153584480286, + 0.24230998754501343 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862321376800537, + "mean_weights": [ + 0.25185489654541016, + 0.25247788429260254, + 0.2484549880027771, + 0.2472122311592102 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859525918960571, + "mean_weights": [ + 0.2546065151691437, + 0.25498151779174805, + 0.2496897578239441, + 0.240722194314003 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861346244812012, + "mean_weights": [ + 0.25202471017837524, + 0.25435036420822144, + 0.2492796778678894, + 0.24434521794319153 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859035968780518, + "mean_weights": [ + 0.2541537284851074, + 0.25645712018013, + 0.2498953938484192, + 0.239493727684021 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861725330352783, + "mean_weights": [ + 0.24928712844848633, + 0.2551161050796509, + 0.2504235506057739, + 0.2451731562614441 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6091647148132324, + "mean_weights": [ + 0.20297083258628845, + 0.20568537712097168, + 0.2007569670677185, + 0.19490554928779602, + 0.19568124413490295 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093813180923462, + "mean_weights": [ + 0.19996923208236694, + 0.20290374755859375, + 0.1990443468093872, + 0.19804075360298157, + 0.20004190504550934 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6088874340057373, + "mean_weights": [ + 0.2043980062007904, + 0.20701202750205994, + 0.20311453938484192, + 0.19479729235172272, + 0.19067814946174622 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093982458114624, + "mean_weights": [ + 0.20016685128211975, + 0.2025349736213684, + 0.20030784606933594, + 0.1992955058813095, + 0.1976948231458664 + ], + "sources": 5 + }, + { + "entropy_mean": 1.608514666557312, + "mean_weights": [ + 0.20291486382484436, + 0.21057641506195068, + 0.20388731360435486, + 0.19458051025867462, + 0.18804088234901428 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6092827320098877, + "mean_weights": [ + 0.20027627050876617, + 0.20464272797107697, + 0.20175954699516296, + 0.19855692982673645, + 0.19476452469825745 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085467338562012, + "mean_weights": [ + 0.20302069187164307, + 0.21112579107284546, + 0.20331549644470215, + 0.19479486346244812, + 0.1877431571483612 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609300971031189, + "mean_weights": [ + 0.2015964686870575, + 0.20404735207557678, + 0.20027168095111847, + 0.19876068830490112, + 0.19532380998134613 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7908003330230713, + "mean_weights": [ + 0.1695440709590912, + 0.17702393233776093, + 0.1705935001373291, + 0.16258949041366577, + 0.1573334038257599, + 0.16291561722755432 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917125225067139, + "mean_weights": [ + 0.1673993021249771, + 0.1689971387386322, + 0.16775749623775482, + 0.16574542224407196, + 0.16472767293453217, + 0.16537295281887054 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7902088165283203, + "mean_weights": [ + 0.1706239879131317, + 0.1804180145263672, + 0.17242389917373657, + 0.16423344612121582, + 0.15748845040798187, + 0.15481218695640564 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917354106903076, + "mean_weights": [ + 0.16661900281906128, + 0.16851259768009186, + 0.16663852334022522, + 0.16634902358055115, + 0.166251540184021, + 0.1656292974948883 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7898740768432617, + "mean_weights": [ + 0.17189405858516693, + 0.18024206161499023, + 0.17466925084590912, + 0.16425329446792603, + 0.1563338339328766, + 0.1526075005531311 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917174100875854, + "mean_weights": [ + 0.16591984033584595, + 0.16852648556232452, + 0.16693192720413208, + 0.1667833924293518, + 0.16574755311012268, + 0.16609077155590057 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7883894443511963, + "mean_weights": [ + 0.17291340231895447, + 0.1850849986076355, + 0.17578130960464478, + 0.16393642127513885, + 0.15380312502384186, + 0.14848074316978455 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917414903640747, + "mean_weights": [ + 0.16680240631103516, + 0.16815051436424255, + 0.1662302166223526, + 0.16708910465240479, + 0.16612885892391205, + 0.16559891402721405 + ], + "sources": 6 + }, + { + "entropy_mean": 1.942643165588379, + "mean_weights": [ + 0.14892661571502686, + 0.15992864966392517, + 0.15346379578113556, + 0.14321936666965485, + 0.13470548391342163, + 0.12978067994117737, + 0.12997540831565857 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458975791931152, + "mean_weights": [ + 0.1420162320137024, + 0.14346325397491455, + 0.14283032715320587, + 0.14311958849430084, + 0.14275522530078888, + 0.1427994668483734, + 0.14301589131355286 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9415864944458008, + "mean_weights": [ + 0.14968237280845642, + 0.1630173623561859, + 0.15403282642364502, + 0.14396074414253235, + 0.13445039093494415, + 0.12876106798648834, + 0.1260952353477478 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458822011947632, + "mean_weights": [ + 0.14301569759845734, + 0.1430681049823761, + 0.1421995460987091, + 0.1414920538663864, + 0.14211012423038483, + 0.14377179741859436, + 0.14434269070625305 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9424850940704346, + "mean_weights": [ + 0.14929383993148804, + 0.16034889221191406, + 0.15305384993553162, + 0.1440136730670929, + 0.13572856783866882, + 0.13007456064224243, + 0.12748664617538452 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458880424499512, + "mean_weights": [ + 0.1432165503501892, + 0.14398843050003052, + 0.14288675785064697, + 0.14205607771873474, + 0.14131470024585724, + 0.14336368441581726, + 0.14317381381988525 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9396767616271973, + "mean_weights": [ + 0.15015524625778198, + 0.16697557270526886, + 0.15582090616226196, + 0.14567962288856506, + 0.13390687108039856, + 0.1263817846775055, + 0.12107999622821808 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9458975791931152, + "mean_weights": [ + 0.14285923540592194, + 0.14344796538352966, + 0.14336764812469482, + 0.14274798333644867, + 0.1419239640235901, + 0.14218640327453613, + 0.14346681535243988 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079418182373047, + "mean_weights": [ + 0.12460789084434509, + 0.12507331371307373, + 0.12491033971309662, + 0.12504473328590393, + 0.12460193037986755, + 0.12599892914295197, + 0.12613064050674438, + 0.12363224476575851 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0793228149414062, + "mean_weights": [ + 0.1242985874414444, + 0.12245838344097137, + 0.12359864264726639, + 0.12376782298088074, + 0.12466956675052643, + 0.12557029724121094, + 0.12764757871627808, + 0.12798911333084106 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0794057846069336, + "mean_weights": [ + 0.12457099556922913, + 0.1243164986371994, + 0.12479126453399658, + 0.12436504662036896, + 0.1239558607339859, + 0.12487402558326721, + 0.1261988878250122, + 0.12692740559577942 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0792927742004395, + "mean_weights": [ + 0.12389404326677322, + 0.12217588722705841, + 0.12296128273010254, + 0.12415566295385361, + 0.1256425678730011, + 0.12552398443222046, + 0.1266198754310608, + 0.12902669608592987 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1897499561309814, + "mean_weights": [ + 0.12039542198181152, + 0.13455089926719666, + 0.12450598180294037, + 0.11651790142059326, + 0.10918775200843811, + 0.10506531596183777, + 0.09920908510684967, + 0.09484627842903137, + 0.09572136402130127 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1970608234405518, + "mean_weights": [ + 0.10995376855134964, + 0.10848130285739899, + 0.10945813357830048, + 0.10999288409948349, + 0.11068244278430939, + 0.11116552352905273, + 0.11218016594648361, + 0.11502142995595932, + 0.11306434124708176 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1907927989959717, + "mean_weights": [ + 0.11952529102563858, + 0.1321919560432434, + 0.12339935451745987, + 0.11665944755077362, + 0.11038602888584137, + 0.10561776161193848, + 0.1004982516169548, + 0.09683522582054138, + 0.09488669782876968 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1971702575683594, + "mean_weights": [ + 0.11098268628120422, + 0.11007878184318542, + 0.10996996611356735, + 0.10998435318470001, + 0.10995156317949295, + 0.11261573433876038, + 0.11266376078128815, + 0.1119106113910675, + 0.11184254288673401 + ], + "sources": 9 + }, + { + "entropy_mean": 2.190373420715332, + "mean_weights": [ + 0.11987130343914032, + 0.1334265172481537, + 0.12377596646547318, + 0.11611881107091904, + 0.10979576408863068, + 0.10602270066738129, + 0.1008126437664032, + 0.09580239653587341, + 0.09437388181686401 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1969289779663086, + "mean_weights": [ + 0.10958327353000641, + 0.10723166167736053, + 0.10892683267593384, + 0.1096184030175209, + 0.1102953851222992, + 0.11106608062982559, + 0.11383499205112457, + 0.11435741931200027, + 0.11508595943450928 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1888015270233154, + "mean_weights": [ + 0.1204891949892044, + 0.1350543051958084, + 0.1249442920088768, + 0.11719748377799988, + 0.11051523685455322, + 0.10586231201887131, + 0.09971407800912857, + 0.09478242695331573, + 0.09144067764282227 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028617624193429947, + 0.014953437261283398, + 0.015275577083230019, + 0.014781011268496513, + 0.015793664380908012, + 0.015444811433553696, + 0.016596680507063866, + 0.016699986532330513, + 0.01783582754433155, + 0.011790595948696136, + 0.012897616252303123, + 0.012997417710721493, + 0.01400078646838665, + 0.014153926633298397, + 0.015287447720766068, + 0.015434568747878075, + 0.01647614873945713, + 0.012522599659860134, + 0.013484092429280281, + 0.013610593043267727, + 0.014503438025712967, + 0.014685511589050293, + 0.01565156690776348, + 0.015789827331900597, + 0.016939746215939522, + 0.013788276351988316, + 0.014673421159386635, + 0.014919825829565525, + 0.015800682827830315, + 0.016082024201750755, + 0.01699197106063366, + 0.01724671944975853, + 0.01826007477939129, + 0.015472620725631714, + 0.016315242275595665, + 0.01667407527565956, + 0.01762295328080654, + 0.017783133313059807, + 0.018875589594244957, + 0.01895430125296116, + 0.02021988108754158, + 0.01760070025920868, + 0.018588852137327194, + 0.01862320490181446, + 0.019797934219241142, + 0.019868170842528343, + 0.021140363067388535, + 0.02087467536330223, + 0.02255064807832241, + 0.02028188295662403, + 0.021092310547828674, + 0.021768642589449883, + 0.022647423669695854, + 0.023528600111603737, + 0.024347737431526184, + 0.025258628651499748, + 0.026221459731459618, + 0.022894909605383873, + 0.024970028549432755, + 0.02442554198205471, + 0.02649672143161297, + 0.02568441815674305, + 0.028459865599870682, + 0.026966959238052368 + ], + "loss_nats": 3.5928292274475098, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1923317909240723, + "mean_weights": [ + 0.0999261811375618, + 0.09311645478010178, + 0.1004323735833168, + 0.10806248337030411, + 0.11519288271665573, + 0.1180460974574089, + 0.12097769975662231, + 0.12243277579545975, + 0.12181305140256882 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.002719750627875328, + 0.005520366132259369, + 0.006244656629860401, + 0.009551605209708214, + 0.009990227408707142, + 0.013989881612360477, + 0.014324123971164227, + 0.01876777783036232, + 0.00279375072568655, + 0.006622526329010725, + 0.00718553876504302, + 0.01178906112909317, + 0.012357620522379875, + 0.017192937433719635, + 0.017739810049533844, + 0.02232205867767334, + 0.0035713587421923876, + 0.0074642314575612545, + 0.008664214983582497, + 0.013203916139900684, + 0.014133833348751068, + 0.019116846844553947, + 0.020305152982473373, + 0.02576720155775547, + 0.004881689324975014, + 0.008968878537416458, + 0.011207306757569313, + 0.016232330352067947, + 0.01854393631219864, + 0.023723822087049484, + 0.026541929692029953, + 0.032115157693624496, + 0.005600836127996445, + 0.009892941452562809, + 0.013628173619508743, + 0.01923578418791294, + 0.02294476144015789, + 0.028606966137886047, + 0.03297396004199982, + 0.03876862674951553, + 0.00841668900102377, + 0.012688412331044674, + 0.017100969329476357, + 0.022661328315734863, + 0.028999267145991325, + 0.03460892662405968, + 0.04026596248149872, + 0.04590345546603203, + 0.009537886828184128, + 0.014741362072527409, + 0.023264342918992043, + 0.02941947802901268, + 0.03990020602941513, + 0.046154435724020004, + 0.05633930861949921, + 0.062365636229515076, + 0.014504852704703808, + 0.01962464675307274, + 0.03175453841686249, + 0.03736087307333946, + 0.048590946942567825, + 0.05551042780280113, + 0.06793668866157532, + 0.07462256401777267 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00014375557657331228, + 0.0001485401880927384, + 0.00014388759154826403, + 0.00013949323329143226, + 0.00014151718642096967, + 0.0001298519637202844, + 0.00011761717178160325, + 0.00010920927888946608, + 0.00010430848487885669, + 9.379995753988624e-05, + 8.673781849211082e-05, + 8.073235949268565e-05, + 7.842259219614789e-05, + 7.23400735296309e-05, + 6.553455750690773e-05, + 6.013928941683844e-05, + 6.403862062143162e-05, + 5.7616263802628964e-05, + 5.167445488041267e-05, + 4.685809835791588e-05, + 4.381359394756146e-05, + 3.964460483985022e-05, + 3.6867295420961455e-05, + 3.45685075444635e-05, + 3.144369111396372e-05, + 2.9637134502991103e-05, + 2.836080420820508e-05, + 2.7697469704435207e-05, + 2.6873205570154823e-05, + 2.6534436983638443e-05, + 2.6418079869472422e-05, + 2.6400597562314942e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.0001342340237897588, + "first_to_last_ratio": 4.807692223124583, + "imbalance_abs_log_ratio": 1.5702171816907322, + "last_quartile_mean": 2.7920677439396968e-05, + "mean": 7.2322943196923e-05, + "normalized": [ + 1.9876898010343749, + 2.053846006906672, + 1.9895151550522872, + 1.928754930667244, + 1.9567398693336162, + 1.7954463408205017, + 1.6262774519747047, + 1.510022602262022, + 1.4422599560811915, + 1.2969599050260578, + 1.1993126200068847, + 1.1162759136179683, + 1.0843390593579216, + 1.000236858899136, + 0.9061378673219692, + 0.8315381918721068, + 0.8854537411048857, + 0.7966526423814049, + 0.7144960173939819, + 0.6479008774619319, + 0.6058049079704145, + 0.5481608337191801, + 0.5097593348846177, + 0.47797429164821686, + 0.4347678582209788, + 0.40978883315484343, + 0.39214117891999306, + 0.3829693383608537, + 0.37157234457375, + 0.36688823505688517, + 0.3652793802589105, + 0.36503765465449367 + ], + "population_cv": 0.5772189754804335 + }, + "activation_output_rms_by_block": [ + 0.021791651844978333, + 0.0364595502614975, + 0.049153994768857956, + 0.059493422508239746, + 0.01833995059132576, + 0.0272492878139019, + 0.033587127923965454, + 0.03895771503448486, + 0.017782878130674362, + 0.026316652074456215, + 0.03198830038309097, + 0.037699077278375626, + 0.016956452280282974, + 0.028001772239804268, + 0.03484874218702316, + 0.041589170694351196, + 0.028101151809096336, + 0.032995712012052536, + 0.04130169749259949, + 0.050095975399017334, + 0.025649942457675934, + 0.03788147494196892, + 0.05563920736312866, + 0.07605093717575073, + 0.02350441925227642, + 0.04479469731450081, + 0.06435965746641159, + 0.08624991774559021, + 0.02724459581077099, + 0.05753488466143608, + 0.08987044543027878, + 0.12657007575035095 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.03562908759340644, + "first_to_last_ratio": 0.5480041850156577, + "imbalance_abs_log_ratio": 0.601472355173396, + "last_quartile_mean": 0.06501608667895198, + "mean": 0.04337689181556925, + "normalized": [ + 0.5023792838277238, + 0.8405293402883026, + 1.1331838845865654, + 1.3715464621392213, + 0.42280462761840876, + 0.6281982565685198, + 0.7743092351285077, + 0.8981214052893892, + 0.4099620186315784, + 0.606697505813692, + 0.737450265433943, + 0.8691050856909095, + 0.3909098040583166, + 0.6455458440605374, + 0.8033941743726994, + 0.9587863250133476, + 0.6478369157609859, + 0.760674880817749, + 0.9521589898189774, + 1.1549000701114414, + 0.5913273492885308, + 0.8733100357451645, + 1.2826923514874364, + 1.7532592583882138, + 0.5418649946661228, + 1.0326857328773074, + 1.4837314240968968, + 1.988384002069797, + 0.6280900883034707, + 1.3263948211426506, + 2.0718507405369606, + 2.9179148263666326 + ], + "population_cv": 0.548274291566511 + }, + "bits_per_byte": 3.0603879324756624, + "branch_output_rms_by_sublayer": [ + 0.010862384922802448, + 0.01740431785583496, + 0.012540988624095917, + 0.013136130757629871, + 0.010843372903764248, + 0.013625179417431355, + 0.008067907765507698, + 0.012595787644386292, + 0.01570705883204937, + 0.008440791629254818, + 0.013323793187737465, + 0.00771812628954649, + 0.015350498259067535, + 0.0072539690881967545, + 0.012476769275963306, + 0.0076688784174621105, + 0.015026160515844822, + 0.008182888850569725, + 0.013977059163153172, + 0.00797984842211008, + 0.014992608688771725, + 0.0075123850256204605, + 0.011131634004414082, + 0.007944026961922646, + 0.01328288670629263, + 0.009265917353332043, + 0.01475453656166792, + 0.008776102215051651, + 0.021935302764177322, + 0.009220737963914871, + 0.018072573468089104, + 0.009505237452685833, + 0.02485089935362339, + 0.010875836946070194, + 0.02256050705909729, + 0.011053016409277916, + 0.02060902863740921, + 0.012081403285264969, + 0.01814373955130577, + 0.013695045374333858, + 0.02028532326221466, + 0.014826158992946148, + 0.018819237127900124, + 0.01723981648683548, + 0.02078024297952652, + 0.01904597505927086, + 0.01184111274778843, + 0.02309560962021351, + 0.01320039015263319, + 0.01857767626643181, + 0.01621861383318901, + 0.019791442900896072, + 0.010392188094556332, + 0.02097492292523384, + 0.005777175538241863, + 0.024696551263332367, + 0.00478855986148119, + 0.02572857216000557, + 0.009363405406475067, + 0.027967121452093124, + 0.004846271127462387, + 0.03223826363682747, + 0.008639557287096977, + 0.03540258854627609 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0005179108405560085, + 0.00037263359364603097, + 0.000341406605551914, + 0.0003080215554515325, + 0.00032112382880302863, + 0.00028785731511441636, + 0.00024682250128882333, + 0.0002384133674256824, + 0.00023125674862092528, + 0.00021459684344220906, + 0.0002038946364372164, + 0.00023401733819201806, + 0.00022906447544329808, + 0.0001756494740024643, + 0.0001540733468324371, + 0.0001552833180993548, + 0.00017546531996443928, + 0.00014549405227956783, + 0.00012831985016828094, + 0.00012258241315709424, + 0.00012182441984913601, + 0.00012087189202411873, + 0.00011862312669425283, + 0.00011139038374427566, + 9.473465022993448e-05, + 9.656034839219119e-05, + 8.314568983290554e-05, + 7.881003157320508e-05, + 8.007379322396748e-05, + 8.165564749800254e-05, + 8.399572983254586e-05, + 8.784936942527602e-05 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00032927370097967957, + "first_to_last_ratio": 3.835312649692945, + "imbalance_abs_log_ratio": 1.3442509568398715, + "last_quartile_mean": 8.585315750100352e-05, + "mean": 0.0001863569533373923, + "normalized": [ + 2.7791334387096907, + 1.9995690365864254, + 1.8320035793556362, + 1.6528578619434229, + 1.7231652645750541, + 1.5446556190112286, + 1.324460916904777, + 1.279337117054301, + 1.2409343707301523, + 1.1515365517576879, + 1.0941080157501435, + 1.2557478215923523, + 1.22917053182655, + 0.9425431724270438, + 0.8267646796816487, + 0.8332574412623078, + 0.9415549933721328, + 0.7807277897281154, + 0.6885702297137399, + 0.6577828782978833, + 0.6537154512747235, + 0.6486041430677646, + 0.6365371646717689, + 0.5977259326761343, + 0.5083504990469603, + 0.5181472795242166, + 0.44616360347109435, + 0.4228982614376747, + 0.42967966469701224, + 0.43816796763235355, + 0.4507249572838569, + 0.4714037649361439 + ], + "population_cv": 0.5503799279855914 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6480265855789185, + "mean_weights": [ + 0.37301427125930786, + 0.6269856691360474 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6829240918159485, + "mean_weights": [ + 0.4617732763290405, + 0.5382266640663147 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6888938546180725, + "mean_weights": [ + 0.4829915761947632, + 0.5170084238052368 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6816734075546265, + "mean_weights": [ + 0.4783531129360199, + 0.5216468572616577 + ], + "sources": 2 + }, + { + "entropy_mean": 0.684533417224884, + "mean_weights": [ + 0.45917052030563354, + 0.5408294200897217 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6844513416290283, + "mean_weights": [ + 0.4720883369445801, + 0.5279116630554199 + ], + "sources": 2 + }, + { + "entropy_mean": 0.688705563545227, + "mean_weights": [ + 0.5119128823280334, + 0.48808711767196655 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6821842789649963, + "mean_weights": [ + 0.46196281909942627, + 0.5380371809005737 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0883762836456299, + "mean_weights": [ + 0.28199130296707153, + 0.35414770245552063, + 0.36386096477508545 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0827150344848633, + "mean_weights": [ + 0.30120694637298584, + 0.3128288686275482, + 0.38596415519714355 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0914238691329956, + "mean_weights": [ + 0.29997941851615906, + 0.3302365243434906, + 0.36978405714035034 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0581061840057373, + "mean_weights": [ + 0.2821844220161438, + 0.2822105884552002, + 0.4356050193309784 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0886261463165283, + "mean_weights": [ + 0.2962369918823242, + 0.3197210431098938, + 0.3840419352054596 + ], + "sources": 3 + }, + { + "entropy_mean": 1.063319206237793, + "mean_weights": [ + 0.2859196066856384, + 0.2900146543979645, + 0.4240657091140747 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0868974924087524, + "mean_weights": [ + 0.29099103808403015, + 0.3350713551044464, + 0.37393757700920105 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0388412475585938, + "mean_weights": [ + 0.27100780606269836, + 0.27708399295806885, + 0.4519082307815552 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3791594505310059, + "mean_weights": [ + 0.22300708293914795, + 0.23990149796009064, + 0.2703501582145691, + 0.2667412757873535 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3425464630126953, + "mean_weights": [ + 0.22761863470077515, + 0.2083243876695633, + 0.2512320876121521, + 0.3128248453140259 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3781578540802002, + "mean_weights": [ + 0.23716571927070618, + 0.22990256547927856, + 0.2531399130821228, + 0.27979180216789246 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3557236194610596, + "mean_weights": [ + 0.24008062481880188, + 0.21694886684417725, + 0.23780174553394318, + 0.3051687479019165 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3803355693817139, + "mean_weights": [ + 0.2525424361228943, + 0.239782452583313, + 0.2367197424173355, + 0.27095532417297363 + ], + "sources": 4 + }, + { + "entropy_mean": 1.357678771018982, + "mean_weights": [ + 0.2421601563692093, + 0.21903681755065918, + 0.23288574814796448, + 0.30591726303100586 + ], + "sources": 4 + }, + { + "entropy_mean": 1.380903959274292, + "mean_weights": [ + 0.2556239366531372, + 0.2312958538532257, + 0.24257367849349976, + 0.2705065608024597 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3466907739639282, + "mean_weights": [ + 0.24823860824108124, + 0.21751916408538818, + 0.21974793076515198, + 0.3144943118095398 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6069625616073608, + "mean_weights": [ + 0.19825324416160583, + 0.19237308204174042, + 0.18861883878707886, + 0.20363827049732208, + 0.217116579413414 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5712603330612183, + "mean_weights": [ + 0.19559121131896973, + 0.17229589819908142, + 0.16191908717155457, + 0.2111104428768158, + 0.2590833306312561 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6035795211791992, + "mean_weights": [ + 0.21712158620357513, + 0.18761470913887024, + 0.17587432265281677, + 0.20126870274543762, + 0.21812066435813904 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5762670040130615, + "mean_weights": [ + 0.2016730159521103, + 0.17546913027763367, + 0.16328641772270203, + 0.20039990544319153, + 0.2591715455055237 + ], + "sources": 5 + }, + { + "entropy_mean": 1.599379301071167, + "mean_weights": [ + 0.21151402592658997, + 0.1859227418899536, + 0.17186129093170166, + 0.2021297812461853, + 0.22857213020324707 + ], + "sources": 5 + }, + { + "entropy_mean": 1.575486183166504, + "mean_weights": [ + 0.20965713262557983, + 0.17234230041503906, + 0.16095492243766785, + 0.19662562012672424, + 0.2604200839996338 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6043035984039307, + "mean_weights": [ + 0.2049970179796219, + 0.19548343122005463, + 0.18123121559619904, + 0.19702549278736115, + 0.2212628424167633 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5845153331756592, + "mean_weights": [ + 0.21390476822853088, + 0.17785052955150604, + 0.16699782013893127, + 0.19135682284832, + 0.2498900592327118 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7863740921020508, + "mean_weights": [ + 0.1603921502828598, + 0.17159342765808105, + 0.1527625322341919, + 0.15952156484127045, + 0.17128480970859528, + 0.18444553017616272 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7709953784942627, + "mean_weights": [ + 0.1689251810312271, + 0.15226224064826965, + 0.1435423493385315, + 0.14707553386688232, + 0.17815452814102173, + 0.2100401669740677 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7821707725524902, + "mean_weights": [ + 0.16392186284065247, + 0.16362451016902924, + 0.14519917964935303, + 0.1562623530626297, + 0.1684192419052124, + 0.20257285237312317 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7748383283615112, + "mean_weights": [ + 0.1656128615140915, + 0.15761655569076538, + 0.14687448740005493, + 0.15072563290596008, + 0.1669767051935196, + 0.21219374239444733 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7809157371520996, + "mean_weights": [ + 0.15301954746246338, + 0.17165207862854004, + 0.15380950272083282, + 0.15615780651569366, + 0.16156405210494995, + 0.20379702746868134 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7780534029006958, + "mean_weights": [ + 0.17337235808372498, + 0.16322240233421326, + 0.1526409089565277, + 0.14656966924667358, + 0.16009068489074707, + 0.2041039913892746 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7756729125976562, + "mean_weights": [ + 0.14886975288391113, + 0.17834702134132385, + 0.15431863069534302, + 0.15470074117183685, + 0.15853270888328552, + 0.20523114502429962 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7775068283081055, + "mean_weights": [ + 0.17614689469337463, + 0.1613134741783142, + 0.1462908685207367, + 0.14917539060115814, + 0.16138209402561188, + 0.20569126307964325 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9368503093719482, + "mean_weights": [ + 0.12421485036611557, + 0.15696582198143005, + 0.14077812433242798, + 0.13455674052238464, + 0.13156186044216156, + 0.15624842047691345, + 0.15567418932914734 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9427490234375, + "mean_weights": [ + 0.142055481672287, + 0.14708712697029114, + 0.14162249863147736, + 0.1358705759048462, + 0.13455010950565338, + 0.1510104238986969, + 0.14780378341674805 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9337153434753418, + "mean_weights": [ + 0.12284686416387558, + 0.16048544645309448, + 0.1446741819381714, + 0.1339469850063324, + 0.12917141616344452, + 0.15231747925281525, + 0.1565576195716858 + ], + "sources": 7 + }, + { + "entropy_mean": 1.940615177154541, + "mean_weights": [ + 0.14123499393463135, + 0.14938606321811676, + 0.14051571488380432, + 0.13213014602661133, + 0.13056766986846924, + 0.14648398756980896, + 0.15968143939971924 + ], + "sources": 7 + }, + { + "entropy_mean": 1.934278130531311, + "mean_weights": [ + 0.11990150809288025, + 0.16120871901512146, + 0.1448434591293335, + 0.13822346925735474, + 0.1333983838558197, + 0.15654350817203522, + 0.14588096737861633 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9416944980621338, + "mean_weights": [ + 0.14197370409965515, + 0.15145939588546753, + 0.1453356146812439, + 0.13323301076889038, + 0.13024437427520752, + 0.13977493345737457, + 0.15797898173332214 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9245680570602417, + "mean_weights": [ + 0.11064852774143219, + 0.17740216851234436, + 0.1620495319366455, + 0.14172744750976562, + 0.13177362084388733, + 0.1467466950416565, + 0.12965205311775208 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9388952255249023, + "mean_weights": [ + 0.14636772871017456, + 0.15082837641239166, + 0.13772042095661163, + 0.12900781631469727, + 0.12763038277626038, + 0.14589904248714447, + 0.16254623234272003 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.066920042037964, + "mean_weights": [ + 0.11896759271621704, + 0.14457373321056366, + 0.12346737086772919, + 0.11815924942493439, + 0.11684338748455048, + 0.13027724623680115, + 0.12518295645713806, + 0.12252847850322723 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.062781572341919, + "mean_weights": [ + 0.11645344644784927, + 0.15088656544685364, + 0.1277526319026947, + 0.12101788818836212, + 0.11955052614212036, + 0.12647059559822083, + 0.12139534205198288, + 0.116472989320755 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.07089900970459, + "mean_weights": [ + 0.1149415373802185, + 0.1418081820011139, + 0.13560262322425842, + 0.12379468232393265, + 0.12024582177400589, + 0.12306073307991028, + 0.12177915871143341, + 0.11876726895570755 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.074303150177002, + "mean_weights": [ + 0.11992637068033218, + 0.13782553374767303, + 0.12513059377670288, + 0.12244556844234467, + 0.12399850785732269, + 0.12712916731834412, + 0.12411939352750778, + 0.11942484974861145 + ], + "sources": 8 + }, + { + "entropy_mean": 2.129868984222412, + "mean_weights": [ + 0.08378762751817703, + 0.165731281042099, + 0.1604594737291336, + 0.12277983129024506, + 0.1079399585723877, + 0.11341231316328049, + 0.09277388453483582, + 0.07517717778682709, + 0.07793846726417542 + ], + "sources": 9 + }, + { + "entropy_mean": 2.176992177963257, + "mean_weights": [ + 0.10307709872722626, + 0.13544538617134094, + 0.11533261090517044, + 0.1129159927368164, + 0.11404357850551605, + 0.12311488389968872, + 0.10620607435703278, + 0.09431935101747513, + 0.09554500877857208 + ], + "sources": 9 + }, + { + "entropy_mean": 2.130573272705078, + "mean_weights": [ + 0.08479803800582886, + 0.16635021567344666, + 0.1631554365158081, + 0.12319765985012054, + 0.10755977034568787, + 0.11119093000888824, + 0.09616850316524506, + 0.0778317004442215, + 0.06974773108959198 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1894257068634033, + "mean_weights": [ + 0.10593724250793457, + 0.12605343759059906, + 0.1170620545744896, + 0.11008726805448532, + 0.11116088926792145, + 0.11413641273975372, + 0.11201208084821701, + 0.10360297560691833, + 0.09994763135910034 + ], + "sources": 9 + }, + { + "entropy_mean": 2.119298219680786, + "mean_weights": [ + 0.0832459032535553, + 0.1668640375137329, + 0.1675834208726883, + 0.12600407004356384, + 0.10965664684772491, + 0.11541351675987244, + 0.09425166249275208, + 0.07515774667263031, + 0.061823006719350815 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1856038570404053, + "mean_weights": [ + 0.10951056331396103, + 0.12276074290275574, + 0.1070999950170517, + 0.1092681884765625, + 0.11565889418125153, + 0.12220809608697891, + 0.11049125343561172, + 0.1012752428650856, + 0.10172701627016068 + ], + "sources": 9 + }, + { + "entropy_mean": 2.115971088409424, + "mean_weights": [ + 0.08332230150699615, + 0.16771969199180603, + 0.1714719533920288, + 0.12766629457473755, + 0.10908275842666626, + 0.11333589255809784, + 0.09316173195838928, + 0.07521909475326538, + 0.05902029573917389 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.043827567249536514, + 0.01710854098200798, + 0.024112526327371597, + 0.026274539530277252, + 0.029621778056025505, + 0.030753593891859055, + 0.0343739353120327, + 0.034574415534734726, + 0.03859054669737816, + 0.025725573301315308, + 0.026694709435105324, + 0.02721203677356243, + 0.0280117429792881, + 0.02873404324054718, + 0.030105754733085632, + 0.030343888327479362, + 0.03175647556781769, + 0.02320888638496399, + 0.02432127296924591, + 0.023536842316389084, + 0.025095980614423752, + 0.02469342201948166, + 0.025839107111096382, + 0.024960478767752647, + 0.02684049680829048, + 0.020482981577515602, + 0.021665707230567932, + 0.02088664472103119, + 0.022258764132857323, + 0.021422095596790314, + 0.022909455001354218, + 0.022066516801714897, + 0.023365508764982224, + 0.018776340410113335, + 0.019467856734991074, + 0.018957460299134254, + 0.019994201138615608, + 0.019893784075975418, + 0.020612923428416252, + 0.02078467421233654, + 0.02170618437230587, + 0.01841813512146473, + 0.019014405086636543, + 0.019375916570425034, + 0.020207909867167473, + 0.020897099748253822, + 0.021754663437604904, + 0.021853726357221603, + 0.024258248507976532, + 0.020733773708343506, + 0.0220684502273798, + 0.022030605003237724, + 0.02330145426094532, + 0.023511003702878952, + 0.024530913680791855, + 0.025112302973866463, + 0.026648549363017082, + 0.022414876148104668, + 0.02524503879249096, + 0.024028129875659943, + 0.028029751032590866, + 0.02510964311659336, + 0.03090256080031395, + 0.026371758431196213 + ], + "loss_nats": 2.1212992668151855, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1655757427215576, + "mean_weights": [ + 0.12656879425048828, + 0.0883428230881691, + 0.09173233807086945, + 0.08944092690944672, + 0.09264708310365677, + 0.10132450610399246, + 0.13680590689182281, + 0.13590916991233826, + 0.13722845911979675 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.010862384922802448, + 0.021791651844978333, + 0.02804987132549286, + 0.0364595502614975, + 0.04057689011096954, + 0.049153994768857956, + 0.0510152168571949, + 0.059493422508239746, + 0.01570705883204937, + 0.01833995059132576, + 0.02399817854166031, + 0.0272492878139019, + 0.030342457816004753, + 0.033587127923965454, + 0.03538822382688522, + 0.03895771503448486, + 0.015026160515844822, + 0.017782878130674362, + 0.022754305973649025, + 0.026316652074456215, + 0.028262270614504814, + 0.03198830038309097, + 0.03345726802945137, + 0.037699077278375626, + 0.01328288670629263, + 0.016956452280282974, + 0.023850729689002037, + 0.028001772239804268, + 0.030193978920578957, + 0.03484874218702316, + 0.03633309528231621, + 0.041589170694351196, + 0.02485089935362339, + 0.028101151809096336, + 0.027999818325042725, + 0.032995712012052536, + 0.03476347401738167, + 0.04130169749259949, + 0.04187580570578575, + 0.050095975399017334, + 0.02028532326221466, + 0.025649942457675934, + 0.026793109253048897, + 0.03788147494196892, + 0.04176015406847, + 0.05563920736312866, + 0.05724496394395828, + 0.07605093717575073, + 0.01320039015263319, + 0.02350441925227642, + 0.030474599450826645, + 0.04479469731450081, + 0.047474056482315063, + 0.06435965746641159, + 0.06577937304973602, + 0.08624991774559021, + 0.00478855986148119, + 0.02724459581077099, + 0.03159519284963608, + 0.05753488466143608, + 0.05940619856119156, + 0.08987044543027878, + 0.0930284634232521, + 0.12657007575035095 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.369824015768245e-05, + 7.647161692148075e-05, + 7.418341556331143e-05, + 7.0989626692608e-05, + 8.238116424763575e-05, + 8.409100701101124e-05, + 7.501178333768621e-05, + 6.846054020570591e-05, + 6.591735291294754e-05, + 6.109200330683962e-05, + 5.751167918788269e-05, + 5.4902153351577e-05, + 5.166455957805738e-05, + 5.07987460878212e-05, + 5.19509230798576e-05, + 5.175487240194343e-05, + 7.442531205015257e-05, + 6.937842408660799e-05, + 6.340057007037103e-05, + 5.818115096190013e-05, + 8.599830471212044e-05, + 7.34730128897354e-05, + 6.025347101967782e-05, + 4.983972030458972e-05, + 3.323980490677059e-05, + 3.103911876678467e-05, + 2.9544622520916164e-05, + 2.8530046620289795e-05, + 3.480308077996597e-05, + 3.140621629427187e-05, + 3.04021559713874e-05, + 3.000968172273133e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 7.566092426714022e-05, + "first_to_last_ratio": 2.431119816910142, + "imbalance_abs_log_ratio": 0.8883519812070944, + "last_quartile_mean": 3.112184094788972e-05, + "mean": 5.7337636803822534e-05, + "normalized": [ + 1.2853379432053817, + 1.3337071631173785, + 1.2937996697897711, + 1.2380982309315427, + 1.4367729268211424, + 1.4665935273670914, + 1.3082468605103967, + 1.1939895681424708, + 1.1496349795245313, + 1.0654782218503778, + 1.0030353951394202, + 0.9575238257450616, + 0.9010584052290947, + 0.8859581405774749, + 0.9060527425921778, + 0.9026335106732736, + 1.2980184778942765, + 1.2099979691172542, + 1.105740899076334, + 1.0147113520036348, + 1.4998578508974612, + 1.2814098554692643, + 1.050853756422359, + 0.8692322021417361, + 0.5797205249406894, + 0.5413393452712962, + 0.5152745067258598, + 0.4975797436141941, + 0.6069849181095958, + 0.5477417285565231, + 0.5302303629186297, + 0.5233853956243043 + ], + "population_cv": 0.3122893464031469 + }, + "activation_output_rms_by_block": [ + 0.09054390341043472, + 0.15911602973937988, + 0.1929183304309845, + 0.2085827887058258, + 0.10474638640880585, + 0.12492278963327408, + 0.13526350259780884, + 0.1433677226305008, + 0.03818253427743912, + 0.06309886276721954, + 0.08916693180799484, + 0.12196294963359833, + 0.0942109227180481, + 0.13388976454734802, + 0.15764053165912628, + 0.1836506575345993, + 0.04554453864693642, + 0.07187420129776001, + 0.0935249924659729, + 0.12195224314928055, + 0.052297890186309814, + 0.07755465060472488, + 0.10850540548563004, + 0.14139576256275177, + 0.06609684973955154, + 0.10618496686220169, + 0.14159242808818817, + 0.18979515135288239, + 0.06161881983280182, + 0.12800303101539612, + 0.19602705538272858, + 0.29343071579933167 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.1449326816946268, + "first_to_last_ratio": 0.9803106456566691, + "imbalance_abs_log_ratio": 0.019885772182817476, + "last_quartile_mean": 0.14784362725913525, + "mean": 0.12302072846796364, + "normalized": [ + 0.7360052613736039, + 1.293408287537624, + 1.5681774350834152, + 1.69550929590003, + 0.8514531470693031, + 1.0154613063099018, + 1.0995179778425177, + 1.1653948437464814, + 0.310374802303194, + 0.5129124461626919, + 0.724812256588248, + 0.9914016211126504, + 0.7658134030850091, + 1.088351257668051, + 1.2814143894472072, + 1.4928431966034454, + 0.3702184112720229, + 0.5842446406621392, + 0.7602376740138402, + 0.9913145911913427, + 0.425114457031759, + 0.6304193737961911, + 0.8820091283550349, + 1.1493653494303055, + 0.5372822170920905, + 0.8631469524248003, + 1.1509639867322106, + 1.5427900136545507, + 0.5008816042643435, + 1.0404996995992424, + 1.5934473630903334, + 2.385213609556419 + ], + "population_cv": 0.44713111656136534 + }, + "bits_per_byte": 2.0641206411821535, + "branch_output_rms_by_sublayer": [ + 0.04541910067200661, + 0.07549113035202026, + 0.0863218680024147, + 0.0204075388610363, + 0.049760881811380386, + 0.008669846691191196, + 0.02478516660630703, + 0.02664596028625965, + 0.10324621945619583, + 0.008682899177074432, + 0.035070084035396576, + 0.006455700844526291, + 0.0327182300388813, + 0.008789212442934513, + 0.03248058632016182, + 0.022402049973607063, + 0.03211447596549988, + 0.02033414877951145, + 0.026294970884919167, + 0.022924279794096947, + 0.024928947910666466, + 0.025966202840209007, + 0.03280605748295784, + 0.028406478464603424, + 0.08478779345750809, + 0.026699643582105637, + 0.03866994008421898, + 0.024151086807250977, + 0.031930942088365555, + 0.02442363277077675, + 0.0336761437356472, + 0.025760695338249207, + 0.03393910825252533, + 0.028162403032183647, + 0.030614903196692467, + 0.029901737347245216, + 0.02581537701189518, + 0.03392363339662552, + 0.02952393889427185, + 0.04758211597800255, + 0.03632239252328873, + 0.03465477004647255, + 0.027726948261260986, + 0.03837638720870018, + 0.03740985691547394, + 0.045615170150995255, + 0.031099481508135796, + 0.05717000365257263, + 0.04496077820658684, + 0.04406982287764549, + 0.041102323681116104, + 0.045916710048913956, + 0.02187858335673809, + 0.05305280536413193, + 0.0147908516228199, + 0.07477929443120956, + 0.009636363945901394, + 0.06045869365334511, + 0.015359818004071712, + 0.074136883020401, + 0.012710005976259708, + 0.07667002826929092, + 0.017902133986353874, + 0.11213062703609467 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00026598823390359727, + 0.00015875542823679532, + 0.00012878203669635153, + 0.00013460550176353457, + 0.0001642930232525195, + 0.0001287828691869054, + 0.00011767249499820491, + 0.00014245088811067115, + 0.00015153023229956872, + 0.00013381904040453275, + 0.00013441954952457836, + 0.00014689987707757415, + 0.00016401246063902104, + 0.00013077065769700926, + 0.00014368035403220883, + 0.0001616833550013946, + 0.0002327383928333144, + 0.0002550839164603749, + 0.00024446360959694225, + 0.00026877838284911375, + 0.0003838962202763265, + 0.00032727214771167046, + 0.0003095439968173968, + 0.00028184024167808157, + 0.00015699859448374935, + 0.0001439855761543242, + 0.00013101920497257446, + 0.00013592949025736044, + 0.00016692578616636617, + 0.00016204530410100578, + 0.00015242782230531033, + 0.00015981352620779232 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00015516630951857246, + "first_to_last_ratio": 1.0266181172571756, + "imbalance_abs_log_ratio": 0.026270018813374475, + "last_quartile_mean": 0.0001511431630810604, + "mean": 0.00018502838174050534, + "normalized": [ + 1.4375536952846228, + 0.8580058191258638, + 0.6960123386744149, + 0.7274856997469352, + 0.8879341737038682, + 0.6960168379331018, + 0.6359699733159625, + 0.7698866885754458, + 0.8189566966655073, + 0.7232352093540353, + 0.7264807066901565, + 0.7939315887418567, + 0.8864178516625721, + 0.7067599925313605, + 0.7765314309115808, + 0.8738300226186317, + 1.2578523934761554, + 1.3786204800629969, + 1.3212222216794414, + 1.4526332680467602, + 2.074796399693562, + 1.7687672811766582, + 1.6729541376604562, + 1.523227080228972, + 0.8485108771255071, + 0.7781810271478137, + 0.7081032852371993, + 0.7346412965336093, + 0.9021631416550607, + 0.8757862041309297, + 0.8238077903047547, + 0.8637243903042084 + ], + "population_cv": 0.3688341566487421 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.3785790205001831, + "mean_weights": [ + 0.8629493713378906, + 0.13705064356327057 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6319173574447632, + "mean_weights": [ + 0.5910738110542297, + 0.40892618894577026 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5624884366989136, + "mean_weights": [ + 0.7353614568710327, + 0.2646385431289673 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6834747791290283, + "mean_weights": [ + 0.5238999128341675, + 0.47610002756118774 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6686824560165405, + "mean_weights": [ + 0.5394197106361389, + 0.4605802893638611 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6902142763137817, + "mean_weights": [ + 0.4950644075870514, + 0.504935622215271 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5484784841537476, + "mean_weights": [ + 0.7485502362251282, + 0.2514497637748718 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6861746311187744, + "mean_weights": [ + 0.5332672595977783, + 0.4667327105998993 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0391020774841309, + "mean_weights": [ + 0.26409250497817993, + 0.2662324905395508, + 0.4696750044822693 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0883302688598633, + "mean_weights": [ + 0.3061772584915161, + 0.30532920360565186, + 0.3884935975074768 + ], + "sources": 3 + }, + { + "entropy_mean": 1.012488842010498, + "mean_weights": [ + 0.27766138315200806, + 0.21789968013763428, + 0.5044389963150024 + ], + "sources": 3 + }, + { + "entropy_mean": 1.080611228942871, + "mean_weights": [ + 0.32859355211257935, + 0.2614467740058899, + 0.40995967388153076 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0074419975280762, + "mean_weights": [ + 0.3071908950805664, + 0.19184687733650208, + 0.5009621977806091 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0840950012207031, + "mean_weights": [ + 0.33475595712661743, + 0.26622527837753296, + 0.3990187644958496 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0398133993148804, + "mean_weights": [ + 0.28682976961135864, + 0.2383115589618683, + 0.4748586416244507 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0893466472625732, + "mean_weights": [ + 0.3625243902206421, + 0.274220734834671, + 0.3632549047470093 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3550174236297607, + "mean_weights": [ + 0.27371320128440857, + 0.16702106595039368, + 0.2583821713924408, + 0.30088353157043457 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3777035474777222, + "mean_weights": [ + 0.28488677740097046, + 0.21318326890468597, + 0.23018285632133484, + 0.2717471122741699 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3640902042388916, + "mean_weights": [ + 0.29274454712867737, + 0.1804964393377304, + 0.24847473204135895, + 0.27828431129455566 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3747847080230713, + "mean_weights": [ + 0.28755056858062744, + 0.22763943672180176, + 0.21047532558441162, + 0.27433469891548157 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3455053567886353, + "mean_weights": [ + 0.309937059879303, + 0.19309401512145996, + 0.19672107696533203, + 0.30024784803390503 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3786201477050781, + "mean_weights": [ + 0.2745283544063568, + 0.21776717901229858, + 0.24027034640312195, + 0.26743412017822266 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3696016073226929, + "mean_weights": [ + 0.2854727804660797, + 0.21360868215560913, + 0.21188855171203613, + 0.28902995586395264 + ], + "sources": 4 + }, + { + "entropy_mean": 1.378506064414978, + "mean_weights": [ + 0.2691721022129059, + 0.24826863408088684, + 0.2384830117225647, + 0.24407625198364258 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5936660766601562, + "mean_weights": [ + 0.1776367872953415, + 0.16571632027626038, + 0.2039409577846527, + 0.2351239174604416, + 0.21758201718330383 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5917437076568604, + "mean_weights": [ + 0.20114099979400635, + 0.19412773847579956, + 0.15619206428527832, + 0.18938219547271729, + 0.2591570317745209 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5984643697738647, + "mean_weights": [ + 0.22242119908332825, + 0.16508570313453674, + 0.1959630846977234, + 0.21597975492477417, + 0.20055025815963745 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5458571910858154, + "mean_weights": [ + 0.1990240216255188, + 0.1777437925338745, + 0.11689791083335876, + 0.1839444637298584, + 0.3223898112773895 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5985828638076782, + "mean_weights": [ + 0.19882062077522278, + 0.16752663254737854, + 0.20795005559921265, + 0.2100883573293686, + 0.21561430394649506 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5318646430969238, + "mean_weights": [ + 0.2090664803981781, + 0.2141328752040863, + 0.10095593333244324, + 0.14793333411216736, + 0.327911376953125 + ], + "sources": 5 + }, + { + "entropy_mean": 1.595567226409912, + "mean_weights": [ + 0.18717510998249054, + 0.17522573471069336, + 0.20343410968780518, + 0.19863323867321014, + 0.2355317771434784 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5768626928329468, + "mean_weights": [ + 0.21477176249027252, + 0.2025144100189209, + 0.1270410418510437, + 0.1795101910829544, + 0.27616259455680847 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7713873386383057, + "mean_weights": [ + 0.13665345311164856, + 0.1412113904953003, + 0.17177271842956543, + 0.17551785707473755, + 0.15357449650764465, + 0.22127008438110352 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7362558841705322, + "mean_weights": [ + 0.1576215922832489, + 0.15164780616760254, + 0.09272358566522598, + 0.13628451526165009, + 0.19581863284111023, + 0.26590386033058167 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7325472831726074, + "mean_weights": [ + 0.12724190950393677, + 0.12745115160942078, + 0.1573503017425537, + 0.1700989454984665, + 0.13188403844833374, + 0.2859736382961273 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7467896938323975, + "mean_weights": [ + 0.15852348506450653, + 0.1697232723236084, + 0.10331815481185913, + 0.13046924769878387, + 0.17749559879302979, + 0.2604702115058899 + ], + "sources": 6 + }, + { + "entropy_mean": 1.71176016330719, + "mean_weights": [ + 0.1224367544054985, + 0.12678959965705872, + 0.15483975410461426, + 0.16201251745224, + 0.12763339281082153, + 0.3062880039215088 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7550915479660034, + "mean_weights": [ + 0.1658829152584076, + 0.17585307359695435, + 0.11018689721822739, + 0.13068366050720215, + 0.1698903739452362, + 0.24750308692455292 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6976137161254883, + "mean_weights": [ + 0.12436187267303467, + 0.1280168890953064, + 0.1539280116558075, + 0.16092172265052795, + 0.12161167711019516, + 0.3111598491668701 + ], + "sources": 6 + }, + { + "entropy_mean": 1.75698721408844, + "mean_weights": [ + 0.17672914266586304, + 0.18012964725494385, + 0.09703885763883591, + 0.13517314195632935, + 0.2119397521018982, + 0.19898945093154907 + ], + "sources": 6 + }, + { + "entropy_mean": 1.8445148468017578, + "mean_weights": [ + 0.09769105166196823, + 0.10662040114402771, + 0.11756686866283417, + 0.11771026253700256, + 0.09032826125621796, + 0.20362034440040588, + 0.2664628326892853 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8765921592712402, + "mean_weights": [ + 0.1309974491596222, + 0.1495974063873291, + 0.08368416130542755, + 0.0985550805926323, + 0.14340372383594513, + 0.13144147396087646, + 0.26232069730758667 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7946264743804932, + "mean_weights": [ + 0.0981016457080841, + 0.09857462346553802, + 0.10899582505226135, + 0.11200127750635147, + 0.08993236720561981, + 0.15801092982292175, + 0.3343833088874817 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8902618885040283, + "mean_weights": [ + 0.1488025188446045, + 0.1571892350912094, + 0.08771530538797379, + 0.10225321352481842, + 0.155007004737854, + 0.11301824450492859, + 0.2360144853591919 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8123081922531128, + "mean_weights": [ + 0.10747420787811279, + 0.10774987936019897, + 0.10940393805503845, + 0.11130791902542114, + 0.09214504808187485, + 0.14360564947128296, + 0.3283133804798126 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9136146306991577, + "mean_weights": [ + 0.14937004446983337, + 0.15937957167625427, + 0.11041536182165146, + 0.10850278288125992, + 0.14625826478004456, + 0.11373643577098846, + 0.21233755350112915 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8280003070831299, + "mean_weights": [ + 0.10496237128973007, + 0.12596656382083893, + 0.11722403019666672, + 0.11585421860218048, + 0.09930695593357086, + 0.12825369834899902, + 0.3084321618080139 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9274578094482422, + "mean_weights": [ + 0.15255476534366608, + 0.16103008389472961, + 0.10426870733499527, + 0.11807939410209656, + 0.16208630800247192, + 0.12749387323856354, + 0.17448684573173523 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0493292808532715, + "mean_weights": [ + 0.12308195978403091, + 0.14865350723266602, + 0.0940496176481247, + 0.09726278483867645, + 0.1432391107082367, + 0.10577862709760666, + 0.11951292306184769, + 0.16842147707939148 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.047658681869507, + "mean_weights": [ + 0.11295606195926666, + 0.12999364733695984, + 0.1088743656873703, + 0.11889934539794922, + 0.11686582863330841, + 0.11139428615570068, + 0.12768806517124176, + 0.17332839965820312 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.042644500732422, + "mean_weights": [ + 0.10885846614837646, + 0.12075205147266388, + 0.11695031821727753, + 0.12590157985687256, + 0.10690770298242569, + 0.10355024039745331, + 0.1465778797864914, + 0.17050176858901978 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0473461151123047, + "mean_weights": [ + 0.09974034130573273, + 0.12601330876350403, + 0.11863301694393158, + 0.13625362515449524, + 0.11470372974872589, + 0.11995761841535568, + 0.15510375797748566, + 0.1295946091413498 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1051414012908936, + "mean_weights": [ + 0.0820235162973404, + 0.11782269179821014, + 0.09505119919776917, + 0.09980016201734543, + 0.08099686354398727, + 0.10466700047254562, + 0.17721277475357056, + 0.12238407135009766, + 0.12004172801971436 + ], + "sources": 9 + }, + { + "entropy_mean": 2.129777669906616, + "mean_weights": [ + 0.09321202337741852, + 0.11980882287025452, + 0.10961809009313583, + 0.1204107254743576, + 0.1079489216208458, + 0.11523287743330002, + 0.12839967012405396, + 0.10353244096040726, + 0.1018364205956459 + ], + "sources": 9 + }, + { + "entropy_mean": 2.082833766937256, + "mean_weights": [ + 0.08703804016113281, + 0.11234879493713379, + 0.09064684063196182, + 0.09352520853281021, + 0.0825805738568306, + 0.08128685504198074, + 0.20930230617523193, + 0.13082829117774963, + 0.11244310438632965 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1616268157958984, + "mean_weights": [ + 0.09485868364572525, + 0.10483861714601517, + 0.11121607571840286, + 0.12326616048812866, + 0.09396112710237503, + 0.1151079535484314, + 0.13723857700824738, + 0.1176939457654953, + 0.10181884467601776 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0905916690826416, + "mean_weights": [ + 0.08528421819210052, + 0.12445193529129028, + 0.09768779575824738, + 0.10738261044025421, + 0.09479787945747375, + 0.09861178696155548, + 0.1932145059108734, + 0.1090645045042038, + 0.08950474858283997 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1701302528381348, + "mean_weights": [ + 0.10169848054647446, + 0.11024472862482071, + 0.11908145248889923, + 0.12035005539655685, + 0.10615777969360352, + 0.11186987906694412, + 0.11161527782678604, + 0.11030349135398865, + 0.10867886245250702 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1155271530151367, + "mean_weights": [ + 0.08723650872707367, + 0.12527386844158173, + 0.11025813221931458, + 0.11668714880943298, + 0.09288282692432404, + 0.09946630895137787, + 0.18652436137199402, + 0.10540175437927246, + 0.07626909017562866 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05597589537501335, + 0.04939329996705055, + 0.05675295367836952, + 0.06440462172031403, + 0.08781132847070694, + 0.09883423149585724, + 0.10768372565507889, + 0.07312089949846268, + 0.10836368054151535, + 0.08462996780872345, + 0.08768022805452347, + 0.08814864605665207, + 0.08735004812479019, + 0.08754805475473404, + 0.08830565959215164, + 0.08891451358795166, + 0.08438876271247864, + 0.05662165954709053, + 0.06177014857530594, + 0.05751814320683479, + 0.0617704801261425, + 0.05530981346964836, + 0.0622403547167778, + 0.05820152163505554, + 0.06727015227079391, + 0.04676000401377678, + 0.05128386616706848, + 0.04851866886019707, + 0.0596720390021801, + 0.0514136403799057, + 0.0679532065987587, + 0.05612101033329964, + 0.06578994542360306, + 0.04387206211686134, + 0.050481073558330536, + 0.04262445122003555, + 0.05157875269651413, + 0.045482736080884933, + 0.05289990082383156, + 0.049256831407547, + 0.0588146336376667, + 0.03938162699341774, + 0.044395361095666885, + 0.039797890931367874, + 0.04719335585832596, + 0.044653378427028656, + 0.04740528762340546, + 0.048932768404483795, + 0.04969678446650505, + 0.04043988883495331, + 0.04311828687787056, + 0.041346948593854904, + 0.04431864619255066, + 0.04291094094514847, + 0.04875316470861435, + 0.04512839391827583, + 0.0516490675508976, + 0.04950067028403282, + 0.04878075793385506, + 0.05538026615977287, + 0.05302857235074043, + 0.0563909187912941, + 0.0590168833732605, + 0.057400740683078766 + ], + "loss_nats": 1.430739402770996, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.9618405103683472, + "mean_weights": [ + 0.11917448043823242, + 0.08477270603179932, + 0.0747370645403862, + 0.06066003441810608, + 0.07317335158586502, + 0.048047471791505814, + 0.11872966587543488, + 0.15306055545806885, + 0.26764464378356934 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.04541910067200661, + 0.09054390341043472, + 0.14894741773605347, + 0.15911602973937988, + 0.18967153131961823, + 0.1929183304309845, + 0.19687645137310028, + 0.2085827887058258, + 0.10324621945619583, + 0.10474638640880585, + 0.12396237999200821, + 0.12492278963327408, + 0.13394367694854736, + 0.13526350259780884, + 0.1393548846244812, + 0.1433677226305008, + 0.03211447596549988, + 0.03818253427743912, + 0.05139917507767677, + 0.06309886276721954, + 0.07576087862253189, + 0.08916693180799484, + 0.10542670637369156, + 0.12196294963359833, + 0.08478779345750809, + 0.0942109227180481, + 0.12496060132980347, + 0.13388976454734802, + 0.14716601371765137, + 0.15764053165912628, + 0.1723417192697525, + 0.1836506575345993, + 0.03393910825252533, + 0.04554453864693642, + 0.058309394866228104, + 0.07187420129776001, + 0.07736696302890778, + 0.0935249924659729, + 0.1000538021326065, + 0.12195224314928055, + 0.03632239252328873, + 0.052297890186309814, + 0.059747010469436646, + 0.07755465060472488, + 0.08864789456129074, + 0.10850540548563004, + 0.11517009139060974, + 0.14139576256275177, + 0.04496077820658684, + 0.06609684973955154, + 0.08208306133747101, + 0.10618496686220169, + 0.11026047170162201, + 0.14159242808818817, + 0.14254237711429596, + 0.18979515135288239, + 0.009636363945901394, + 0.06161881983280182, + 0.06500925868749619, + 0.12800303101539612, + 0.13073177635669708, + 0.19602705538272858, + 0.1991204023361206, + 0.29343071579933167 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.635422323597595e-05, + 6.822785508120432e-05, + 6.660076905973256e-05, + 6.435788964154199e-05, + 7.44270219001919e-05, + 7.514818571507931e-05, + 6.738647789461538e-05, + 6.192061846377328e-05, + 6.186081736814231e-05, + 5.723660797229968e-05, + 5.3653307986678556e-05, + 5.0935963372467086e-05, + 4.907218681182712e-05, + 4.8334790335502476e-05, + 4.9843765737023205e-05, + 5.065252844360657e-05, + 7.570943853352219e-05, + 6.963799387449399e-05, + 6.188120460137725e-05, + 5.612407039734535e-05, + 8.995633106678724e-05, + 7.592375186504796e-05, + 6.122550257714465e-05, + 4.897315375274047e-05, + 3.343266507727094e-05, + 3.1235002097673714e-05, + 2.963913902931381e-05, + 2.8734533771057613e-05, + 3.6414166970644146e-05, + 3.128827302134596e-05, + 2.9787350285914727e-05, + 2.9325223295018077e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 6.805288012401434e-05, + "first_to_last_ratio": 2.1789441543538923, + "imbalance_abs_log_ratio": 0.7788404266087634, + "last_quartile_mean": 3.123204419352987e-05, + "mean": 5.485315028863624e-05, + "normalized": [ + 1.2096702356532174, + 1.2438274688361677, + 1.2141648876916014, + 1.1732760890284695, + 1.356841339258699, + 1.3699885115011803, + 1.2284887475018118, + 1.1288434326551557, + 1.127753229169735, + 1.0434516098186106, + 0.9781262827085748, + 0.9285877493716042, + 0.8946101828903208, + 0.8811670812189586, + 0.9086764474737777, + 0.9234205907422622, + 1.3802204273617933, + 1.2695349949466928, + 1.1281248984927852, + 1.0231695007856714, + 1.639948309138832, + 1.3841274651599518, + 1.1161711269995835, + 0.8928047613499801, + 0.6094939835059406, + 0.5694295028328495, + 0.5403361315321585, + 0.5238447312480147, + 0.663848234404651, + 0.5704006580607979, + 0.5430380958828117, + 0.5346132927773393 + ], + "population_cv": 0.3005545074205614 + }, + "activation_output_rms_by_block": [ + 0.12735538184642792, + 0.22348231077194214, + 0.26064634323120117, + 0.277464359998703, + 0.13131454586982727, + 0.1476469337940216, + 0.16177311539649963, + 0.17404833436012268, + 0.052131909877061844, + 0.08374633640050888, + 0.11783560365438461, + 0.16801035404205322, + 0.10621181130409241, + 0.15759189426898956, + 0.18511100113391876, + 0.21714943647384644, + 0.0585859939455986, + 0.09142377227544785, + 0.1213267594575882, + 0.16095399856567383, + 0.06635415554046631, + 0.10004211962223053, + 0.14167872071266174, + 0.18646042048931122, + 0.08901729434728622, + 0.1387544423341751, + 0.18761782348155975, + 0.23980984091758728, + 0.07888609170913696, + 0.15977811813354492, + 0.2390616536140442, + 0.3599124848842621 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.18796641565859318, + "first_to_last_ratio": 1.007297226943363, + "imbalance_abs_log_ratio": 0.007270731002680294, + "last_quartile_mean": 0.18660471867769957, + "mean": 0.156599480076693, + "normalized": [ + 0.8132554577068641, + 1.427094845158451, + 1.6644138480124728, + 1.7718089476594536, + 0.8385375596746382, + 0.9428315708437411, + 1.0330373722650477, + 1.1114234496492854, + 0.332899635755692, + 0.5347804242995888, + 0.7524648459667671, + 1.0728666146258714, + 0.6782385947391157, + 1.0063372764188652, + 1.1820665116082285, + 1.3866549005622477, + 0.37411359167288877, + 0.5838063589398508, + 0.774758379773481, + 1.0278067237953041, + 0.4237188751071845, + 0.6388406881889769, + 0.9047202496667041, + 1.1906835220525263, + 0.5684392713417115, + 0.8860466348050552, + 1.1980743702959666, + 1.531357835927315, + 0.5037442759739899, + 1.020297883845433, + 1.5265801233629013, + 2.29829936030438 + ], + "population_cv": 0.4389727385633961 + }, + "bits_per_byte": 1.900553101426062, + "branch_output_rms_by_sublayer": [ + 0.06633151322603226, + 0.1054496169090271, + 0.1283734142780304, + 0.0174406711012125, + 0.05543844401836395, + 0.011477037332952023, + 0.029607228934764862, + 0.03444645181298256, + 0.12885451316833496, + 0.011310109868645668, + 0.03228320553898811, + 0.0048401192761957645, + 0.038674481213092804, + 0.010341182351112366, + 0.03941256180405617, + 0.030718112364411354, + 0.0431947223842144, + 0.02795775979757309, + 0.036978550255298615, + 0.03194068372249603, + 0.036215826869010925, + 0.03605865687131882, + 0.04983788728713989, + 0.042246751487255096, + 0.09164635837078094, + 0.037713672965765, + 0.04682238772511482, + 0.03397047519683838, + 0.037299398332834244, + 0.031024832278490067, + 0.042043957859277725, + 0.03339482843875885, + 0.043044839054346085, + 0.03731323406100273, + 0.036717742681503296, + 0.04126390069723129, + 0.033820051699876785, + 0.0487959161400795, + 0.03861285373568535, + 0.06982608884572983, + 0.04444851353764534, + 0.04609315097332001, + 0.034697387367486954, + 0.05422809347510338, + 0.04958192631602287, + 0.06646102666854858, + 0.04347517341375351, + 0.08457137644290924, + 0.059857066720724106, + 0.05966509133577347, + 0.05223448947072029, + 0.06272348016500473, + 0.04876803979277611, + 0.07299946993589401, + 0.022116826847195625, + 0.09206356853246689, + 0.015298238955438137, + 0.07670208066701889, + 0.020649967715144157, + 0.0958925113081932, + 0.019100233912467957, + 0.09452088922262192, + 0.025575943291187286, + 0.1443253457546234 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00029287198606393655, + 0.00014458277692277753, + 0.00010951406586451765, + 0.00014530258766030837, + 0.00015323544242677847, + 0.00010902005366543604, + 0.00011371422007108556, + 0.00014572956273840553, + 0.00017010165480142522, + 0.00014578936973658711, + 0.00015156606913339834, + 0.0001722085855825187, + 0.00016108287305356382, + 0.0001429493592286421, + 0.00015734835012100933, + 0.0001841843973257514, + 0.00028566044646358623, + 0.0003075779788817468, + 0.00030423368231922797, + 0.00033943250723711063, + 0.0004790776738048971, + 0.0004309071573521118, + 0.00041483321856469247, + 0.0003553184967411939, + 0.00019557126025535645, + 0.00017650553063165768, + 0.00016778394473606075, + 0.00016094542819015603, + 0.00020888375001312183, + 0.00019402861498697634, + 0.00017809767376358575, + 0.00018210391593738592 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0001517463369266557, + "first_to_last_ratio": 0.8292602035179877, + "imbalance_abs_log_ratio": 0.1872212967169728, + "last_quartile_mean": 0.00018299001481428758, + "mean": 0.00021500508232109405, + "normalized": [ + 1.3621630842500463, + 0.6724621360664114, + 0.5093557076988839, + 0.6758100138456717, + 0.7127061405829132, + 0.5070580308544648, + 0.5288908468743164, + 0.6777958974977591, + 0.7911517856465881, + 0.6780740630068529, + 0.704941797175965, + 0.8009512320519848, + 0.7492049551321702, + 0.6648650240516543, + 0.7318354916188508, + 0.8566513653416143, + 1.3286218324689352, + 1.4305614339962531, + 1.4150069339518108, + 1.5787185287564203, + 2.228215578129592, + 2.00417196049619, + 1.9294112217550747, + 1.6526051054484014, + 0.9096122665755657, + 0.820936550551209, + 0.7803719936512382, + 0.7485656918090711, + 0.9715293599486617, + 0.9024373419099418, + 0.8283416923959507, + 0.8469749364595359 + ], + "population_cv": 0.45411891746165434 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.2236010581254959, + "mean_weights": [ + 0.9334083795547485, + 0.06659168004989624 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5658613443374634, + "mean_weights": [ + 0.6809539794921875, + 0.3190460801124573 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5787239670753479, + "mean_weights": [ + 0.7171684503555298, + 0.2828315496444702 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6867446899414062, + "mean_weights": [ + 0.5213377475738525, + 0.47866225242614746 + ], + "sources": 2 + }, + { + "entropy_mean": 0.661613941192627, + "mean_weights": [ + 0.4331015348434448, + 0.5668984651565552 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6896805763244629, + "mean_weights": [ + 0.5011572241783142, + 0.4988427758216858 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5224308967590332, + "mean_weights": [ + 0.7686694860458374, + 0.2313305139541626 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6845217943191528, + "mean_weights": [ + 0.5385724306106567, + 0.46142756938934326 + ], + "sources": 2 + }, + { + "entropy_mean": 0.9951757192611694, + "mean_weights": [ + 0.23048561811447144, + 0.23995241522789001, + 0.5295619368553162 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0840057134628296, + "mean_weights": [ + 0.29915863275527954, + 0.2971389889717102, + 0.40370234847068787 + ], + "sources": 3 + }, + { + "entropy_mean": 0.8066211938858032, + "mean_weights": [ + 0.17861568927764893, + 0.13552500307559967, + 0.6858593225479126 + ], + "sources": 3 + }, + { + "entropy_mean": 1.07478928565979, + "mean_weights": [ + 0.34209051728248596, + 0.2460784614086151, + 0.4118310213088989 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9757064580917358, + "mean_weights": [ + 0.2831856310367584, + 0.1782369613647461, + 0.5385773777961731 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0756715536117554, + "mean_weights": [ + 0.3450500965118408, + 0.24701634049415588, + 0.4079335629940033 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0234577655792236, + "mean_weights": [ + 0.27019333839416504, + 0.23257920145988464, + 0.49722743034362793 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0839393138885498, + "mean_weights": [ + 0.39990490674972534, + 0.2667364478111267, + 0.33335861563682556 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3409738540649414, + "mean_weights": [ + 0.2721319794654846, + 0.1531679928302765, + 0.24795927107334137, + 0.32674074172973633 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3699771165847778, + "mean_weights": [ + 0.3014451861381531, + 0.20016807317733765, + 0.2227502167224884, + 0.2756364941596985 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3592333793640137, + "mean_weights": [ + 0.29968228936195374, + 0.1748049557209015, + 0.24605749547481537, + 0.2794552445411682 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3674365282058716, + "mean_weights": [ + 0.3034948408603668, + 0.2154068648815155, + 0.20510005950927734, + 0.27599823474884033 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3196260929107666, + "mean_weights": [ + 0.3254879117012024, + 0.1787443459033966, + 0.18688705563545227, + 0.30888068675994873 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3730896711349487, + "mean_weights": [ + 0.2819298505783081, + 0.2116852104663849, + 0.2306504249572754, + 0.27573448419570923 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3550171852111816, + "mean_weights": [ + 0.2975759506225586, + 0.20246897637844086, + 0.20265282690525055, + 0.29730224609375 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3808577060699463, + "mean_weights": [ + 0.27357345819473267, + 0.2521625757217407, + 0.23361149430274963, + 0.24065247178077698 + ], + "sources": 4 + }, + { + "entropy_mean": 1.581494688987732, + "mean_weights": [ + 0.17383453249931335, + 0.1573139727115631, + 0.19445936381816864, + 0.2494467794895172, + 0.22494536638259888 + ], + "sources": 5 + }, + { + "entropy_mean": 1.58505380153656, + "mean_weights": [ + 0.20145860314369202, + 0.19476401805877686, + 0.15732140839099884, + 0.16731679439544678, + 0.2791392207145691 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5910776853561401, + "mean_weights": [ + 0.23440338671207428, + 0.1555396020412445, + 0.18933308124542236, + 0.2164338231086731, + 0.20429012179374695 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5312368869781494, + "mean_weights": [ + 0.2017480432987213, + 0.18904206156730652, + 0.10897619277238846, + 0.15747220814228058, + 0.3427615165710449 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5852502584457397, + "mean_weights": [ + 0.2077007293701172, + 0.15047292411327362, + 0.1842176914215088, + 0.2159450352191925, + 0.2416636347770691 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5154869556427002, + "mean_weights": [ + 0.2167026698589325, + 0.22732354700565338, + 0.09435274451971054, + 0.1269843429327011, + 0.3346366882324219 + ], + "sources": 5 + }, + { + "entropy_mean": 1.582499384880066, + "mean_weights": [ + 0.19783833622932434, + 0.16202415525913239, + 0.18370243906974792, + 0.18426953256130219, + 0.27216553688049316 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5695383548736572, + "mean_weights": [ + 0.21839022636413574, + 0.21536628901958466, + 0.12513050436973572, + 0.1589856594800949, + 0.282127320766449 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7493925094604492, + "mean_weights": [ + 0.1377107799053192, + 0.13169628381729126, + 0.15729999542236328, + 0.1622483879327774, + 0.14511629939079285, + 0.2659282684326172 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7149434089660645, + "mean_weights": [ + 0.1561584174633026, + 0.15892019867897034, + 0.08651649951934814, + 0.11649481952190399, + 0.1902061402797699, + 0.2917039096355438 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7114665508270264, + "mean_weights": [ + 0.12863850593566895, + 0.12410931289196014, + 0.14670227468013763, + 0.15985789895057678, + 0.1266813576221466, + 0.3140106797218323 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7381787300109863, + "mean_weights": [ + 0.16339676082134247, + 0.1825663298368454, + 0.10229989886283875, + 0.11394046247005463, + 0.17227163910865784, + 0.2655249238014221 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6968426704406738, + "mean_weights": [ + 0.12940490245819092, + 0.12274232506752014, + 0.14679908752441406, + 0.15200942754745483, + 0.12413809448480606, + 0.3249061405658722 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7454856634140015, + "mean_weights": [ + 0.17135141789913177, + 0.18437139689922333, + 0.1062040627002716, + 0.11575409770011902, + 0.17066100239753723, + 0.25165802240371704 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6915192604064941, + "mean_weights": [ + 0.1340675950050354, + 0.12858635187149048, + 0.14245550334453583, + 0.14664071798324585, + 0.1272827833890915, + 0.32096701860427856 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7428178787231445, + "mean_weights": [ + 0.1819876730442047, + 0.20376534759998322, + 0.08828818798065186, + 0.11637885868549347, + 0.21473100781440735, + 0.1948489248752594 + ], + "sources": 6 + }, + { + "entropy_mean": 1.8245186805725098, + "mean_weights": [ + 0.10190614312887192, + 0.1044919341802597, + 0.10341107845306396, + 0.10877151787281036, + 0.09093066304922104, + 0.20504078269004822, + 0.285447895526886 + ], + "sources": 7 + }, + { + "entropy_mean": 1.848238229751587, + "mean_weights": [ + 0.13188236951828003, + 0.1489405632019043, + 0.07543676346540451, + 0.08802894502878189, + 0.1452489197254181, + 0.12170998007059097, + 0.2887524366378784 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7550859451293945, + "mean_weights": [ + 0.09834608435630798, + 0.09047922492027283, + 0.10089698433876038, + 0.10686840116977692, + 0.08452128618955612, + 0.15423445403575897, + 0.3646535873413086 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8550364971160889, + "mean_weights": [ + 0.14567047357559204, + 0.16652268171310425, + 0.07491736859083176, + 0.08634625375270844, + 0.15591919422149658, + 0.10529892146587372, + 0.265325129032135 + ], + "sources": 7 + }, + { + "entropy_mean": 1.774239182472229, + "mean_weights": [ + 0.110105499625206, + 0.10095183551311493, + 0.1043158695101738, + 0.10433977842330933, + 0.08756626397371292, + 0.1333833634853363, + 0.35933738946914673 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8950581550598145, + "mean_weights": [ + 0.14970523118972778, + 0.16348037123680115, + 0.09331737458705902, + 0.1009049266576767, + 0.15285363793373108, + 0.11107703298330307, + 0.2286614179611206 + ], + "sources": 7 + }, + { + "entropy_mean": 1.782465934753418, + "mean_weights": [ + 0.10454025119543076, + 0.11640097200870514, + 0.10713401436805725, + 0.11336822807788849, + 0.09254641830921173, + 0.11902390420436859, + 0.34698620438575745 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9169561862945557, + "mean_weights": [ + 0.15299510955810547, + 0.16764168441295624, + 0.09622728824615479, + 0.10477937012910843, + 0.16221746802330017, + 0.12927085161209106, + 0.18686822056770325 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0353779792785645, + "mean_weights": [ + 0.12306150794029236, + 0.14678965508937836, + 0.08182619512081146, + 0.08973748236894608, + 0.14154119789600372, + 0.10366955399513245, + 0.12779513001441956, + 0.18557925522327423 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.03346586227417, + "mean_weights": [ + 0.11826661974191666, + 0.1384633481502533, + 0.09414755553007126, + 0.10380144417285919, + 0.12691730260849, + 0.1074572503566742, + 0.13156013190746307, + 0.17938634753227234 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.043306350708008, + "mean_weights": [ + 0.10347998142242432, + 0.12309165298938751, + 0.11448083817958832, + 0.1402483582496643, + 0.11489638686180115, + 0.11068805307149887, + 0.13517531752586365, + 0.15793940424919128 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0447089672088623, + "mean_weights": [ + 0.10259832441806793, + 0.12581557035446167, + 0.11522121727466583, + 0.13812653720378876, + 0.11700352281332016, + 0.12368310987949371, + 0.14790496230125427, + 0.12964677810668945 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0584933757781982, + "mean_weights": [ + 0.08828666061162949, + 0.11706617474555969, + 0.09118273109197617, + 0.09794487059116364, + 0.07609466463327408, + 0.10622192919254303, + 0.18549251556396484, + 0.15259304642677307, + 0.0851174145936966 + ], + "sources": 9 + }, + { + "entropy_mean": 2.104215145111084, + "mean_weights": [ + 0.09777434170246124, + 0.11320681869983673, + 0.09563681483268738, + 0.1292821168899536, + 0.10229292511940002, + 0.11832459270954132, + 0.13642163574695587, + 0.10282324999570847, + 0.10423751175403595 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0337367057800293, + "mean_weights": [ + 0.08985085785388947, + 0.10937829315662384, + 0.08087727427482605, + 0.08739082515239716, + 0.07259232550859451, + 0.06665779650211334, + 0.2110060155391693, + 0.15033027529716492, + 0.131916344165802 + ], + "sources": 9 + }, + { + "entropy_mean": 2.144676923751831, + "mean_weights": [ + 0.09115856885910034, + 0.10643699765205383, + 0.11013875156641006, + 0.1342342048883438, + 0.09498853981494904, + 0.12425043433904648, + 0.14002016186714172, + 0.10460683703422546, + 0.09416551887989044 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0499138832092285, + "mean_weights": [ + 0.08553025126457214, + 0.11924248188734055, + 0.09109945595264435, + 0.1086561530828476, + 0.08561622351408005, + 0.08875375241041183, + 0.19702328741550446, + 0.11887158453464508, + 0.10520680248737335 + ], + "sources": 9 + }, + { + "entropy_mean": 2.161529064178467, + "mean_weights": [ + 0.1020309254527092, + 0.11191397905349731, + 0.1065034568309784, + 0.12505993247032166, + 0.11901188641786575, + 0.10741540044546127, + 0.10972147434949875, + 0.10706284642219543, + 0.11128009110689163 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1078615188598633, + "mean_weights": [ + 0.0905262678861618, + 0.123862624168396, + 0.09877298772335052, + 0.11872135102748871, + 0.08372841775417328, + 0.09360919892787933, + 0.187142014503479, + 0.11672355234622955, + 0.0869135856628418 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05934165418148041, + 0.056099649518728256, + 0.06580447405576706, + 0.08275710791349411, + 0.11824531108140945, + 0.15209494531154633, + 0.1418721228837967, + 0.08564361184835434, + 0.13938024640083313, + 0.1050071269273758, + 0.10961606353521347, + 0.11429986357688904, + 0.10225209593772888, + 0.1062200739979744, + 0.10328132659196854, + 0.10813663899898529, + 0.09859473258256912, + 0.0653785690665245, + 0.07274007052183151, + 0.0697697103023529, + 0.07451662421226501, + 0.0667681023478508, + 0.07608497142791748, + 0.07076340913772583, + 0.08278226852416992, + 0.05711434409022331, + 0.06146874278783798, + 0.0575575977563858, + 0.07372982054948807, + 0.062103815376758575, + 0.0847519114613533, + 0.06907622516155243, + 0.08220508694648743, + 0.050246987491846085, + 0.061270471662282944, + 0.051980238407850266, + 0.06498272716999054, + 0.056721556931734085, + 0.06716923415660858, + 0.0632857084274292, + 0.07637867331504822, + 0.05010642111301422, + 0.0557682067155838, + 0.05070976912975311, + 0.061345092952251434, + 0.0584816075861454, + 0.06197848170995712, + 0.06649797409772873, + 0.06426215916872025, + 0.05134814977645874, + 0.054289888590574265, + 0.05254536494612694, + 0.05770144984126091, + 0.054670173674821854, + 0.06198704242706299, + 0.057158827781677246, + 0.06496675312519073, + 0.06891956925392151, + 0.06152588874101639, + 0.07430952042341232, + 0.06446245312690735, + 0.07342272251844406, + 0.07153722643852234, + 0.07330406457185745 + ], + "loss_nats": 1.3173630237579346, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.8983385562896729, + "mean_weights": [ + 0.10611078143119812, + 0.08491367101669312, + 0.06894227862358093, + 0.055786699056625366, + 0.06929998099803925, + 0.04338888078927994, + 0.10485145449638367, + 0.15557360649108887, + 0.31113263964653015 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.06633151322603226, + 0.12735538184642792, + 0.21601833403110504, + 0.22348231077194214, + 0.2564253509044647, + 0.26064634323120117, + 0.2635253071784973, + 0.277464359998703, + 0.12885451316833496, + 0.13131454586982727, + 0.14671505987644196, + 0.1476469337940216, + 0.16035981476306915, + 0.16177311539649963, + 0.16735433042049408, + 0.17404833436012268, + 0.0431947223842144, + 0.052131909877061844, + 0.06744951754808426, + 0.08374633640050888, + 0.09851458668708801, + 0.11783560365438461, + 0.14188311994075775, + 0.16801035404205322, + 0.09164635837078094, + 0.10621181130409241, + 0.1441553682088852, + 0.15759189426898956, + 0.17191539704799652, + 0.18511100113391876, + 0.20303158462047577, + 0.21714943647384644, + 0.043044839054346085, + 0.0585859939455986, + 0.0728224590420723, + 0.09142377227544785, + 0.09907020628452301, + 0.1213267594575882, + 0.13032910227775574, + 0.16095399856567383, + 0.04444851353764534, + 0.06635415554046631, + 0.07589157670736313, + 0.10004211962223053, + 0.11477663367986679, + 0.14167872071266174, + 0.150556281208992, + 0.18646042048931122, + 0.059857066720724106, + 0.08901729434728622, + 0.10840647667646408, + 0.1387544423341751, + 0.15091489255428314, + 0.18761782348155975, + 0.18941630423069, + 0.23980984091758728, + 0.015298238955438137, + 0.07888609170913696, + 0.08265326172113419, + 0.15977811813354492, + 0.1628996580839157, + 0.2390616536140442, + 0.24262696504592896, + 0.3599124848842621 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00011286069639027119, + 0.00011551808711374179, + 0.0001130274249590002, + 0.00010967965499730781, + 0.00012856301327701658, + 0.0001273026573471725, + 0.00011791360884672031, + 0.00011041056859539822, + 0.0001144223497249186, + 0.00010640535765560344, + 9.945721103576943e-05, + 9.377022797707468e-05, + 8.994352538138628e-05, + 8.844584954204038e-05, + 9.083787153940648e-05, + 9.304411651100963e-05, + 0.00014625214680563658, + 0.0001291196676902473, + 0.00011372811422916129, + 0.0001000549818854779, + 0.00016139348736032844, + 0.00013271211355458945, + 0.00010506954276934266, + 8.063438144745305e-05, + 5.8612142311176285e-05, + 5.450717799249105e-05, + 5.115715612191707e-05, + 4.9292317271465436e-05, + 6.816338282078505e-05, + 5.253752169664949e-05, + 4.7662029828643426e-05, + 4.528104545897804e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00011690946394082857, + "first_to_last_ratio": 2.189250344411854, + "imbalance_abs_log_ratio": 0.7835591767397477, + "last_quartile_mean": 5.340159668776323e-05, + "mean": 9.711810719181813e-05, + "normalized": [ + 1.1620973642676113, + 1.1894598283879425, + 1.1638141251636995, + 1.129343004808355, + 1.323780054970507, + 1.3108024963433114, + 1.214125895326903, + 1.1368690328501372, + 1.1781773042479384, + 1.0956284129945206, + 1.0240851465456522, + 0.9655277546942812, + 0.9261251890313171, + 0.910704008752465, + 0.9353340398201178, + 0.9580511729624018, + 1.505920482127743, + 1.3295117813120352, + 1.1710289411277004, + 1.030240238186059, + 1.661826944810198, + 1.3665022660755686, + 1.0818738730339919, + 0.8302713446442275, + 0.603514051148169, + 0.5612462965823031, + 0.5267519889043605, + 0.5075502261808718, + 0.701860701281538, + 0.5409652557672129, + 0.49076357856219416, + 0.46624719908866613 + ], + "population_cv": 0.31106279622053273 + }, + "activation_output_rms_by_block": [ + 0.112685427069664, + 0.21922698616981506, + 0.25489342212677, + 0.2656901776790619, + 0.13385510444641113, + 0.14834310114383698, + 0.15773193538188934, + 0.17191331088542938, + 0.04261620715260506, + 0.06956788152456284, + 0.09812623262405396, + 0.1476527601480484, + 0.0980270579457283, + 0.14125338196754456, + 0.16483257710933685, + 0.19266656041145325, + 0.052173662930727005, + 0.08166498690843582, + 0.10914134979248047, + 0.14517217874526978, + 0.05995834618806839, + 0.09112747758626938, + 0.12867455184459686, + 0.1700005978345871, + 0.084324412047863, + 0.1263056993484497, + 0.16806621849536896, + 0.20813708007335663, + 0.06699028611183167, + 0.13289068639278412, + 0.19392795860767365, + 0.2826451063156128 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.18304243311285973, + "first_to_last_ratio": 1.1591498577183288, + "imbalance_abs_log_ratio": 0.1476868551522012, + "last_quartile_mean": 0.15791093092411757, + "mean": 0.14125883509404957, + "normalized": [ + 0.7977230379582169, + 1.5519523860143305, + 1.804442334223293, + 1.8808747608754273, + 0.9475874861723936, + 1.0501509590183915, + 1.116616424571759, + 1.2170092636752257, + 0.301688790823111, + 0.49248517077352943, + 0.6946555417841432, + 1.045263894819335, + 0.6939534640821742, + 0.9999613962092964, + 1.1668833103401424, + 1.3639257345084053, + 0.36934796252560054, + 0.5781230381382063, + 0.7726337946919539, + 1.0277033549697243, + 0.4244573172938058, + 0.6451099325971156, + 0.9109133015214649, + 1.2034687792901688, + 0.5969496491438582, + 0.8941437133072482, + 1.1897749148467147, + 1.4734446870865088, + 0.47423784903245025, + 0.9407601747834464, + 1.3728554286785473, + 2.0009021462440124 + ], + "population_cv": 0.4300551761457633 + }, + "bits_per_byte": 1.8562386757435438, + "branch_output_rms_by_sublayer": [ + 0.06376994401216507, + 0.0893719270825386, + 0.13826815783977509, + 0.012125417590141296, + 0.04956018924713135, + 0.011939814314246178, + 0.023104721680283546, + 0.025088787078857422, + 0.13105179369449615, + 0.010653985664248466, + 0.02707556262612343, + 0.0037635224871337414, + 0.0279815886169672, + 0.008720523677766323, + 0.034021858125925064, + 0.02992928773164749, + 0.03591933101415634, + 0.022252332419157028, + 0.030579224228858948, + 0.02659093402326107, + 0.03153296932578087, + 0.030294356867671013, + 0.048588357865810394, + 0.039768703281879425, + 0.08358240127563477, + 0.03535241633653641, + 0.03906446695327759, + 0.03152184188365936, + 0.030991801992058754, + 0.02869572676718235, + 0.03578810393810272, + 0.029983263462781906, + 0.037315741181373596, + 0.03435273841023445, + 0.031990379095077515, + 0.03991268202662468, + 0.030398933216929436, + 0.0480409674346447, + 0.033455584198236465, + 0.06781420856714249, + 0.03767154738306999, + 0.044399797916412354, + 0.028634775429964066, + 0.053760360926389694, + 0.0421283133327961, + 0.0663343071937561, + 0.03824872151017189, + 0.08417567610740662, + 0.05500945448875427, + 0.05761333554983139, + 0.0438532792031765, + 0.061463043093681335, + 0.042473942041397095, + 0.06964308023452759, + 0.022093990817666054, + 0.08274602890014648, + 0.013804801739752293, + 0.0652693659067154, + 0.017453746870160103, + 0.08362726122140884, + 0.01626753807067871, + 0.07811809331178665, + 0.02098197303712368, + 0.11215418577194214 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0004229317372874992, + 0.0002445031849129385, + 0.00017799962227732323, + 0.00019829918059481726, + 0.000249425045218403, + 0.00013352663223756338, + 0.00016264802987594294, + 0.0002218576110695471, + 0.00027756889336923586, + 0.00025509960948202383, + 0.0002725558147937901, + 0.0003136651413935982, + 0.00028505768030899373, + 0.0002561422651919067, + 0.00027478080377229196, + 0.00034913887680594396, + 0.0005446800846079912, + 0.000571281042845494, + 0.0005723843821277269, + 0.0006201231494136244, + 0.0008383813977186006, + 0.0007670347934454351, + 0.0007400323687890561, + 0.0006144381280003681, + 0.0003742135137473206, + 0.0003188244169136918, + 0.0002962178942676787, + 0.0002776650851383431, + 0.0003737948877851981, + 0.000327165464961136, + 0.0002805578606957678, + 0.0002776428687025192 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00022639888043425433, + "first_to_last_ratio": 0.7169961422702223, + "imbalance_abs_log_ratio": 0.3326848187732499, + "last_quartile_mean": 0.0003157602490264569, + "mean": 0.00037155117086724285, + "normalized": [ + 1.138286649185705, + 0.6580605959126442, + 0.4790716225220117, + 0.533706245984811, + 0.6713073858338716, + 0.3593761578678301, + 0.4377540501253269, + 0.5971118609361561, + 0.747054282513384, + 0.6865800177310496, + 0.7335619859779038, + 0.8442044218605689, + 0.7672097483736535, + 0.6893862414537448, + 0.7395503643035822, + 0.9396791187362268, + 1.465962503459875, + 1.5375568364165257, + 1.5405263850783095, + 1.66901142571072, + 2.256435892159141, + 2.0644122629329584, + 1.9917374154998249, + 1.6537106492389713, + 1.00716548106648, + 0.8580901956775409, + 0.7972465638481837, + 0.7473131748992773, + 1.0060387830637652, + 0.8805394535495459, + 0.7550988469257511, + 0.7472533811546578 + ], + "population_cv": 0.4915845509569984 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.20489901304244995, + "mean_weights": [ + 0.9417868256568909, + 0.058213189244270325 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5417418479919434, + "mean_weights": [ + 0.7154836654663086, + 0.284516304731369 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6360999941825867, + "mean_weights": [ + 0.6399655342102051, + 0.36003440618515015 + ], + "sources": 2 + }, + { + "entropy_mean": 0.689984142780304, + "mean_weights": [ + 0.4929400384426117, + 0.5070599317550659 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6258687973022461, + "mean_weights": [ + 0.3577972948551178, + 0.6422027349472046 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6889487504959106, + "mean_weights": [ + 0.4809175431728363, + 0.5190825462341309 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5806001424789429, + "mean_weights": [ + 0.7126924991607666, + 0.2873075604438782 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6853691339492798, + "mean_weights": [ + 0.5331022143363953, + 0.46689778566360474 + ], + "sources": 2 + }, + { + "entropy_mean": 1.004859209060669, + "mean_weights": [ + 0.22505608201026917, + 0.2539703845977783, + 0.5209735631942749 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0778453350067139, + "mean_weights": [ + 0.284685343503952, + 0.28999078273773193, + 0.42532384395599365 + ], + "sources": 3 + }, + { + "entropy_mean": 0.8005489110946655, + "mean_weights": [ + 0.16932852566242218, + 0.13586211204528809, + 0.6948093771934509 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0729284286499023, + "mean_weights": [ + 0.33095037937164307, + 0.24655158817768097, + 0.42249804735183716 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9625074863433838, + "mean_weights": [ + 0.26453888416290283, + 0.17643973231315613, + 0.5590213537216187 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0746655464172363, + "mean_weights": [ + 0.3280792236328125, + 0.252571702003479, + 0.4193490743637085 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0206949710845947, + "mean_weights": [ + 0.2595077157020569, + 0.23315896093845367, + 0.5073332786560059 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0863780975341797, + "mean_weights": [ + 0.379249632358551, + 0.26598525047302246, + 0.3547651171684265 + ], + "sources": 3 + }, + { + "entropy_mean": 1.341073989868164, + "mean_weights": [ + 0.2542946934700012, + 0.1510404348373413, + 0.2655471861362457, + 0.32911768555641174 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3711183071136475, + "mean_weights": [ + 0.2912205159664154, + 0.20097798109054565, + 0.2207832932472229, + 0.28701820969581604 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3567297458648682, + "mean_weights": [ + 0.27403461933135986, + 0.16812831163406372, + 0.2507767975330353, + 0.3070603013038635 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3679343461990356, + "mean_weights": [ + 0.29195448756217957, + 0.21413873136043549, + 0.20624984800815582, + 0.28765690326690674 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3197410106658936, + "mean_weights": [ + 0.29432952404022217, + 0.1726282835006714, + 0.1945701241493225, + 0.33847206830978394 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3736048936843872, + "mean_weights": [ + 0.27173638343811035, + 0.20676466822624207, + 0.23934081196784973, + 0.28215816617012024 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3521652221679688, + "mean_weights": [ + 0.27595657110214233, + 0.1993924379348755, + 0.20179325342178345, + 0.32285773754119873 + ], + "sources": 4 + }, + { + "entropy_mean": 1.381324291229248, + "mean_weights": [ + 0.2670099437236786, + 0.25774136185646057, + 0.23019102215766907, + 0.24505767226219177 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5744428634643555, + "mean_weights": [ + 0.16565194725990295, + 0.14992284774780273, + 0.1911824345588684, + 0.2672228217124939, + 0.22601991891860962 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5826877355575562, + "mean_weights": [ + 0.19575577974319458, + 0.1891566962003708, + 0.15589091181755066, + 0.17277422547340393, + 0.28642240166664124 + ], + "sources": 5 + }, + { + "entropy_mean": 1.587158441543579, + "mean_weights": [ + 0.2228369265794754, + 0.14338329434394836, + 0.1864481419324875, + 0.23181000351905823, + 0.21552161872386932 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5396997928619385, + "mean_weights": [ + 0.19954928755760193, + 0.19654099643230438, + 0.11069682240486145, + 0.15833479166030884, + 0.3348781168460846 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5828174352645874, + "mean_weights": [ + 0.202924907207489, + 0.14161095023155212, + 0.19405998289585114, + 0.21129687130451202, + 0.2501072883605957 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5043025016784668, + "mean_weights": [ + 0.20454180240631104, + 0.23216864466667175, + 0.08493556827306747, + 0.13164377212524414, + 0.346710205078125 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5767673254013062, + "mean_weights": [ + 0.19306333363056183, + 0.15463149547576904, + 0.18734106421470642, + 0.17873430252075195, + 0.28622978925704956 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5668692588806152, + "mean_weights": [ + 0.21234489977359772, + 0.21990114450454712, + 0.12000303715467453, + 0.16270337998867035, + 0.2850475311279297 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7344534397125244, + "mean_weights": [ + 0.13368560373783112, + 0.12121029198169708, + 0.1532086282968521, + 0.1579805612564087, + 0.145574152469635, + 0.2883407473564148 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7098653316497803, + "mean_weights": [ + 0.15194454789161682, + 0.16084054112434387, + 0.08314592391252518, + 0.11595913767814636, + 0.191949263215065, + 0.29616057872772217 + ], + "sources": 6 + }, + { + "entropy_mean": 1.691382884979248, + "mean_weights": [ + 0.12500253319740295, + 0.1146039366722107, + 0.14198225736618042, + 0.15820369124412537, + 0.12448806315660477, + 0.3357195258140564 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7358124256134033, + "mean_weights": [ + 0.16035935282707214, + 0.17951011657714844, + 0.09638158977031708, + 0.11847307533025742, + 0.178960919380188, + 0.26631492376327515 + ], + "sources": 6 + }, + { + "entropy_mean": 1.680541753768921, + "mean_weights": [ + 0.12679925560951233, + 0.11613219231367111, + 0.14216846227645874, + 0.14908844232559204, + 0.12225303053855896, + 0.3435586094856262 + ], + "sources": 6 + }, + { + "entropy_mean": 1.749640941619873, + "mean_weights": [ + 0.1672825813293457, + 0.18405666947364807, + 0.103073850274086, + 0.12309282273054123, + 0.17773863673210144, + 0.24475543200969696 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6834077835083008, + "mean_weights": [ + 0.13208967447280884, + 0.12381511926651001, + 0.14079470932483673, + 0.14811252057552338, + 0.12349426746368408, + 0.33169370889663696 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7408825159072876, + "mean_weights": [ + 0.17686429619789124, + 0.20980793237686157, + 0.08485338091850281, + 0.12087466567754745, + 0.21813850104808807, + 0.18946118652820587 + ], + "sources": 6 + }, + { + "entropy_mean": 1.8164830207824707, + "mean_weights": [ + 0.09976128488779068, + 0.09634461253881454, + 0.10337363183498383, + 0.11114586889743805, + 0.09079629927873611, + 0.20262205600738525, + 0.29595625400543213 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8437135219573975, + "mean_weights": [ + 0.12964163720607758, + 0.1510496884584427, + 0.0726064145565033, + 0.09053978323936462, + 0.14789986610412598, + 0.1161271184682846, + 0.29213547706604004 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7404718399047852, + "mean_weights": [ + 0.09536662697792053, + 0.08780238032341003, + 0.10210233926773071, + 0.10861509293317795, + 0.08034101873636246, + 0.15046317875385284, + 0.3753093481063843 + ], + "sources": 7 + }, + { + "entropy_mean": 1.8459525108337402, + "mean_weights": [ + 0.14058244228363037, + 0.1685217022895813, + 0.0693541169166565, + 0.0880737155675888, + 0.16265514492988586, + 0.10036318004131317, + 0.270449697971344 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7632360458374023, + "mean_weights": [ + 0.10711254179477692, + 0.0956469476222992, + 0.10506488382816315, + 0.10788024961948395, + 0.08400966972112656, + 0.13367655873298645, + 0.3666091561317444 + ], + "sources": 7 + }, + { + "entropy_mean": 1.890692114830017, + "mean_weights": [ + 0.14600783586502075, + 0.16169379651546478, + 0.09178381413221359, + 0.10166816413402557, + 0.15640902519226074, + 0.10768906027078629, + 0.23474833369255066 + ], + "sources": 7 + }, + { + "entropy_mean": 1.7769545316696167, + "mean_weights": [ + 0.10480362176895142, + 0.11373230814933777, + 0.10652139782905579, + 0.11335475742816925, + 0.09359428286552429, + 0.11423048377037048, + 0.3537631630897522 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9106683731079102, + "mean_weights": [ + 0.15153241157531738, + 0.17026472091674805, + 0.09009622782468796, + 0.10499827563762665, + 0.16738030314445496, + 0.12463030219078064, + 0.19109776616096497 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0251288414001465, + "mean_weights": [ + 0.12309062480926514, + 0.15000396966934204, + 0.07806398719549179, + 0.08456641435623169, + 0.14079469442367554, + 0.09767232835292816, + 0.12716160714626312, + 0.19864636659622192 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0246033668518066, + "mean_weights": [ + 0.11882127076387405, + 0.1422310173511505, + 0.09053680300712585, + 0.09606730937957764, + 0.12964919209480286, + 0.10240590572357178, + 0.12990963459014893, + 0.19037887454032898 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.043313503265381, + "mean_weights": [ + 0.10487936437129974, + 0.13700152933597565, + 0.12033849209547043, + 0.15013068914413452, + 0.11825264990329742, + 0.11039704084396362, + 0.11670267581939697, + 0.14229753613471985 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.048006057739258, + "mean_weights": [ + 0.10414238274097443, + 0.13406702876091003, + 0.10987317562103271, + 0.13409963250160217, + 0.1260664165019989, + 0.12239565700292587, + 0.14089587330818176, + 0.12845981121063232 + ], + "sources": 8 + }, + { + "entropy_mean": 2.04211688041687, + "mean_weights": [ + 0.08480311930179596, + 0.10890309512615204, + 0.0835762545466423, + 0.09109923243522644, + 0.07195177674293518, + 0.09794797003269196, + 0.1738617718219757, + 0.16131983697414398, + 0.12653693556785583 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1147680282592773, + "mean_weights": [ + 0.09766561537981033, + 0.11648178100585938, + 0.09464797377586365, + 0.12388858199119568, + 0.11199226975440979, + 0.11646085977554321, + 0.12424062192440033, + 0.1007767915725708, + 0.11384551227092743 + ], + "sources": 9 + }, + { + "entropy_mean": 1.994852066040039, + "mean_weights": [ + 0.08600901067256927, + 0.0992315337061882, + 0.07518987357616425, + 0.07912145555019379, + 0.06638798117637634, + 0.05950283631682396, + 0.20053142309188843, + 0.16675196588039398, + 0.16727390885353088 + ], + "sources": 9 + }, + { + "entropy_mean": 2.146540403366089, + "mean_weights": [ + 0.09105421602725983, + 0.10908372700214386, + 0.10561807453632355, + 0.1381063610315323, + 0.09987117350101471, + 0.11679299175739288, + 0.1337788701057434, + 0.10354103893041611, + 0.10215355455875397 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0234761238098145, + "mean_weights": [ + 0.08250690996646881, + 0.11078023165464401, + 0.08806271851062775, + 0.09900778532028198, + 0.08088505268096924, + 0.08615157008171082, + 0.19158735871315002, + 0.1270119696855545, + 0.13400638103485107 + ], + "sources": 9 + }, + { + "entropy_mean": 2.159276247024536, + "mean_weights": [ + 0.10330037772655487, + 0.11992266774177551, + 0.09903984516859055, + 0.11558051407337189, + 0.12950149178504944, + 0.10201072692871094, + 0.10745715349912643, + 0.10560345649719238, + 0.11758378148078918 + ], + "sources": 9 + }, + { + "entropy_mean": 2.101466178894043, + "mean_weights": [ + 0.08965107053518295, + 0.11943937093019485, + 0.09308905899524689, + 0.10822755098342896, + 0.08204977214336395, + 0.08472812175750732, + 0.1853669136762619, + 0.1248210072517395, + 0.11262711882591248 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06226532533764839, + 0.059186093509197235, + 0.06218352168798447, + 0.09392116963863373, + 0.12165758013725281, + 0.16486792266368866, + 0.142787903547287, + 0.09440050274133682, + 0.13547813892364502, + 0.10104379057884216, + 0.10270584374666214, + 0.11180124431848526, + 0.09696419537067413, + 0.10233785212039948, + 0.0983380600810051, + 0.10237180441617966, + 0.09253966063261032, + 0.061393674463033676, + 0.06685728579759598, + 0.06312520802021027, + 0.0678187683224678, + 0.05965995043516159, + 0.06806015968322754, + 0.06168249249458313, + 0.07398500293493271, + 0.04790724813938141, + 0.05253538861870766, + 0.047994233667850494, + 0.0645052120089531, + 0.05162183567881584, + 0.0781417042016983, + 0.058539148420095444, + 0.07275772094726562, + 0.04042136296629906, + 0.0532713308930397, + 0.0429304875433445, + 0.05665335804224014, + 0.04794013127684593, + 0.05866985023021698, + 0.05434121564030647, + 0.0680156797170639, + 0.04235260188579559, + 0.04911918565630913, + 0.04336932674050331, + 0.0551820732653141, + 0.051026612520217896, + 0.054394111037254333, + 0.05851686745882034, + 0.05690229684114456, + 0.04360710829496384, + 0.04690462723374367, + 0.044645342975854874, + 0.05018632113933563, + 0.04650896042585373, + 0.05238434672355652, + 0.048579052090644836, + 0.055429521948099136, + 0.05886467918753624, + 0.051186539232730865, + 0.06579510122537613, + 0.054073598235845566, + 0.06446471810340881, + 0.059442270547151566, + 0.06403441727161407 + ], + "loss_nats": 1.2866466045379639, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.7767908573150635, + "mean_weights": [ + 0.09728685021400452, + 0.07046866416931152, + 0.06362532824277878, + 0.04932419955730438, + 0.06091274321079254, + 0.03856506198644638, + 0.08886311948299408, + 0.1408468782901764, + 0.390107125043869 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.06376994401216507, + 0.112685427069664, + 0.21483486890792847, + 0.21922698616981506, + 0.2503068745136261, + 0.25489342212677, + 0.2573912739753723, + 0.2656901776790619, + 0.13105179369449615, + 0.13385510444641113, + 0.1476495862007141, + 0.14834310114383698, + 0.15647916495800018, + 0.15773193538188934, + 0.16485002636909485, + 0.17191331088542938, + 0.03591933101415634, + 0.04261620715260506, + 0.056517306715250015, + 0.06956788152456284, + 0.08238552510738373, + 0.09812623262405396, + 0.12359381467103958, + 0.1476527601480484, + 0.08358240127563477, + 0.0980270579457283, + 0.12861736118793488, + 0.14125338196754456, + 0.15270014107227325, + 0.16483257710933685, + 0.18010228872299194, + 0.19266656041145325, + 0.037315741181373596, + 0.052173662930727005, + 0.06366626918315887, + 0.08166498690843582, + 0.08791448920965195, + 0.10914134979248047, + 0.1161920577287674, + 0.14517217874526978, + 0.03767154738306999, + 0.05995834618806839, + 0.06697625666856766, + 0.09112747758626938, + 0.10235358774662018, + 0.12867455184459686, + 0.1355234980583191, + 0.1700005978345871, + 0.05500945448875427, + 0.084324412047863, + 0.09822092950344086, + 0.1263056993484497, + 0.13654646277427673, + 0.16806621849536896, + 0.16952820122241974, + 0.20813708007335663, + 0.013804801739752293, + 0.06699028611183167, + 0.06987427920103073, + 0.13289068639278412, + 0.1348741054534912, + 0.19392795860767365, + 0.19459034502506256, + 0.2826451063156128 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 8.061194200882085, + "cross_entropy_nats": 5.587594032287598, + "step": 0 + }, + { + "bits_per_byte": 5.1922921818337, + "cross_entropy_nats": 3.599022686481476, + "step": 100 + }, + { + "bits_per_byte": 3.0889617320292873, + "cross_entropy_nats": 2.1411051154136658, + "step": 500 + }, + { + "bits_per_byte": 1.95113741261976, + "cross_entropy_nats": 1.3524253964424133, + "step": 2000 + }, + { + "bits_per_byte": 1.7775269889621976, + "cross_entropy_nats": 1.2320878207683563, + "step": 4000 + }, + { + "bits_per_byte": 1.6977998306188322, + "cross_entropy_nats": 1.1768251657485962, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 49, + 51, + 53, + 55 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.49.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "49": true, + "51": true, + "53": true, + "55": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.40.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.41.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.42.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.43.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.44.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.45.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.46.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.47.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.48.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.50.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.52.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.54.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_group_7_mlp_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "ed432ee3ffbd51f4f70ab763b1a839e4a3f1de8caab9c9ee2684b2af81a4afa7", + "final_model_state": "a741586f267e32b9cfa434dee36831800593021202e910e951dbe499f23ee75e", + "final_optimizer_state": "827d5081ccb5bd654e8f9fa3a8d779c128b208d2938c708b1721d8856ba0aa00", + "final_public_parameters": "011c58919183929fba1ef19ac37b186a75eb06ef61911783727faf6adfecbcf2", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "3f93c11463780b386fff00c53b0424f687e1136b1b80f7b626f058adef96c764" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "3eff5c07fa0cfbd5ceed5922cbd09525ad514f801cc0a440c03fd2fa15df4c84", + "1": "e83e4001c5811b19c69a54c7ea7a802af996a34aa57072cde43a76bc8369e2d0", + "7999": "d04c4ca879f0209183a48ac54a0c4ffe3a0f62a78888479d4641f6c304426442" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "85c11a0a1b2cb8760f8efab4d267456191dfd001736bd427cb9a78828f3de4e8", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073003, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 210.63127594077326, + "measured_steps": 7980, + "median_ms": 217.71888200601097, + "p95_ms": 220.9140715538524, + "peak_allocated_bytes": 12208699392, + "peak_reserved_bytes": 12712935424, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 8.082576459898554, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.602415084838867, + "step": 1, + "unclipped_grad_norm": 19.083236694335938 + }, + { + "bits_per_byte": 7.337145176957849, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.085721492767334, + "step": 10, + "unclipped_grad_norm": 11.7355318069458 + }, + { + "bits_per_byte": 6.794624528949912, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.709674835205078, + "step": 20, + "unclipped_grad_norm": 3.370473623275757 + }, + { + "bits_per_byte": 6.6697238485873065, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.623100280761719, + "step": 30, + "unclipped_grad_norm": 3.037773847579956 + }, + { + "bits_per_byte": 6.488635060326613, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.497579097747803, + "step": 40, + "unclipped_grad_norm": 3.071342945098877 + }, + { + "bits_per_byte": 6.170626068199815, + "learning_rate": 3.75e-05, + "loss_nats": 4.277152061462402, + "step": 50, + "unclipped_grad_norm": 2.8248815536499023 + }, + { + "bits_per_byte": 5.946016726172987, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.121464729309082, + "step": 60, + "unclipped_grad_norm": 2.602710247039795 + }, + { + "bits_per_byte": 5.764103109118756, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.9953718185424805, + "step": 70, + "unclipped_grad_norm": 2.5199975967407227 + }, + { + "bits_per_byte": 5.595939848966308, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.878809928894043, + "step": 80, + "unclipped_grad_norm": 2.5971291065216064 + }, + { + "bits_per_byte": 5.405385823532731, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.74672794342041, + "step": 90, + "unclipped_grad_norm": 2.3173539638519287 + }, + { + "bits_per_byte": 5.178403464889297, + "learning_rate": 7.5e-05, + "loss_nats": 3.589395761489868, + "step": 100, + "unclipped_grad_norm": 2.216301918029785 + }, + { + "bits_per_byte": 4.995956357652091, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.46293306350708, + "step": 110, + "unclipped_grad_norm": 2.0339794158935547 + }, + { + "bits_per_byte": 4.799490256612453, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.3267531394958496, + "step": 120, + "unclipped_grad_norm": 1.8057299852371216 + }, + { + "bits_per_byte": 4.634010613744068, + "learning_rate": 9.75e-05, + "loss_nats": 3.2120513916015625, + "step": 130, + "unclipped_grad_norm": 1.5956865549087524 + }, + { + "bits_per_byte": 4.477776070255465, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.103757858276367, + "step": 140, + "unclipped_grad_norm": 1.3932298421859741 + }, + { + "bits_per_byte": 4.276258494794685, + "learning_rate": 0.0001125, + "loss_nats": 2.964076519012451, + "step": 150, + "unclipped_grad_norm": 1.2066383361816406 + }, + { + "bits_per_byte": 4.129976587311512, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.8626816272735596, + "step": 160, + "unclipped_grad_norm": 1.0521245002746582 + }, + { + "bits_per_byte": 4.022640495081525, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.7882819175720215, + "step": 170, + "unclipped_grad_norm": 0.9153721928596497 + }, + { + "bits_per_byte": 3.9203906177681542, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.717407703399658, + "step": 180, + "unclipped_grad_norm": 0.8919755816459656 + }, + { + "bits_per_byte": 3.8313999149057025, + "learning_rate": 0.0001425, + "loss_nats": 2.655724048614502, + "step": 190, + "unclipped_grad_norm": 0.80049729347229 + }, + { + "bits_per_byte": 3.7637020400812733, + "learning_rate": 0.00015, + "loss_nats": 2.608799457550049, + "step": 200, + "unclipped_grad_norm": 0.7152890563011169 + }, + { + "bits_per_byte": 3.6861571265990434, + "learning_rate": 0.0001575, + "loss_nats": 2.555049419403076, + "step": 210, + "unclipped_grad_norm": 1.0002022981643677 + }, + { + "bits_per_byte": 3.651245680369798, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.530850648880005, + "step": 220, + "unclipped_grad_norm": 0.7646846175193787 + }, + { + "bits_per_byte": 3.6125581830582094, + "learning_rate": 0.0001725, + "loss_nats": 2.5040345191955566, + "step": 230, + "unclipped_grad_norm": 0.9416015148162842 + }, + { + "bits_per_byte": 3.5828292620300646, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.4834280014038086, + "step": 240, + "unclipped_grad_norm": 0.7748287916183472 + }, + { + "bits_per_byte": 3.510992108764669, + "learning_rate": 0.0001875, + "loss_nats": 2.4336342811584473, + "step": 250, + "unclipped_grad_norm": 0.8496595621109009 + }, + { + "bits_per_byte": 3.5180619715763375, + "learning_rate": 0.000195, + "loss_nats": 2.438534736633301, + "step": 260, + "unclipped_grad_norm": 1.2136850357055664 + }, + { + "bits_per_byte": 3.510447267726736, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.4332566261291504, + "step": 270, + "unclipped_grad_norm": 1.0394660234451294 + }, + { + "bits_per_byte": 3.4528010909146345, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.3932993412017822, + "step": 280, + "unclipped_grad_norm": 1.0233465433120728 + }, + { + "bits_per_byte": 3.4419565528818294, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.385782480239868, + "step": 290, + "unclipped_grad_norm": 1.2728065252304077 + }, + { + "bits_per_byte": 3.4209464643218253, + "learning_rate": 0.000225, + "loss_nats": 2.3712193965911865, + "step": 300, + "unclipped_grad_norm": 0.8485052585601807 + }, + { + "bits_per_byte": 3.3318126718938554, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.309436559677124, + "step": 310, + "unclipped_grad_norm": 1.0929526090621948 + }, + { + "bits_per_byte": 3.3946943445635225, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.353022813796997, + "step": 320, + "unclipped_grad_norm": 0.969825804233551 + }, + { + "bits_per_byte": 3.3985016964883235, + "learning_rate": 0.0002475, + "loss_nats": 2.3556618690490723, + "step": 330, + "unclipped_grad_norm": 0.9803057909011841 + }, + { + "bits_per_byte": 3.3304185805259614, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.3084702491760254, + "step": 340, + "unclipped_grad_norm": 1.337324857711792 + }, + { + "bits_per_byte": 3.3031589863989046, + "learning_rate": 0.0002625, + "loss_nats": 2.2895753383636475, + "step": 350, + "unclipped_grad_norm": 1.0087926387786865 + }, + { + "bits_per_byte": 3.3106667170396697, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.2947793006896973, + "step": 360, + "unclipped_grad_norm": 1.0914205312728882 + }, + { + "bits_per_byte": 3.3107396376836356, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.294829845428467, + "step": 370, + "unclipped_grad_norm": 1.1424835920333862 + }, + { + "bits_per_byte": 3.208417871622204, + "learning_rate": 0.000285, + "loss_nats": 2.2239058017730713, + "step": 380, + "unclipped_grad_norm": 1.0868791341781616 + }, + { + "bits_per_byte": 3.2866386769223253, + "learning_rate": 0.0002925, + "loss_nats": 2.2781243324279785, + "step": 390, + "unclipped_grad_norm": 1.0824024677276611 + }, + { + "bits_per_byte": 3.265791627915339, + "learning_rate": 0.0003, + "loss_nats": 2.263674259185791, + "step": 400, + "unclipped_grad_norm": 0.9879770278930664 + }, + { + "bits_per_byte": 3.283704308933304, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.276090383529663, + "step": 410, + "unclipped_grad_norm": 1.0128034353256226 + }, + { + "bits_per_byte": 3.218989645170741, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.231233596801758, + "step": 420, + "unclipped_grad_norm": 1.0376091003417969 + }, + { + "bits_per_byte": 3.2238853033102015, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.2346270084381104, + "step": 430, + "unclipped_grad_norm": 0.8619301915168762 + }, + { + "bits_per_byte": 3.169007359246388, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.1965885162353516, + "step": 440, + "unclipped_grad_norm": 0.897327721118927 + }, + { + "bits_per_byte": 3.2470857629115946, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.2507083415985107, + "step": 450, + "unclipped_grad_norm": 1.154920220375061 + }, + { + "bits_per_byte": 3.168560204354145, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.1962785720825195, + "step": 460, + "unclipped_grad_norm": 1.0526434183120728 + }, + { + "bits_per_byte": 3.141423405839804, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.177468776702881, + "step": 470, + "unclipped_grad_norm": 1.176223874092102 + }, + { + "bits_per_byte": 3.1264499083250845, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.1670899391174316, + "step": 480, + "unclipped_grad_norm": 1.2546966075897217 + }, + { + "bits_per_byte": 3.102171805433385, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.150261640548706, + "step": 490, + "unclipped_grad_norm": 1.2452603578567505 + }, + { + "bits_per_byte": 3.038328749745395, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.106009006500244, + "step": 500, + "unclipped_grad_norm": 0.8808537125587463 + }, + { + "bits_per_byte": 3.1266280823513783, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.1672134399414062, + "step": 510, + "unclipped_grad_norm": 1.8561674356460571 + }, + { + "bits_per_byte": 3.141152361182044, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.177280902862549, + "step": 520, + "unclipped_grad_norm": 1.2077789306640625 + }, + { + "bits_per_byte": 2.9845563101613277, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.0687367916107178, + "step": 530, + "unclipped_grad_norm": 1.161313772201538 + }, + { + "bits_per_byte": 3.005288818722839, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.0831074714660645, + "step": 540, + "unclipped_grad_norm": 1.0671039819717407 + }, + { + "bits_per_byte": 2.98657194682944, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.070133924484253, + "step": 550, + "unclipped_grad_norm": 0.7664366960525513 + }, + { + "bits_per_byte": 3.0165612495909913, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.090920925140381, + "step": 560, + "unclipped_grad_norm": 1.6851940155029297 + }, + { + "bits_per_byte": 2.980228194769714, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.065736770629883, + "step": 570, + "unclipped_grad_norm": 1.4854934215545654 + }, + { + "bits_per_byte": 2.969506452349625, + "learning_rate": 0.000299626473885547, + "loss_nats": 2.058305025100708, + "step": 580, + "unclipped_grad_norm": 1.1472057104110718 + }, + { + "bits_per_byte": 2.9234642889565388, + "learning_rate": 0.00029958384005397226, + "loss_nats": 2.02639102935791, + "step": 590, + "unclipped_grad_norm": 1.2558393478393555 + }, + { + "bits_per_byte": 2.8785169671188457, + "learning_rate": 0.0002995389065559004, + "loss_nats": 1.9952359199523926, + "step": 600, + "unclipped_grad_norm": 1.0852571725845337 + }, + { + "bits_per_byte": 2.886233484697758, + "learning_rate": 0.000299491674159121, + "loss_nats": 2.000584602355957, + "step": 610, + "unclipped_grad_norm": 1.307874083518982 + }, + { + "bits_per_byte": 2.872304609804379, + "learning_rate": 0.00029944214367070535, + "loss_nats": 1.9909298419952393, + "step": 620, + "unclipped_grad_norm": 0.9391064047813416 + }, + { + "bits_per_byte": 2.7929576621064087, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.9359307289123535, + "step": 630, + "unclipped_grad_norm": 1.0197348594665527 + }, + { + "bits_per_byte": 2.7870357834892494, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.9318259954452515, + "step": 640, + "unclipped_grad_norm": 0.9835532903671265 + }, + { + "bits_per_byte": 2.7715578608595495, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.9210975170135498, + "step": 650, + "unclipped_grad_norm": 0.7845844626426697 + }, + { + "bits_per_byte": 2.798538671109558, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.939799189567566, + "step": 660, + "unclipped_grad_norm": 1.0332822799682617 + }, + { + "bits_per_byte": 2.775885632285861, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.9240972995758057, + "step": 670, + "unclipped_grad_norm": 0.8689291477203369 + }, + { + "bits_per_byte": 2.7583200122573373, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.911921739578247, + "step": 680, + "unclipped_grad_norm": 1.4010518789291382 + }, + { + "bits_per_byte": 2.7316871229100244, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.8934612274169922, + "step": 690, + "unclipped_grad_norm": 1.2936400175094604 + }, + { + "bits_per_byte": 2.705706735740085, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.875452995300293, + "step": 700, + "unclipped_grad_norm": 1.1370997428894043 + }, + { + "bits_per_byte": 2.686462909004442, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.8621141910552979, + "step": 710, + "unclipped_grad_norm": 1.2321908473968506 + }, + { + "bits_per_byte": 2.6936407769208524, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.8670895099639893, + "step": 720, + "unclipped_grad_norm": 1.0070140361785889 + }, + { + "bits_per_byte": 2.6632684687958292, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.8460370302200317, + "step": 730, + "unclipped_grad_norm": 1.0374186038970947 + }, + { + "bits_per_byte": 2.6231308137721654, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.8182157278060913, + "step": 740, + "unclipped_grad_norm": 0.9135240316390991 + }, + { + "bits_per_byte": 2.635020146408962, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.8264567852020264, + "step": 750, + "unclipped_grad_norm": 1.0210453271865845 + }, + { + "bits_per_byte": 2.5473131220525507, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.7656629085540771, + "step": 760, + "unclipped_grad_norm": 1.3631796836853027 + }, + { + "bits_per_byte": 2.577853801192768, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.7868320941925049, + "step": 770, + "unclipped_grad_norm": 1.02455472946167 + }, + { + "bits_per_byte": 2.5163450660249387, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.7441974878311157, + "step": 780, + "unclipped_grad_norm": 1.0328238010406494 + }, + { + "bits_per_byte": 2.52076639601332, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.7472621202468872, + "step": 790, + "unclipped_grad_norm": 1.1721470355987549 + }, + { + "bits_per_byte": 2.4420958560811927, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.6927318572998047, + "step": 800, + "unclipped_grad_norm": 0.8851104974746704 + }, + { + "bits_per_byte": 2.4918886171242947, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.727245569229126, + "step": 810, + "unclipped_grad_norm": 1.1024259328842163 + }, + { + "bits_per_byte": 2.452030089942981, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.6996177434921265, + "step": 820, + "unclipped_grad_norm": 0.912616491317749 + }, + { + "bits_per_byte": 2.4660296497058645, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.7093214988708496, + "step": 830, + "unclipped_grad_norm": 0.904116690158844 + }, + { + "bits_per_byte": 2.462990372299817, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.7072148323059082, + "step": 840, + "unclipped_grad_norm": 1.005014181137085 + }, + { + "bits_per_byte": 2.3901656946531675, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.6567366123199463, + "step": 850, + "unclipped_grad_norm": 1.194644808769226 + }, + { + "bits_per_byte": 2.450779260123256, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.6987507343292236, + "step": 860, + "unclipped_grad_norm": 1.0945597887039185 + }, + { + "bits_per_byte": 2.4317177349975276, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.6855382919311523, + "step": 870, + "unclipped_grad_norm": 1.0461328029632568 + }, + { + "bits_per_byte": 2.382890484556369, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.6516938209533691, + "step": 880, + "unclipped_grad_norm": 0.867928683757782 + }, + { + "bits_per_byte": 2.332987310651413, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.6171035766601562, + "step": 890, + "unclipped_grad_norm": 0.8912314176559448 + }, + { + "bits_per_byte": 2.418546271698548, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.6764085292816162, + "step": 900, + "unclipped_grad_norm": 1.5286786556243896 + }, + { + "bits_per_byte": 2.3662745527259097, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.64017653465271, + "step": 910, + "unclipped_grad_norm": 1.015963077545166 + }, + { + "bits_per_byte": 2.3704392845992035, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.6430633068084717, + "step": 920, + "unclipped_grad_norm": 0.9063786864280701 + }, + { + "bits_per_byte": 2.2980965023402717, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.592919111251831, + "step": 930, + "unclipped_grad_norm": 1.0599802732467651 + }, + { + "bits_per_byte": 2.3622845552258926, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.6374108791351318, + "step": 940, + "unclipped_grad_norm": 1.1740024089813232 + }, + { + "bits_per_byte": 2.3604168236375225, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.6361162662506104, + "step": 950, + "unclipped_grad_norm": 0.9503087997436523 + }, + { + "bits_per_byte": 2.296936651342853, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.5921151638031006, + "step": 960, + "unclipped_grad_norm": 1.0555206537246704 + }, + { + "bits_per_byte": 2.2441090844426257, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5554978847503662, + "step": 970, + "unclipped_grad_norm": 0.9488848447799683 + }, + { + "bits_per_byte": 2.3860136894960378, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.6538586616516113, + "step": 980, + "unclipped_grad_norm": 1.1395503282546997 + }, + { + "bits_per_byte": 2.2404891936072655, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.5529887676239014, + "step": 990, + "unclipped_grad_norm": 1.2346168756484985 + }, + { + "bits_per_byte": 2.2299753782120693, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.5457011461257935, + "step": 1000, + "unclipped_grad_norm": 1.0828826427459717 + }, + { + "bits_per_byte": 2.2414068930322695, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.5536248683929443, + "step": 1010, + "unclipped_grad_norm": 1.242470145225525 + }, + { + "bits_per_byte": 2.2047018397371763, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.528182864189148, + "step": 1020, + "unclipped_grad_norm": 0.8821380138397217 + }, + { + "bits_per_byte": 2.188689567011245, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.517084002494812, + "step": 1030, + "unclipped_grad_norm": 0.9808963537216187 + }, + { + "bits_per_byte": 2.28820543612385, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.5860631465911865, + "step": 1040, + "unclipped_grad_norm": 1.0366183519363403 + }, + { + "bits_per_byte": 2.1940466546029707, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5207972526550293, + "step": 1050, + "unclipped_grad_norm": 0.9590810537338257 + }, + { + "bits_per_byte": 2.2275732965274684, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.5440361499786377, + "step": 1060, + "unclipped_grad_norm": 0.9160453081130981 + }, + { + "bits_per_byte": 2.269115361878078, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.5728309154510498, + "step": 1070, + "unclipped_grad_norm": 0.9917046427726746 + }, + { + "bits_per_byte": 2.2459345082988835, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.5567631721496582, + "step": 1080, + "unclipped_grad_norm": 0.8373965620994568 + }, + { + "bits_per_byte": 2.1792107432089414, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.5105137825012207, + "step": 1090, + "unclipped_grad_norm": 0.9263890981674194 + }, + { + "bits_per_byte": 2.1782015490136786, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5098142623901367, + "step": 1100, + "unclipped_grad_norm": 0.9069724678993225 + }, + { + "bits_per_byte": 2.1476727366763706, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.488653302192688, + "step": 1110, + "unclipped_grad_norm": 1.292263388633728 + }, + { + "bits_per_byte": 2.1810538812782383, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.5117913484573364, + "step": 1120, + "unclipped_grad_norm": 0.9168121814727783 + }, + { + "bits_per_byte": 2.2131604624545616, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.534045934677124, + "step": 1130, + "unclipped_grad_norm": 0.9439221620559692 + }, + { + "bits_per_byte": 2.133988249129868, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.4791679382324219, + "step": 1140, + "unclipped_grad_norm": 1.0234383344650269 + }, + { + "bits_per_byte": 2.1822923283471014, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.5126497745513916, + "step": 1150, + "unclipped_grad_norm": 0.8971675634384155 + }, + { + "bits_per_byte": 2.0966979388738882, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.4533202648162842, + "step": 1160, + "unclipped_grad_norm": 0.9143848419189453 + }, + { + "bits_per_byte": 2.1553952736480633, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.4940061569213867, + "step": 1170, + "unclipped_grad_norm": 0.8700084686279297 + }, + { + "bits_per_byte": 2.1536847342025816, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.4928205013275146, + "step": 1180, + "unclipped_grad_norm": 0.818593442440033 + }, + { + "bits_per_byte": 2.122175104807403, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.4709796905517578, + "step": 1190, + "unclipped_grad_norm": 0.8767768144607544 + }, + { + "bits_per_byte": 2.1252907425104337, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.4731392860412598, + "step": 1200, + "unclipped_grad_norm": 0.7778198719024658 + }, + { + "bits_per_byte": 2.1517808862575305, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.4915008544921875, + "step": 1210, + "unclipped_grad_norm": 0.9066429734230042 + }, + { + "bits_per_byte": 2.1004155158549387, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4558970928192139, + "step": 1220, + "unclipped_grad_norm": 0.8138765692710876 + }, + { + "bits_per_byte": 2.112482162604775, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.4642610549926758, + "step": 1230, + "unclipped_grad_norm": 0.8797006011009216 + }, + { + "bits_per_byte": 2.0932641332667616, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.4509401321411133, + "step": 1240, + "unclipped_grad_norm": 0.8060585856437683 + }, + { + "bits_per_byte": 2.0272850530550555, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.4052069187164307, + "step": 1250, + "unclipped_grad_norm": 0.8649603128433228 + }, + { + "bits_per_byte": 2.0811730649805025, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.4425592422485352, + "step": 1260, + "unclipped_grad_norm": 0.9500141143798828 + }, + { + "bits_per_byte": 2.102492550329409, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.4573367834091187, + "step": 1270, + "unclipped_grad_norm": 0.8442755341529846 + }, + { + "bits_per_byte": 2.127505535088245, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.4746744632720947, + "step": 1280, + "unclipped_grad_norm": 0.8093211054801941 + }, + { + "bits_per_byte": 2.080182272928882, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.441872477531433, + "step": 1290, + "unclipped_grad_norm": 0.8258889317512512 + }, + { + "bits_per_byte": 2.0315113547173578, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.4081363677978516, + "step": 1300, + "unclipped_grad_norm": 0.938977062702179 + }, + { + "bits_per_byte": 2.0627139950807463, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.4297643899917603, + "step": 1310, + "unclipped_grad_norm": 0.835930347442627 + }, + { + "bits_per_byte": 2.02743949347553, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.4053139686584473, + "step": 1320, + "unclipped_grad_norm": 0.8066462278366089 + }, + { + "bits_per_byte": 2.0477824613328375, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.4194146394729614, + "step": 1330, + "unclipped_grad_norm": 0.8747845888137817 + }, + { + "bits_per_byte": 2.030569921686535, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.4074838161468506, + "step": 1340, + "unclipped_grad_norm": 0.8193429112434387 + }, + { + "bits_per_byte": 2.0556384568415997, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.4248600006103516, + "step": 1350, + "unclipped_grad_norm": 0.8351138234138489 + }, + { + "bits_per_byte": 2.0213186309313227, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.401071310043335, + "step": 1360, + "unclipped_grad_norm": 0.8565230369567871 + }, + { + "bits_per_byte": 2.0392915058770895, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4135291576385498, + "step": 1370, + "unclipped_grad_norm": 0.8084989190101624 + }, + { + "bits_per_byte": 2.017209793419559, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.3982232809066772, + "step": 1380, + "unclipped_grad_norm": 0.8856196403503418 + }, + { + "bits_per_byte": 2.064509665938405, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.43100905418396, + "step": 1390, + "unclipped_grad_norm": 0.7775274515151978 + }, + { + "bits_per_byte": 2.175997075394919, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.5082862377166748, + "step": 1400, + "unclipped_grad_norm": 0.8742821216583252 + }, + { + "bits_per_byte": 2.044112523546076, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.4168708324432373, + "step": 1410, + "unclipped_grad_norm": 0.8183583617210388 + }, + { + "bits_per_byte": 2.0460301301031962, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.4182000160217285, + "step": 1420, + "unclipped_grad_norm": 0.7739344835281372 + }, + { + "bits_per_byte": 1.982891859318442, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.3744359016418457, + "step": 1430, + "unclipped_grad_norm": 0.8165663480758667 + }, + { + "bits_per_byte": 2.0114564578002456, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.394235372543335, + "step": 1440, + "unclipped_grad_norm": 0.8117446899414062 + }, + { + "bits_per_byte": 1.990985018902744, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.3800456523895264, + "step": 1450, + "unclipped_grad_norm": 0.8170514106750488 + }, + { + "bits_per_byte": 2.095291464755132, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.452345371246338, + "step": 1460, + "unclipped_grad_norm": 0.7954216003417969 + }, + { + "bits_per_byte": 2.0753321901919, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.4385106563568115, + "step": 1470, + "unclipped_grad_norm": 0.7490127086639404 + }, + { + "bits_per_byte": 2.0216622522677468, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.4013094902038574, + "step": 1480, + "unclipped_grad_norm": 0.7507129311561584 + }, + { + "bits_per_byte": 2.052454026078224, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.4226527214050293, + "step": 1490, + "unclipped_grad_norm": 0.9437978267669678 + }, + { + "bits_per_byte": 1.9253657263966428, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.334561824798584, + "step": 1500, + "unclipped_grad_norm": 0.8041139841079712 + }, + { + "bits_per_byte": 1.937312329256177, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.3428425788879395, + "step": 1510, + "unclipped_grad_norm": 0.8608347773551941 + }, + { + "bits_per_byte": 1.9981711419863957, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.3850266933441162, + "step": 1520, + "unclipped_grad_norm": 0.9105530977249146 + }, + { + "bits_per_byte": 2.0316162641343842, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.4082090854644775, + "step": 1530, + "unclipped_grad_norm": 0.7693362832069397 + }, + { + "bits_per_byte": 2.0085539066016342, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3922234773635864, + "step": 1540, + "unclipped_grad_norm": 0.7877471446990967 + }, + { + "bits_per_byte": 2.066966954053933, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.4327123165130615, + "step": 1550, + "unclipped_grad_norm": 0.7693803310394287 + }, + { + "bits_per_byte": 1.9350347630108007, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.3412638902664185, + "step": 1560, + "unclipped_grad_norm": 0.718927264213562 + }, + { + "bits_per_byte": 1.9434791111681646, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.3471170663833618, + "step": 1570, + "unclipped_grad_norm": 0.722200870513916 + }, + { + "bits_per_byte": 1.9595871782306262, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.3582823276519775, + "step": 1580, + "unclipped_grad_norm": 0.7644280195236206 + }, + { + "bits_per_byte": 1.9522772276383533, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3532154560089111, + "step": 1590, + "unclipped_grad_norm": 0.7328327894210815 + }, + { + "bits_per_byte": 1.9667896396661098, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.3632746934890747, + "step": 1600, + "unclipped_grad_norm": 0.8042165637016296 + }, + { + "bits_per_byte": 1.9243668511604317, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.333869457244873, + "step": 1610, + "unclipped_grad_norm": 0.8300740122795105 + }, + { + "bits_per_byte": 1.9323576810674703, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.339408278465271, + "step": 1620, + "unclipped_grad_norm": 0.7256625294685364 + }, + { + "bits_per_byte": 1.9799870723453692, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.372422456741333, + "step": 1630, + "unclipped_grad_norm": 0.7714943289756775 + }, + { + "bits_per_byte": 1.9178665948884208, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.3293638229370117, + "step": 1640, + "unclipped_grad_norm": 0.7875189781188965 + }, + { + "bits_per_byte": 1.9144902314866823, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.3270235061645508, + "step": 1650, + "unclipped_grad_norm": 0.7027429342269897 + }, + { + "bits_per_byte": 1.952266392731349, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.3532079458236694, + "step": 1660, + "unclipped_grad_norm": 0.8333705067634583 + }, + { + "bits_per_byte": 1.8988571805059038, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.3161875009536743, + "step": 1670, + "unclipped_grad_norm": 0.7823120355606079 + }, + { + "bits_per_byte": 1.9302804746103492, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.3379684686660767, + "step": 1680, + "unclipped_grad_norm": 0.7404139041900635 + }, + { + "bits_per_byte": 1.9467739547933944, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.3494008779525757, + "step": 1690, + "unclipped_grad_norm": 0.7751686573028564 + }, + { + "bits_per_byte": 1.8838143739828865, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.3057606220245361, + "step": 1700, + "unclipped_grad_norm": 0.7513799071311951 + }, + { + "bits_per_byte": 1.9325669839535704, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3395533561706543, + "step": 1710, + "unclipped_grad_norm": 0.6842343211174011 + }, + { + "bits_per_byte": 1.8549531054283444, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.2857555150985718, + "step": 1720, + "unclipped_grad_norm": 0.7686919569969177 + }, + { + "bits_per_byte": 1.9481938435589177, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.350385069847107, + "step": 1730, + "unclipped_grad_norm": 0.8331624865531921 + }, + { + "bits_per_byte": 1.9507735833218598, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.3521732091903687, + "step": 1740, + "unclipped_grad_norm": 0.7184111475944519 + }, + { + "bits_per_byte": 1.902781824598593, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.3189078569412231, + "step": 1750, + "unclipped_grad_norm": 0.6777728199958801 + }, + { + "bits_per_byte": 1.9539000559318949, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.3543403148651123, + "step": 1760, + "unclipped_grad_norm": 0.7190402150154114 + }, + { + "bits_per_byte": 1.9356196760063853, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.3416693210601807, + "step": 1770, + "unclipped_grad_norm": 0.7220019102096558 + }, + { + "bits_per_byte": 1.995209600738538, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.3829739093780518, + "step": 1780, + "unclipped_grad_norm": 0.7851740121841431 + }, + { + "bits_per_byte": 1.8853307450155439, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3068116903305054, + "step": 1790, + "unclipped_grad_norm": 0.762315571308136 + }, + { + "bits_per_byte": 2.018881292803295, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.3993818759918213, + "step": 1800, + "unclipped_grad_norm": 0.8214138746261597 + }, + { + "bits_per_byte": 1.8867274161232008, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.3077797889709473, + "step": 1810, + "unclipped_grad_norm": 0.8148207664489746 + }, + { + "bits_per_byte": 1.9277700438557048, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.336228370666504, + "step": 1820, + "unclipped_grad_norm": 0.7489343285560608 + }, + { + "bits_per_byte": 1.965684995099618, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.36250901222229, + "step": 1830, + "unclipped_grad_norm": 0.7481802701950073 + }, + { + "bits_per_byte": 1.94583028598811, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.3487467765808105, + "step": 1840, + "unclipped_grad_norm": 0.7497571110725403 + }, + { + "bits_per_byte": 1.875044806632741, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.2996820211410522, + "step": 1850, + "unclipped_grad_norm": 0.7759279012680054 + }, + { + "bits_per_byte": 1.9093629127165093, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.3234695196151733, + "step": 1860, + "unclipped_grad_norm": 0.75089430809021 + }, + { + "bits_per_byte": 1.9908316103781745, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.379939317703247, + "step": 1870, + "unclipped_grad_norm": 0.755817174911499 + }, + { + "bits_per_byte": 1.8767018594738043, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.300830602645874, + "step": 1880, + "unclipped_grad_norm": 0.7015770077705383 + }, + { + "bits_per_byte": 1.8996357459663598, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3167271614074707, + "step": 1890, + "unclipped_grad_norm": 0.7695561647415161 + }, + { + "bits_per_byte": 1.903293301002259, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.3192623853683472, + "step": 1900, + "unclipped_grad_norm": 0.7171405553817749 + }, + { + "bits_per_byte": 1.943133941987883, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.3468778133392334, + "step": 1910, + "unclipped_grad_norm": 0.7770541906356812 + }, + { + "bits_per_byte": 1.8756907734693817, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.300129771232605, + "step": 1920, + "unclipped_grad_norm": 0.7630769610404968 + }, + { + "bits_per_byte": 1.9268630073550543, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.335599660873413, + "step": 1930, + "unclipped_grad_norm": 0.7087206840515137 + }, + { + "bits_per_byte": 1.947787448554929, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.3501033782958984, + "step": 1940, + "unclipped_grad_norm": 0.7327392101287842 + }, + { + "bits_per_byte": 1.883164623527927, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.3053102493286133, + "step": 1950, + "unclipped_grad_norm": 0.7805818915367126 + }, + { + "bits_per_byte": 1.8449655569447885, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.2788326740264893, + "step": 1960, + "unclipped_grad_norm": 0.6896875500679016 + }, + { + "bits_per_byte": 1.8737879574202356, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.2988108396530151, + "step": 1970, + "unclipped_grad_norm": 0.7341562509536743 + }, + { + "bits_per_byte": 1.8907045149244033, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.310536503791809, + "step": 1980, + "unclipped_grad_norm": 0.7556041479110718 + }, + { + "bits_per_byte": 1.8592789850454965, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.2887539863586426, + "step": 1990, + "unclipped_grad_norm": 0.7086564302444458 + }, + { + "bits_per_byte": 1.942528391074195, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.346458077430725, + "step": 2000, + "unclipped_grad_norm": 0.8556731343269348 + }, + { + "bits_per_byte": 1.9381970080122153, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.3434557914733887, + "step": 2010, + "unclipped_grad_norm": 0.7243759036064148 + }, + { + "bits_per_byte": 1.8463857896756135, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.2798171043395996, + "step": 2020, + "unclipped_grad_norm": 0.6738364696502686 + }, + { + "bits_per_byte": 1.8699346861276545, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.2961399555206299, + "step": 2030, + "unclipped_grad_norm": 0.6850672960281372 + }, + { + "bits_per_byte": 1.8793380095462293, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.3026578426361084, + "step": 2040, + "unclipped_grad_norm": 0.8650861382484436 + }, + { + "bits_per_byte": 1.8393431001227813, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.2749354839324951, + "step": 2050, + "unclipped_grad_norm": 0.7513278126716614 + }, + { + "bits_per_byte": 1.8171215657700996, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.2595326900482178, + "step": 2060, + "unclipped_grad_norm": 0.6367834210395813 + }, + { + "bits_per_byte": 1.9370622664818224, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.3426692485809326, + "step": 2070, + "unclipped_grad_norm": 0.7583683133125305 + }, + { + "bits_per_byte": 1.92415324870806, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.333721399307251, + "step": 2080, + "unclipped_grad_norm": 0.7473825812339783 + }, + { + "bits_per_byte": 1.8324606984005536, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.270164966583252, + "step": 2090, + "unclipped_grad_norm": 0.6806753873825073 + }, + { + "bits_per_byte": 1.7697735381090125, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.2267135381698608, + "step": 2100, + "unclipped_grad_norm": 0.6726358532905579 + }, + { + "bits_per_byte": 1.834129962009828, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.2713220119476318, + "step": 2110, + "unclipped_grad_norm": 0.7028688192367554 + }, + { + "bits_per_byte": 1.8343755532352601, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.2714922428131104, + "step": 2120, + "unclipped_grad_norm": 0.7491693496704102 + }, + { + "bits_per_byte": 1.8749737777979347, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.2996327877044678, + "step": 2130, + "unclipped_grad_norm": 0.726253092288971 + }, + { + "bits_per_byte": 1.9073871760233974, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.322100043296814, + "step": 2140, + "unclipped_grad_norm": 0.7610151767730713 + }, + { + "bits_per_byte": 1.8010492710990176, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2483922243118286, + "step": 2150, + "unclipped_grad_norm": 0.7295191287994385 + }, + { + "bits_per_byte": 1.9008336051296193, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.317557454109192, + "step": 2160, + "unclipped_grad_norm": 0.70853191614151 + }, + { + "bits_per_byte": 1.8242574699196994, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.2644789218902588, + "step": 2170, + "unclipped_grad_norm": 0.7063313722610474 + }, + { + "bits_per_byte": 1.8907669446266664, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.310579776763916, + "step": 2180, + "unclipped_grad_norm": 0.7122225165367126 + }, + { + "bits_per_byte": 1.870281059186492, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.2963800430297852, + "step": 2190, + "unclipped_grad_norm": 0.7267935872077942 + }, + { + "bits_per_byte": 1.8296795669534511, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.2682372331619263, + "step": 2200, + "unclipped_grad_norm": 0.6863276958465576 + }, + { + "bits_per_byte": 1.8399612057699823, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.2753639221191406, + "step": 2210, + "unclipped_grad_norm": 0.7334514856338501 + }, + { + "bits_per_byte": 1.8237373943834903, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2641184329986572, + "step": 2220, + "unclipped_grad_norm": 0.7128761410713196 + }, + { + "bits_per_byte": 1.7809286768092758, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.2344456911087036, + "step": 2230, + "unclipped_grad_norm": 0.6696409583091736 + }, + { + "bits_per_byte": 1.8906536080597478, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.310501217842102, + "step": 2240, + "unclipped_grad_norm": 0.7485113739967346 + }, + { + "bits_per_byte": 1.853308779303445, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2846157550811768, + "step": 2250, + "unclipped_grad_norm": 0.7387778162956238 + }, + { + "bits_per_byte": 1.9012721608893193, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.3178614377975464, + "step": 2260, + "unclipped_grad_norm": 0.7693575024604797 + }, + { + "bits_per_byte": 1.791799528187663, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.241980791091919, + "step": 2270, + "unclipped_grad_norm": 0.7290365099906921 + }, + { + "bits_per_byte": 1.8533393922152985, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.2846369743347168, + "step": 2280, + "unclipped_grad_norm": 0.706203043460846 + }, + { + "bits_per_byte": 1.7976562253801454, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.2460403442382812, + "step": 2290, + "unclipped_grad_norm": 0.7111358642578125 + }, + { + "bits_per_byte": 1.8053084214476351, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.2513444423675537, + "step": 2300, + "unclipped_grad_norm": 0.7287721037864685 + }, + { + "bits_per_byte": 1.8147779581867922, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.2579082250595093, + "step": 2310, + "unclipped_grad_norm": 0.6768566966056824 + }, + { + "bits_per_byte": 1.8712725391687162, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.2970672845840454, + "step": 2320, + "unclipped_grad_norm": 0.7173076272010803 + }, + { + "bits_per_byte": 1.7882852346999325, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.2395448684692383, + "step": 2330, + "unclipped_grad_norm": 0.6917617917060852 + }, + { + "bits_per_byte": 1.8129167619391549, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.2566181421279907, + "step": 2340, + "unclipped_grad_norm": 0.7177107930183411 + }, + { + "bits_per_byte": 1.8716729147799247, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2973448038101196, + "step": 2350, + "unclipped_grad_norm": 0.7809521555900574 + }, + { + "bits_per_byte": 1.8011393900080697, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.2484546899795532, + "step": 2360, + "unclipped_grad_norm": 0.7523290514945984 + }, + { + "bits_per_byte": 1.7302151206534544, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.1992937326431274, + "step": 2370, + "unclipped_grad_norm": 0.6691831946372986 + }, + { + "bits_per_byte": 1.8263635694621654, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.2659387588500977, + "step": 2380, + "unclipped_grad_norm": 0.7132697701454163 + }, + { + "bits_per_byte": 1.830821875720482, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.2690290212631226, + "step": 2390, + "unclipped_grad_norm": 0.6670538783073425 + }, + { + "bits_per_byte": 1.859929079465758, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.2892045974731445, + "step": 2400, + "unclipped_grad_norm": 0.7368744611740112 + }, + { + "bits_per_byte": 1.8030573405304917, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2497841119766235, + "step": 2410, + "unclipped_grad_norm": 0.7534800171852112 + }, + { + "bits_per_byte": 1.8176222072667614, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.2598797082901, + "step": 2420, + "unclipped_grad_norm": 0.6879056096076965 + }, + { + "bits_per_byte": 1.8309150903172495, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.269093632698059, + "step": 2430, + "unclipped_grad_norm": 0.6876195669174194 + }, + { + "bits_per_byte": 1.7750346593815567, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.2303602695465088, + "step": 2440, + "unclipped_grad_norm": 0.7317672371864319 + }, + { + "bits_per_byte": 1.893689617795429, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.312605619430542, + "step": 2450, + "unclipped_grad_norm": 0.8147894740104675 + }, + { + "bits_per_byte": 1.7761933065004194, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.2311633825302124, + "step": 2460, + "unclipped_grad_norm": 0.7546675801277161 + }, + { + "bits_per_byte": 1.7741062970321992, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.2297167778015137, + "step": 2470, + "unclipped_grad_norm": 0.7132146954536438 + }, + { + "bits_per_byte": 1.869709560837675, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.295983910560608, + "step": 2480, + "unclipped_grad_norm": 0.7493357062339783 + }, + { + "bits_per_byte": 1.849160557764634, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.281740427017212, + "step": 2490, + "unclipped_grad_norm": 0.6876279711723328 + }, + { + "bits_per_byte": 1.8096757488836452, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2543716430664062, + "step": 2500, + "unclipped_grad_norm": 0.743791401386261 + }, + { + "bits_per_byte": 1.7666941887453138, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.224579095840454, + "step": 2510, + "unclipped_grad_norm": 0.721203625202179 + }, + { + "bits_per_byte": 1.7932759992453204, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.2430042028427124, + "step": 2520, + "unclipped_grad_norm": 0.7382795214653015 + }, + { + "bits_per_byte": 1.8122365705549925, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.2561466693878174, + "step": 2530, + "unclipped_grad_norm": 0.7209014892578125 + }, + { + "bits_per_byte": 1.812251017097665, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.2561566829681396, + "step": 2540, + "unclipped_grad_norm": 0.742669403553009 + }, + { + "bits_per_byte": 1.7529601701953625, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.2150593996047974, + "step": 2550, + "unclipped_grad_norm": 0.7028670310974121 + }, + { + "bits_per_byte": 1.7352891248018598, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.2028107643127441, + "step": 2560, + "unclipped_grad_norm": 0.6918957829475403 + }, + { + "bits_per_byte": 1.7657372772759132, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2239158153533936, + "step": 2570, + "unclipped_grad_norm": 0.6825562119483948 + }, + { + "bits_per_byte": 1.7525438001976237, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.214770793914795, + "step": 2580, + "unclipped_grad_norm": 0.7184486985206604 + }, + { + "bits_per_byte": 1.745061866954488, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.2095847129821777, + "step": 2590, + "unclipped_grad_norm": 0.6832425594329834 + }, + { + "bits_per_byte": 1.7670106368229015, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.2247984409332275, + "step": 2600, + "unclipped_grad_norm": 0.6995975971221924 + }, + { + "bits_per_byte": 1.7903504023714931, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.240976333618164, + "step": 2610, + "unclipped_grad_norm": 0.7396151423454285 + }, + { + "bits_per_byte": 1.731659602938051, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.2002949714660645, + "step": 2620, + "unclipped_grad_norm": 0.6839298009872437 + }, + { + "bits_per_byte": 1.724905328290716, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.1956132650375366, + "step": 2630, + "unclipped_grad_norm": 0.6664920449256897 + }, + { + "bits_per_byte": 1.706663988426952, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.182969331741333, + "step": 2640, + "unclipped_grad_norm": 0.6552896499633789 + }, + { + "bits_per_byte": 1.7896167243829124, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2404677867889404, + "step": 2650, + "unclipped_grad_norm": 0.7217641472816467 + }, + { + "bits_per_byte": 1.6914314850400332, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.1724109649658203, + "step": 2660, + "unclipped_grad_norm": 0.6797831654548645 + }, + { + "bits_per_byte": 1.763040933275686, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.2220468521118164, + "step": 2670, + "unclipped_grad_norm": 0.7010027170181274 + }, + { + "bits_per_byte": 1.858818415506486, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.2884347438812256, + "step": 2680, + "unclipped_grad_norm": 0.7553756833076477 + }, + { + "bits_per_byte": 1.7901995735866865, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.240871787071228, + "step": 2690, + "unclipped_grad_norm": 0.7245635390281677 + }, + { + "bits_per_byte": 1.7041855464453683, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.1812514066696167, + "step": 2700, + "unclipped_grad_norm": 0.6864461898803711 + }, + { + "bits_per_byte": 1.772678497064736, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.228727102279663, + "step": 2710, + "unclipped_grad_norm": 0.6734927892684937 + }, + { + "bits_per_byte": 1.829608194153343, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.2681877613067627, + "step": 2720, + "unclipped_grad_norm": 0.7714908719062805 + }, + { + "bits_per_byte": 1.7186312292045898, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.1912643909454346, + "step": 2730, + "unclipped_grad_norm": 0.6988846063613892 + }, + { + "bits_per_byte": 1.788071632247561, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.2393968105316162, + "step": 2740, + "unclipped_grad_norm": 0.7007521986961365 + }, + { + "bits_per_byte": 1.7522283840159412, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.2145521640777588, + "step": 2750, + "unclipped_grad_norm": 0.6860843300819397 + }, + { + "bits_per_byte": 1.8760005142235856, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.300344467163086, + "step": 2760, + "unclipped_grad_norm": 0.8516666889190674 + }, + { + "bits_per_byte": 1.7840018347975435, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.2365758419036865, + "step": 2770, + "unclipped_grad_norm": 0.6746839284896851 + }, + { + "bits_per_byte": 1.7953630087135406, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.2444508075714111, + "step": 2780, + "unclipped_grad_norm": 0.7184997797012329 + }, + { + "bits_per_byte": 1.7590812047222208, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2193021774291992, + "step": 2790, + "unclipped_grad_norm": 0.723415195941925 + }, + { + "bits_per_byte": 1.7686214263308824, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.2259149551391602, + "step": 2800, + "unclipped_grad_norm": 0.6828456521034241 + }, + { + "bits_per_byte": 1.7113491398017568, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.1862168312072754, + "step": 2810, + "unclipped_grad_norm": 0.7083114385604858 + }, + { + "bits_per_byte": 1.781849815887297, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2350841760635376, + "step": 2820, + "unclipped_grad_norm": 0.7157456874847412 + }, + { + "bits_per_byte": 1.7286908384188573, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.1982371807098389, + "step": 2830, + "unclipped_grad_norm": 0.6711781620979309 + }, + { + "bits_per_byte": 1.7693685189662305, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.2264328002929688, + "step": 2840, + "unclipped_grad_norm": 0.7228206396102905 + }, + { + "bits_per_byte": 1.7116552689202926, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.1864290237426758, + "step": 2850, + "unclipped_grad_norm": 0.737615168094635 + }, + { + "bits_per_byte": 1.7514998655068863, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.2140471935272217, + "step": 2860, + "unclipped_grad_norm": 0.7378557324409485 + }, + { + "bits_per_byte": 1.744272122621726, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.2090373039245605, + "step": 2870, + "unclipped_grad_norm": 0.7257314324378967 + }, + { + "bits_per_byte": 1.7425237469930548, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2078254222869873, + "step": 2880, + "unclipped_grad_norm": 0.672211229801178 + }, + { + "bits_per_byte": 1.709284144112847, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.1847854852676392, + "step": 2890, + "unclipped_grad_norm": 0.7092587351799011 + }, + { + "bits_per_byte": 1.7405380353061928, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.206449031829834, + "step": 2900, + "unclipped_grad_norm": 0.6935831904411316 + }, + { + "bits_per_byte": 1.7068591887356814, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.1831046342849731, + "step": 2910, + "unclipped_grad_norm": 0.711025595664978 + }, + { + "bits_per_byte": 1.7553340467252219, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.2167048454284668, + "step": 2920, + "unclipped_grad_norm": 0.6738850474357605 + }, + { + "bits_per_byte": 1.7049768386219881, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.1817998886108398, + "step": 2930, + "unclipped_grad_norm": 0.6651322841644287 + }, + { + "bits_per_byte": 1.7141916690552172, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.1881871223449707, + "step": 2940, + "unclipped_grad_norm": 0.689257025718689 + }, + { + "bits_per_byte": 1.7680182831743065, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2254968881607056, + "step": 2950, + "unclipped_grad_norm": 0.6821886301040649 + }, + { + "bits_per_byte": 1.7801122751656302, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.233879804611206, + "step": 2960, + "unclipped_grad_norm": 0.8224455118179321 + }, + { + "bits_per_byte": 1.7479522073848883, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.2115881443023682, + "step": 2970, + "unclipped_grad_norm": 0.6831454634666443 + }, + { + "bits_per_byte": 1.77221483183801, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.2284057140350342, + "step": 2980, + "unclipped_grad_norm": 0.6796787977218628 + }, + { + "bits_per_byte": 1.7299076156737117, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.1990805864334106, + "step": 2990, + "unclipped_grad_norm": 0.7482106685638428 + }, + { + "bits_per_byte": 1.7914466197880927, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.2417361736297607, + "step": 3000, + "unclipped_grad_norm": 0.7187989950180054 + }, + { + "bits_per_byte": 1.6769216527520399, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.162353515625, + "step": 3010, + "unclipped_grad_norm": 0.697723388671875 + }, + { + "bits_per_byte": 1.7481943589573032, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.2117559909820557, + "step": 3020, + "unclipped_grad_norm": 0.6844891309738159 + }, + { + "bits_per_byte": 1.6947856626798106, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.1747359037399292, + "step": 3030, + "unclipped_grad_norm": 0.6614864468574524 + }, + { + "bits_per_byte": 1.6767701360366296, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.1622484922409058, + "step": 3040, + "unclipped_grad_norm": 0.6673623323440552 + }, + { + "bits_per_byte": 1.8249316419110815, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.2649462223052979, + "step": 3050, + "unclipped_grad_norm": 0.7603517174720764 + }, + { + "bits_per_byte": 1.7745770135476107, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.2300430536270142, + "step": 3060, + "unclipped_grad_norm": 0.7315406203269958 + }, + { + "bits_per_byte": 1.6806209995720989, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.1649177074432373, + "step": 3070, + "unclipped_grad_norm": 0.7228949069976807 + }, + { + "bits_per_byte": 1.760089023056277, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.2200007438659668, + "step": 3080, + "unclipped_grad_norm": 0.6559469103813171 + }, + { + "bits_per_byte": 1.742009346884324, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.207468867301941, + "step": 3090, + "unclipped_grad_norm": 0.6904670000076294 + }, + { + "bits_per_byte": 1.7085502941416155, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.184276819229126, + "step": 3100, + "unclipped_grad_norm": 0.7423045635223389 + }, + { + "bits_per_byte": 1.8358556359285854, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.2725181579589844, + "step": 3110, + "unclipped_grad_norm": 0.7265564203262329 + }, + { + "bits_per_byte": 1.7298083816841638, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.1990118026733398, + "step": 3120, + "unclipped_grad_norm": 0.7553577423095703 + }, + { + "bits_per_byte": 1.6804574440711282, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.1648043394088745, + "step": 3130, + "unclipped_grad_norm": 0.7492260336875916 + }, + { + "bits_per_byte": 1.7151860727425061, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.1888763904571533, + "step": 3140, + "unclipped_grad_norm": 0.7066723704338074 + }, + { + "bits_per_byte": 1.6904284822202014, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.1717157363891602, + "step": 3150, + "unclipped_grad_norm": 0.6893080472946167 + }, + { + "bits_per_byte": 1.7011340582711094, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.1791362762451172, + "step": 3160, + "unclipped_grad_norm": 0.6745665073394775 + }, + { + "bits_per_byte": 1.755486767319188, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.216810703277588, + "step": 3170, + "unclipped_grad_norm": 0.7334657311439514 + }, + { + "bits_per_byte": 1.748718562077133, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.2121193408966064, + "step": 3180, + "unclipped_grad_norm": 0.7552371025085449 + }, + { + "bits_per_byte": 1.7330464710346087, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.201256275177002, + "step": 3190, + "unclipped_grad_norm": 0.6962037086486816 + }, + { + "bits_per_byte": 1.7172849490136357, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.190331220626831, + "step": 3200, + "unclipped_grad_norm": 0.6874222755432129 + }, + { + "bits_per_byte": 1.6881820448346314, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.1701586246490479, + "step": 3210, + "unclipped_grad_norm": 0.658416211605072 + }, + { + "bits_per_byte": 1.673619413872823, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.1600645780563354, + "step": 3220, + "unclipped_grad_norm": 0.6980633735656738 + }, + { + "bits_per_byte": 1.7251007005820962, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.1957486867904663, + "step": 3230, + "unclipped_grad_norm": 0.6744846701622009 + }, + { + "bits_per_byte": 1.7536018374657316, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.2155041694641113, + "step": 3240, + "unclipped_grad_norm": 0.7512171864509583 + }, + { + "bits_per_byte": 1.7109523758262164, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.1859418153762817, + "step": 3250, + "unclipped_grad_norm": 0.6887981295585632 + }, + { + "bits_per_byte": 1.7057646911455906, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.182345986366272, + "step": 3260, + "unclipped_grad_norm": 0.7228181958198547 + }, + { + "bits_per_byte": 1.7540730699290958, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.2158308029174805, + "step": 3270, + "unclipped_grad_norm": 0.7397167682647705 + }, + { + "bits_per_byte": 1.747389824116567, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.211198329925537, + "step": 3280, + "unclipped_grad_norm": 0.696940541267395 + }, + { + "bits_per_byte": 1.7075223538374085, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.183564305305481, + "step": 3290, + "unclipped_grad_norm": 0.7643493413925171 + }, + { + "bits_per_byte": 1.7117904472838708, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.1865227222442627, + "step": 3300, + "unclipped_grad_norm": 0.7417942881584167 + }, + { + "bits_per_byte": 1.6974556933344558, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.176586627960205, + "step": 3310, + "unclipped_grad_norm": 0.6708903312683105 + }, + { + "bits_per_byte": 1.7037664247252156, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.1809608936309814, + "step": 3320, + "unclipped_grad_norm": 0.6724581718444824 + }, + { + "bits_per_byte": 1.6970162776615012, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.1762820482254028, + "step": 3330, + "unclipped_grad_norm": 0.7151966094970703 + }, + { + "bits_per_byte": 1.6474059902109646, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.141894817352295, + "step": 3340, + "unclipped_grad_norm": 0.7047701478004456 + }, + { + "bits_per_byte": 1.6916306409497324, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.1725490093231201, + "step": 3350, + "unclipped_grad_norm": 0.6814690232276917 + }, + { + "bits_per_byte": 1.7195062769321798, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.191870927810669, + "step": 3360, + "unclipped_grad_norm": 0.7149270176887512 + }, + { + "bits_per_byte": 1.6516697840911554, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.144850254058838, + "step": 3370, + "unclipped_grad_norm": 0.6849899888038635 + }, + { + "bits_per_byte": 1.6877283546016553, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.169844150543213, + "step": 3380, + "unclipped_grad_norm": 0.6879368424415588 + }, + { + "bits_per_byte": 1.7049427860571171, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.1817762851715088, + "step": 3390, + "unclipped_grad_norm": 0.741426408290863 + }, + { + "bits_per_byte": 1.68899105122429, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.1707193851470947, + "step": 3400, + "unclipped_grad_norm": 0.6700107455253601 + }, + { + "bits_per_byte": 1.678036960242885, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.1631265878677368, + "step": 3410, + "unclipped_grad_norm": 0.7095836400985718 + }, + { + "bits_per_byte": 1.6666838695114785, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.155257225036621, + "step": 3420, + "unclipped_grad_norm": 0.6851669549942017 + }, + { + "bits_per_byte": 1.702539672476611, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1801105737686157, + "step": 3430, + "unclipped_grad_norm": 0.7018667459487915 + }, + { + "bits_per_byte": 1.7037034790749999, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.1809172630310059, + "step": 3440, + "unclipped_grad_norm": 0.6927136778831482 + }, + { + "bits_per_byte": 1.7831694387673676, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.2359988689422607, + "step": 3450, + "unclipped_grad_norm": 0.7570765614509583 + }, + { + "bits_per_byte": 1.766653944805012, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.2245512008666992, + "step": 3460, + "unclipped_grad_norm": 0.7171066403388977 + }, + { + "bits_per_byte": 1.7307556621251161, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.1996684074401855, + "step": 3470, + "unclipped_grad_norm": 0.6849956512451172 + }, + { + "bits_per_byte": 1.694352782347589, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.1744358539581299, + "step": 3480, + "unclipped_grad_norm": 0.7314668893814087 + }, + { + "bits_per_byte": 1.6294754229973099, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.1294662952423096, + "step": 3490, + "unclipped_grad_norm": 0.7287701368331909 + }, + { + "bits_per_byte": 1.6418867229794754, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1380691528320312, + "step": 3500, + "unclipped_grad_norm": 0.6725860834121704 + }, + { + "bits_per_byte": 1.6915549685833526, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.1724965572357178, + "step": 3510, + "unclipped_grad_norm": 0.7451209425926208 + }, + { + "bits_per_byte": 1.6580145680467862, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.1492481231689453, + "step": 3520, + "unclipped_grad_norm": 0.7006548047065735 + }, + { + "bits_per_byte": 1.6737219155327372, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.1601356267929077, + "step": 3530, + "unclipped_grad_norm": 0.7069442272186279 + }, + { + "bits_per_byte": 1.685546238727508, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.1683316230773926, + "step": 3540, + "unclipped_grad_norm": 0.6971255540847778 + }, + { + "bits_per_byte": 1.7215852032158094, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.1933119297027588, + "step": 3550, + "unclipped_grad_norm": 0.7197291254997253 + }, + { + "bits_per_byte": 1.6612316755138257, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.1514780521392822, + "step": 3560, + "unclipped_grad_norm": 0.7382361888885498 + }, + { + "bits_per_byte": 1.7117773766024051, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.1865136623382568, + "step": 3570, + "unclipped_grad_norm": 0.7445778846740723 + }, + { + "bits_per_byte": 1.7688462076555602, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.226070761680603, + "step": 3580, + "unclipped_grad_norm": 0.7952014803886414 + }, + { + "bits_per_byte": 1.7005940327474003, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.1787619590759277, + "step": 3590, + "unclipped_grad_norm": 0.6882477402687073 + }, + { + "bits_per_byte": 1.6791687780682132, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.1639111042022705, + "step": 3600, + "unclipped_grad_norm": 0.6702791452407837 + }, + { + "bits_per_byte": 1.674588536110435, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.160736322402954, + "step": 3610, + "unclipped_grad_norm": 0.708904504776001 + }, + { + "bits_per_byte": 1.6718041369929661, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.158806324005127, + "step": 3620, + "unclipped_grad_norm": 0.7181576490402222 + }, + { + "bits_per_byte": 1.6791369612778035, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.1638890504837036, + "step": 3630, + "unclipped_grad_norm": 0.7148865461349487 + }, + { + "bits_per_byte": 1.646821249198031, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.1414895057678223, + "step": 3640, + "unclipped_grad_norm": 0.7093749642372131 + }, + { + "bits_per_byte": 1.7110602089483073, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.18601655960083, + "step": 3650, + "unclipped_grad_norm": 0.7810537219047546 + }, + { + "bits_per_byte": 1.622731983256979, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.1247920989990234, + "step": 3660, + "unclipped_grad_norm": 0.7055104970932007 + }, + { + "bits_per_byte": 1.6377288704122162, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.1351871490478516, + "step": 3670, + "unclipped_grad_norm": 0.6846752166748047 + }, + { + "bits_per_byte": 1.702971348930277, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.1804097890853882, + "step": 3680, + "unclipped_grad_norm": 0.7328780293464661 + }, + { + "bits_per_byte": 1.6621517826959418, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.152115821838379, + "step": 3690, + "unclipped_grad_norm": 0.6881498098373413 + }, + { + "bits_per_byte": 1.5981437956457742, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.1077488660812378, + "step": 3700, + "unclipped_grad_norm": 0.6903472542762756 + }, + { + "bits_per_byte": 1.6591137091684502, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.1500099897384644, + "step": 3710, + "unclipped_grad_norm": 0.7048502564430237 + }, + { + "bits_per_byte": 1.6694465988149387, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.1571722030639648, + "step": 3720, + "unclipped_grad_norm": 0.7675372958183289 + }, + { + "bits_per_byte": 1.7174380135729037, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.1904373168945312, + "step": 3730, + "unclipped_grad_norm": 0.7256482243537903 + }, + { + "bits_per_byte": 1.6449074262592296, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.1401629447937012, + "step": 3740, + "unclipped_grad_norm": 0.6591376662254333 + }, + { + "bits_per_byte": 1.627415414805275, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1280384063720703, + "step": 3750, + "unclipped_grad_norm": 0.6807434558868408 + }, + { + "bits_per_byte": 1.6393941784205206, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.1363414525985718, + "step": 3760, + "unclipped_grad_norm": 0.7151883244514465 + }, + { + "bits_per_byte": 1.6884372670885117, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.1703355312347412, + "step": 3770, + "unclipped_grad_norm": 0.7338826060295105 + }, + { + "bits_per_byte": 1.6452051282278732, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.1403692960739136, + "step": 3780, + "unclipped_grad_norm": 0.6906322240829468 + }, + { + "bits_per_byte": 1.6702163931602005, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.1577057838439941, + "step": 3790, + "unclipped_grad_norm": 0.6747716665267944 + }, + { + "bits_per_byte": 1.6265929937688492, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.1274683475494385, + "step": 3800, + "unclipped_grad_norm": 0.7121833562850952 + }, + { + "bits_per_byte": 1.6256699628816684, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.1268285512924194, + "step": 3810, + "unclipped_grad_norm": 0.6992501616477966 + }, + { + "bits_per_byte": 1.7193974119141837, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1917954683303833, + "step": 3820, + "unclipped_grad_norm": 0.7106893658638 + }, + { + "bits_per_byte": 1.6451141494055668, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.1403062343597412, + "step": 3830, + "unclipped_grad_norm": 0.7492547035217285 + }, + { + "bits_per_byte": 1.6404155833839948, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.1370494365692139, + "step": 3840, + "unclipped_grad_norm": 0.7060278058052063 + }, + { + "bits_per_byte": 1.670812657010742, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1581190824508667, + "step": 3850, + "unclipped_grad_norm": 0.7059534788131714 + }, + { + "bits_per_byte": 1.6163459234651414, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.1203656196594238, + "step": 3860, + "unclipped_grad_norm": 0.735011637210846 + }, + { + "bits_per_byte": 1.6744963534095725, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.1606724262237549, + "step": 3870, + "unclipped_grad_norm": 0.7251392602920532 + }, + { + "bits_per_byte": 1.6320964385964591, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1312830448150635, + "step": 3880, + "unclipped_grad_norm": 0.7253851890563965 + }, + { + "bits_per_byte": 1.6710525728086956, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.15828537940979, + "step": 3890, + "unclipped_grad_norm": 0.7244810461997986 + }, + { + "bits_per_byte": 1.6577510906256643, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.1490654945373535, + "step": 3900, + "unclipped_grad_norm": 0.7093306183815002 + }, + { + "bits_per_byte": 1.68089032440335, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1651043891906738, + "step": 3910, + "unclipped_grad_norm": 0.6779525279998779 + }, + { + "bits_per_byte": 1.6486009256691594, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.1427230834960938, + "step": 3920, + "unclipped_grad_norm": 0.6875896453857422 + }, + { + "bits_per_byte": 1.6241145517872653, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.1257504224777222, + "step": 3930, + "unclipped_grad_norm": 0.7081995010375977 + }, + { + "bits_per_byte": 1.6312575072255504, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.1307015419006348, + "step": 3940, + "unclipped_grad_norm": 0.7231290936470032 + }, + { + "bits_per_byte": 1.7042273382295279, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1812803745269775, + "step": 3950, + "unclipped_grad_norm": 0.7233274579048157 + }, + { + "bits_per_byte": 1.708786598303901, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.1844406127929688, + "step": 3960, + "unclipped_grad_norm": 0.7738246917724609 + }, + { + "bits_per_byte": 1.6828430154212464, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1664578914642334, + "step": 3970, + "unclipped_grad_norm": 0.6724381446838379 + }, + { + "bits_per_byte": 1.6766316900026852, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.1621525287628174, + "step": 3980, + "unclipped_grad_norm": 0.7247830033302307 + }, + { + "bits_per_byte": 1.6937418999717242, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1740124225616455, + "step": 3990, + "unclipped_grad_norm": 0.6913743615150452 + }, + { + "bits_per_byte": 1.6939156024490956, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.1741328239440918, + "step": 4000, + "unclipped_grad_norm": 0.7157939076423645 + }, + { + "bits_per_byte": 1.7096000762424821, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.185004472732544, + "step": 4010, + "unclipped_grad_norm": 0.7507659792900085 + }, + { + "bits_per_byte": 1.579852924778562, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.0950706005096436, + "step": 4020, + "unclipped_grad_norm": 0.6602710485458374 + }, + { + "bits_per_byte": 1.6214957719625773, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.1239352226257324, + "step": 4030, + "unclipped_grad_norm": 0.7207183241844177 + }, + { + "bits_per_byte": 1.6331384814780368, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1320053339004517, + "step": 4040, + "unclipped_grad_norm": 0.7153538465499878 + }, + { + "bits_per_byte": 1.6100788756677007, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.1160216331481934, + "step": 4050, + "unclipped_grad_norm": 0.7091026306152344 + }, + { + "bits_per_byte": 1.648860963437264, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1429033279418945, + "step": 4060, + "unclipped_grad_norm": 0.7409337162971497 + }, + { + "bits_per_byte": 1.6165222056822757, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.1204878091812134, + "step": 4070, + "unclipped_grad_norm": 0.7250375747680664 + }, + { + "bits_per_byte": 1.6476598366036381, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.1420707702636719, + "step": 4080, + "unclipped_grad_norm": 0.7233843207359314 + }, + { + "bits_per_byte": 1.6078716503265273, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.1144917011260986, + "step": 4090, + "unclipped_grad_norm": 0.7093282341957092 + }, + { + "bits_per_byte": 1.5779012656565707, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.0937178134918213, + "step": 4100, + "unclipped_grad_norm": 0.6735255122184753 + }, + { + "bits_per_byte": 1.6798283315342721, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.1643682718276978, + "step": 4110, + "unclipped_grad_norm": 0.733401894569397 + }, + { + "bits_per_byte": 1.6327946881589621, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.1317670345306396, + "step": 4120, + "unclipped_grad_norm": 0.6985018849372864 + }, + { + "bits_per_byte": 1.593084582005343, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.1042420864105225, + "step": 4130, + "unclipped_grad_norm": 0.6837909817695618 + }, + { + "bits_per_byte": 1.617565968390362, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.121211290359497, + "step": 4140, + "unclipped_grad_norm": 0.7155304551124573 + }, + { + "bits_per_byte": 1.6561760735090627, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1479737758636475, + "step": 4150, + "unclipped_grad_norm": 0.7386810779571533 + }, + { + "bits_per_byte": 1.6284524701899779, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.1287572383880615, + "step": 4160, + "unclipped_grad_norm": 0.7266976237297058 + }, + { + "bits_per_byte": 1.5764414769160473, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.0927059650421143, + "step": 4170, + "unclipped_grad_norm": 0.7132578492164612 + }, + { + "bits_per_byte": 1.5700554171242096, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.0882794857025146, + "step": 4180, + "unclipped_grad_norm": 0.7138552665710449 + }, + { + "bits_per_byte": 1.7181954251673035, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.190962314605713, + "step": 4190, + "unclipped_grad_norm": 0.7419083118438721 + }, + { + "bits_per_byte": 1.6071365964767397, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1139822006225586, + "step": 4200, + "unclipped_grad_norm": 0.6905852556228638 + }, + { + "bits_per_byte": 1.6148593054310831, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.1193351745605469, + "step": 4210, + "unclipped_grad_norm": 0.7368191480636597 + }, + { + "bits_per_byte": 1.6559972115521653, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.1478497982025146, + "step": 4220, + "unclipped_grad_norm": 0.743370532989502 + }, + { + "bits_per_byte": 1.6298150887327638, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.1297017335891724, + "step": 4230, + "unclipped_grad_norm": 0.685749888420105 + }, + { + "bits_per_byte": 1.6664630437877708, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.155104160308838, + "step": 4240, + "unclipped_grad_norm": 0.7345937490463257 + }, + { + "bits_per_byte": 1.6900939759642732, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1714838743209839, + "step": 4250, + "unclipped_grad_norm": 0.703519880771637 + }, + { + "bits_per_byte": 1.670806465635311, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1581147909164429, + "step": 4260, + "unclipped_grad_norm": 0.748995304107666 + }, + { + "bits_per_byte": 1.6504036478155033, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.143972635269165, + "step": 4270, + "unclipped_grad_norm": 0.7237050533294678 + }, + { + "bits_per_byte": 1.6818689056867593, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1657826900482178, + "step": 4280, + "unclipped_grad_norm": 0.7543016076087952 + }, + { + "bits_per_byte": 1.6377570755669575, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.135206699371338, + "step": 4290, + "unclipped_grad_norm": 0.7121536135673523 + }, + { + "bits_per_byte": 1.5767508737049496, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.0929204225540161, + "step": 4300, + "unclipped_grad_norm": 0.7414191961288452 + }, + { + "bits_per_byte": 1.6990393095836005, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.1776843070983887, + "step": 4310, + "unclipped_grad_norm": 0.80052250623703 + }, + { + "bits_per_byte": 1.5806983914902035, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.0956566333770752, + "step": 4320, + "unclipped_grad_norm": 0.7040880918502808 + }, + { + "bits_per_byte": 1.5868787600316094, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.099940538406372, + "step": 4330, + "unclipped_grad_norm": 0.7220457196235657 + }, + { + "bits_per_byte": 1.5836730034195268, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.0977184772491455, + "step": 4340, + "unclipped_grad_norm": 0.717460036277771 + }, + { + "bits_per_byte": 1.5059991029491495, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.0438790321350098, + "step": 4350, + "unclipped_grad_norm": 0.7002816200256348 + }, + { + "bits_per_byte": 1.6642601180128689, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.153577208518982, + "step": 4360, + "unclipped_grad_norm": 0.7683370113372803 + }, + { + "bits_per_byte": 1.6222497439039598, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.124457836151123, + "step": 4370, + "unclipped_grad_norm": 0.7472379803657532 + }, + { + "bits_per_byte": 1.5798536127091654, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.0950710773468018, + "step": 4380, + "unclipped_grad_norm": 0.7642439007759094 + }, + { + "bits_per_byte": 1.6207043078033065, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1233866214752197, + "step": 4390, + "unclipped_grad_norm": 0.7720315456390381 + }, + { + "bits_per_byte": 1.59053855084197, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.1024773120880127, + "step": 4400, + "unclipped_grad_norm": 0.7390428185462952 + }, + { + "bits_per_byte": 1.5823848533645641, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.0968255996704102, + "step": 4410, + "unclipped_grad_norm": 0.7486209273338318 + }, + { + "bits_per_byte": 1.5744389109294008, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.091317892074585, + "step": 4420, + "unclipped_grad_norm": 0.7394227981567383 + }, + { + "bits_per_byte": 1.5744903337420086, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.0913535356521606, + "step": 4430, + "unclipped_grad_norm": 0.7019861936569214 + }, + { + "bits_per_byte": 1.591644399287018, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.1032438278198242, + "step": 4440, + "unclipped_grad_norm": 0.7432880401611328 + }, + { + "bits_per_byte": 1.5914445554467154, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.1031053066253662, + "step": 4450, + "unclipped_grad_norm": 0.7175650000572205 + }, + { + "bits_per_byte": 1.5483157781592454, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.0732107162475586, + "step": 4460, + "unclipped_grad_norm": 0.6954054832458496 + }, + { + "bits_per_byte": 1.650105601881558, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1437660455703735, + "step": 4470, + "unclipped_grad_norm": 0.7265033721923828 + }, + { + "bits_per_byte": 1.5840228161313816, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.0979609489440918, + "step": 4480, + "unclipped_grad_norm": 0.7616819143295288 + }, + { + "bits_per_byte": 1.5191843248601975, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.05301833152771, + "step": 4490, + "unclipped_grad_norm": 0.7025213837623596 + }, + { + "bits_per_byte": 1.6158881056485446, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.1200482845306396, + "step": 4500, + "unclipped_grad_norm": 0.7285298705101013 + }, + { + "bits_per_byte": 1.5999570087338209, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.1090056896209717, + "step": 4510, + "unclipped_grad_norm": 0.7420633435249329 + }, + { + "bits_per_byte": 1.5864930029457243, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.0996731519699097, + "step": 4520, + "unclipped_grad_norm": 0.7540821433067322 + }, + { + "bits_per_byte": 1.598844968913342, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.1082348823547363, + "step": 4530, + "unclipped_grad_norm": 0.722756564617157 + }, + { + "bits_per_byte": 1.556158531003891, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.0786468982696533, + "step": 4540, + "unclipped_grad_norm": 0.7224688529968262 + }, + { + "bits_per_byte": 1.6224736653153833, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.1246130466461182, + "step": 4550, + "unclipped_grad_norm": 0.7299609780311584 + }, + { + "bits_per_byte": 1.6260385217024675, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.1270840167999268, + "step": 4560, + "unclipped_grad_norm": 0.7680655717849731 + }, + { + "bits_per_byte": 1.582489246833638, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.0968979597091675, + "step": 4570, + "unclipped_grad_norm": 0.7242801189422607 + }, + { + "bits_per_byte": 1.5466220930135484, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.0720367431640625, + "step": 4580, + "unclipped_grad_norm": 0.7660850882530212 + }, + { + "bits_per_byte": 1.6632400889106016, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.1528701782226562, + "step": 4590, + "unclipped_grad_norm": 0.7880666255950928 + }, + { + "bits_per_byte": 1.6104257646744908, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.1162620782852173, + "step": 4600, + "unclipped_grad_norm": 0.7475829124450684 + }, + { + "bits_per_byte": 1.5999250199607604, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.1089835166931152, + "step": 4610, + "unclipped_grad_norm": 0.7532138824462891 + }, + { + "bits_per_byte": 1.5277858651604501, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.0589804649353027, + "step": 4620, + "unclipped_grad_norm": 0.6750307679176331 + }, + { + "bits_per_byte": 1.6102266087647918, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.1161240339279175, + "step": 4630, + "unclipped_grad_norm": 0.7599968910217285 + }, + { + "bits_per_byte": 1.5795020801708017, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.09482741355896, + "step": 4640, + "unclipped_grad_norm": 0.7219563126564026 + }, + { + "bits_per_byte": 1.544038913597589, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.0702462196350098, + "step": 4650, + "unclipped_grad_norm": 0.7356395125389099 + }, + { + "bits_per_byte": 1.5804662149115387, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.0954957008361816, + "step": 4660, + "unclipped_grad_norm": 0.7338905930519104 + }, + { + "bits_per_byte": 1.6047737268365356, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.1123443841934204, + "step": 4670, + "unclipped_grad_norm": 0.75016188621521 + }, + { + "bits_per_byte": 1.5709146424479201, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.0888750553131104, + "step": 4680, + "unclipped_grad_norm": 0.7285161018371582 + }, + { + "bits_per_byte": 1.5393572018758015, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.0670011043548584, + "step": 4690, + "unclipped_grad_norm": 0.7614326477050781 + }, + { + "bits_per_byte": 1.6445514221719437, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.139916181564331, + "step": 4700, + "unclipped_grad_norm": 0.7450862526893616 + }, + { + "bits_per_byte": 1.5630474680668518, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.0834219455718994, + "step": 4710, + "unclipped_grad_norm": 0.7619648575782776 + }, + { + "bits_per_byte": 1.524771009290825, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.0568907260894775, + "step": 4720, + "unclipped_grad_norm": 0.7072386741638184 + }, + { + "bits_per_byte": 1.6629296602257944, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.152655005455017, + "step": 4730, + "unclipped_grad_norm": 0.7649170160293579 + }, + { + "bits_per_byte": 1.594043385283903, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.104906678199768, + "step": 4740, + "unclipped_grad_norm": 0.721769392490387 + }, + { + "bits_per_byte": 1.632265841507559, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.131400465965271, + "step": 4750, + "unclipped_grad_norm": 0.7643818259239197 + }, + { + "bits_per_byte": 1.514971781809964, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.0500984191894531, + "step": 4760, + "unclipped_grad_norm": 0.7153230905532837 + }, + { + "bits_per_byte": 1.6084302499765297, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.1148788928985596, + "step": 4770, + "unclipped_grad_norm": 0.7273650169372559 + }, + { + "bits_per_byte": 1.5650889021325933, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.0848369598388672, + "step": 4780, + "unclipped_grad_norm": 0.717960000038147 + }, + { + "bits_per_byte": 1.601536497399345, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.110100507736206, + "step": 4790, + "unclipped_grad_norm": 0.7582246661186218 + }, + { + "bits_per_byte": 1.5698084500375704, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.0881083011627197, + "step": 4800, + "unclipped_grad_norm": 0.7384725213050842 + }, + { + "bits_per_byte": 1.5311065061833091, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.0612821578979492, + "step": 4810, + "unclipped_grad_norm": 0.7566121816635132 + }, + { + "bits_per_byte": 1.6115369445817156, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.1170322895050049, + "step": 4820, + "unclipped_grad_norm": 0.8017722964286804 + }, + { + "bits_per_byte": 1.5892017296968133, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.1015506982803345, + "step": 4830, + "unclipped_grad_norm": 0.7956211566925049 + }, + { + "bits_per_byte": 1.6100489506864504, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.1160008907318115, + "step": 4840, + "unclipped_grad_norm": 0.7549191117286682 + }, + { + "bits_per_byte": 1.534060136229227, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.0633294582366943, + "step": 4850, + "unclipped_grad_norm": 0.7463700175285339 + }, + { + "bits_per_byte": 1.555477995654427, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.0781751871109009, + "step": 4860, + "unclipped_grad_norm": 0.7801370620727539 + }, + { + "bits_per_byte": 1.5639209679505839, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.0840274095535278, + "step": 4870, + "unclipped_grad_norm": 0.7722324132919312 + }, + { + "bits_per_byte": 1.7060075306486089, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.1825143098831177, + "step": 4880, + "unclipped_grad_norm": 0.8102167248725891 + }, + { + "bits_per_byte": 1.5002206578628101, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.039873719215393, + "step": 4890, + "unclipped_grad_norm": 0.7102228999137878 + }, + { + "bits_per_byte": 1.5950303937172048, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.1055908203125, + "step": 4900, + "unclipped_grad_norm": 0.7528427839279175 + }, + { + "bits_per_byte": 1.5771070497748865, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.0931673049926758, + "step": 4910, + "unclipped_grad_norm": 0.7582761645317078 + }, + { + "bits_per_byte": 1.511712194628161, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.0478390455245972, + "step": 4920, + "unclipped_grad_norm": 0.7099639773368835 + }, + { + "bits_per_byte": 1.5437035474284064, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.0700137615203857, + "step": 4930, + "unclipped_grad_norm": 0.7167920470237732 + }, + { + "bits_per_byte": 1.5536687381673502, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.0769211053848267, + "step": 4940, + "unclipped_grad_norm": 0.7444012761116028 + }, + { + "bits_per_byte": 1.5773209961925596, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.093315601348877, + "step": 4950, + "unclipped_grad_norm": 0.7740675210952759 + }, + { + "bits_per_byte": 1.5945730918485606, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.1052738428115845, + "step": 4960, + "unclipped_grad_norm": 0.7550216317176819 + }, + { + "bits_per_byte": 1.5635383065524142, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.0837621688842773, + "step": 4970, + "unclipped_grad_norm": 0.7642179727554321 + }, + { + "bits_per_byte": 1.6121729364246062, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.1174731254577637, + "step": 4980, + "unclipped_grad_norm": 0.7683852314949036 + }, + { + "bits_per_byte": 1.5603712460367756, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.0815669298171997, + "step": 4990, + "unclipped_grad_norm": 0.735556423664093 + }, + { + "bits_per_byte": 1.5757466670065616, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.092224359512329, + "step": 5000, + "unclipped_grad_norm": 0.7441115379333496 + }, + { + "bits_per_byte": 1.6233487130429731, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.1252195835113525, + "step": 5010, + "unclipped_grad_norm": 0.7864952683448792 + }, + { + "bits_per_byte": 1.586583637802729, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.099735975265503, + "step": 5020, + "unclipped_grad_norm": 0.7992230653762817 + }, + { + "bits_per_byte": 1.5164520364859402, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.0511244535446167, + "step": 5030, + "unclipped_grad_norm": 0.7306111454963684 + }, + { + "bits_per_byte": 1.577993792322735, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.0937819480895996, + "step": 5040, + "unclipped_grad_norm": 0.7398955821990967 + }, + { + "bits_per_byte": 1.5536202390598068, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.0768874883651733, + "step": 5050, + "unclipped_grad_norm": 0.761589765548706 + }, + { + "bits_per_byte": 1.6107637105834363, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.1164963245391846, + "step": 5060, + "unclipped_grad_norm": 0.7791648507118225 + }, + { + "bits_per_byte": 1.5549512127948342, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.0778100490570068, + "step": 5070, + "unclipped_grad_norm": 0.768591046333313 + }, + { + "bits_per_byte": 1.616881477439928, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.120736837387085, + "step": 5080, + "unclipped_grad_norm": 0.8012626767158508 + }, + { + "bits_per_byte": 1.5595562202543367, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.0810019969940186, + "step": 5090, + "unclipped_grad_norm": 0.735996425151825 + }, + { + "bits_per_byte": 1.5482942803278874, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.0731958150863647, + "step": 5100, + "unclipped_grad_norm": 0.8076115250587463 + }, + { + "bits_per_byte": 1.5852843088754602, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.0988353490829468, + "step": 5110, + "unclipped_grad_norm": 0.7947574257850647 + }, + { + "bits_per_byte": 1.4884489613592033, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0317142009735107, + "step": 5120, + "unclipped_grad_norm": 0.7292161583900452 + }, + { + "bits_per_byte": 1.6172976754550161, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.1210253238677979, + "step": 5130, + "unclipped_grad_norm": 0.7981391549110413 + }, + { + "bits_per_byte": 1.5992291781553694, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.1085011959075928, + "step": 5140, + "unclipped_grad_norm": 0.7671464681625366 + }, + { + "bits_per_byte": 1.534920393448843, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.0639257431030273, + "step": 5150, + "unclipped_grad_norm": 0.7683079242706299 + }, + { + "bits_per_byte": 1.5510136700033301, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.0750807523727417, + "step": 5160, + "unclipped_grad_norm": 0.7727853059768677 + }, + { + "bits_per_byte": 1.622798368560212, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.12483811378479, + "step": 5170, + "unclipped_grad_norm": 0.8305050730705261 + }, + { + "bits_per_byte": 1.5594497629934527, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.0809282064437866, + "step": 5180, + "unclipped_grad_norm": 0.7719935774803162 + }, + { + "bits_per_byte": 1.474968101271322, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.0223699808120728, + "step": 5190, + "unclipped_grad_norm": 0.7704627513885498 + }, + { + "bits_per_byte": 1.5402169431474648, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.0675970315933228, + "step": 5200, + "unclipped_grad_norm": 0.7530068755149841 + }, + { + "bits_per_byte": 1.5204955205903756, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.0539271831512451, + "step": 5210, + "unclipped_grad_norm": 0.7825141549110413 + }, + { + "bits_per_byte": 1.533782040282782, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.0631366968154907, + "step": 5220, + "unclipped_grad_norm": 0.7376784086227417 + }, + { + "bits_per_byte": 1.6072934446543266, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.114090919494629, + "step": 5230, + "unclipped_grad_norm": 0.800700843334198 + }, + { + "bits_per_byte": 1.5818967666014156, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.096487283706665, + "step": 5240, + "unclipped_grad_norm": 0.8445186018943787 + }, + { + "bits_per_byte": 1.573877043609032, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0909284353256226, + "step": 5250, + "unclipped_grad_norm": 0.8148460388183594 + }, + { + "bits_per_byte": 1.5025164542691776, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.0414650440216064, + "step": 5260, + "unclipped_grad_norm": 0.7522324919700623 + }, + { + "bits_per_byte": 1.5603046887508918, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.0815207958221436, + "step": 5270, + "unclipped_grad_norm": 0.7337029576301575 + }, + { + "bits_per_byte": 1.4910900989285034, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.033544898033142, + "step": 5280, + "unclipped_grad_norm": 0.7956797480583191 + }, + { + "bits_per_byte": 1.511882973400468, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.047957420349121, + "step": 5290, + "unclipped_grad_norm": 0.78285151720047 + }, + { + "bits_per_byte": 1.5036622026892257, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.0422592163085938, + "step": 5300, + "unclipped_grad_norm": 0.7855217456817627 + }, + { + "bits_per_byte": 1.4939862867690332, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.0355523824691772, + "step": 5310, + "unclipped_grad_norm": 0.7658283710479736 + }, + { + "bits_per_byte": 1.516399581777427, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0510880947113037, + "step": 5320, + "unclipped_grad_norm": 0.7995138168334961 + }, + { + "bits_per_byte": 1.5621627893108134, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.0828087329864502, + "step": 5330, + "unclipped_grad_norm": 0.8070236444473267 + }, + { + "bits_per_byte": 1.612768684327195, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.1178860664367676, + "step": 5340, + "unclipped_grad_norm": 0.8107616305351257 + }, + { + "bits_per_byte": 1.4848815252323562, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.0292414426803589, + "step": 5350, + "unclipped_grad_norm": 0.7562969923019409 + }, + { + "bits_per_byte": 1.5092863793377413, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.0461575984954834, + "step": 5360, + "unclipped_grad_norm": 0.7855019569396973 + }, + { + "bits_per_byte": 1.5929220584002777, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.104129433631897, + "step": 5370, + "unclipped_grad_norm": 0.8431105017662048 + }, + { + "bits_per_byte": 1.5151238144733268, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.050203800201416, + "step": 5380, + "unclipped_grad_norm": 0.7589862942695618 + }, + { + "bits_per_byte": 1.5062727273466723, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.044068694114685, + "step": 5390, + "unclipped_grad_norm": 0.782349169254303 + }, + { + "bits_per_byte": 1.555041675669188, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.0778727531433105, + "step": 5400, + "unclipped_grad_norm": 0.8342399001121521 + }, + { + "bits_per_byte": 1.4638716086550028, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.0146784782409668, + "step": 5410, + "unclipped_grad_norm": 0.7854043245315552 + }, + { + "bits_per_byte": 1.600609854876496, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.1094582080841064, + "step": 5420, + "unclipped_grad_norm": 0.8390663862228394 + }, + { + "bits_per_byte": 1.5396103603378717, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.0671765804290771, + "step": 5430, + "unclipped_grad_norm": 0.7932195663452148 + }, + { + "bits_per_byte": 1.5411712748771025, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.06825852394104, + "step": 5440, + "unclipped_grad_norm": 0.7833191752433777 + }, + { + "bits_per_byte": 1.555050618767033, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.0778789520263672, + "step": 5450, + "unclipped_grad_norm": 0.8194807171821594 + }, + { + "bits_per_byte": 1.5671931098658998, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.086295485496521, + "step": 5460, + "unclipped_grad_norm": 0.7595216631889343 + }, + { + "bits_per_byte": 1.5390588119765547, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0667942762374878, + "step": 5470, + "unclipped_grad_norm": 0.7744208574295044 + }, + { + "bits_per_byte": 1.4761611449203573, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.0231969356536865, + "step": 5480, + "unclipped_grad_norm": 0.7626774311065674 + }, + { + "bits_per_byte": 1.5463238750969523, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.0718300342559814, + "step": 5490, + "unclipped_grad_norm": 0.8239845633506775 + }, + { + "bits_per_byte": 1.560069760449813, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.0813579559326172, + "step": 5500, + "unclipped_grad_norm": 0.8127777576446533 + }, + { + "bits_per_byte": 1.5553546840937584, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.078089714050293, + "step": 5510, + "unclipped_grad_norm": 0.827265739440918 + }, + { + "bits_per_byte": 1.5442236229646156, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.0703742504119873, + "step": 5520, + "unclipped_grad_norm": 0.8176153302192688 + }, + { + "bits_per_byte": 1.5507465809465402, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.0748956203460693, + "step": 5530, + "unclipped_grad_norm": 0.8071540594100952 + }, + { + "bits_per_byte": 1.588215581176766, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.1008671522140503, + "step": 5540, + "unclipped_grad_norm": 0.8446843028068542 + }, + { + "bits_per_byte": 1.523353184317112, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.055907964706421, + "step": 5550, + "unclipped_grad_norm": 0.7665615677833557 + }, + { + "bits_per_byte": 1.5270532190677746, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.0584726333618164, + "step": 5560, + "unclipped_grad_norm": 0.7955387830734253 + }, + { + "bits_per_byte": 1.5780457310832956, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.093817949295044, + "step": 5570, + "unclipped_grad_norm": 0.8932857513427734 + }, + { + "bits_per_byte": 1.4604078780666259, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.012277603149414, + "step": 5580, + "unclipped_grad_norm": 0.7716569304466248 + }, + { + "bits_per_byte": 1.5339301173451747, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.063239336013794, + "step": 5590, + "unclipped_grad_norm": 0.8064618110656738 + }, + { + "bits_per_byte": 1.5199751010888647, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.0535664558410645, + "step": 5600, + "unclipped_grad_norm": 0.8003355860710144 + }, + { + "bits_per_byte": 1.531945265571567, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.0618635416030884, + "step": 5610, + "unclipped_grad_norm": 0.7417186498641968 + }, + { + "bits_per_byte": 1.4807124937927907, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.0263516902923584, + "step": 5620, + "unclipped_grad_norm": 0.7973346710205078 + }, + { + "bits_per_byte": 1.483432571398837, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0282371044158936, + "step": 5630, + "unclipped_grad_norm": 0.8100676536560059 + }, + { + "bits_per_byte": 1.4630239061689, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.014090895652771, + "step": 5640, + "unclipped_grad_norm": 0.7916391491889954 + }, + { + "bits_per_byte": 1.5855539776720131, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.0990222692489624, + "step": 5650, + "unclipped_grad_norm": 0.8808712959289551 + }, + { + "bits_per_byte": 1.529998765929102, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.0605143308639526, + "step": 5660, + "unclipped_grad_norm": 0.7903205156326294 + }, + { + "bits_per_byte": 1.5227873613957734, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.0555157661437988, + "step": 5670, + "unclipped_grad_norm": 0.7989382743835449 + }, + { + "bits_per_byte": 1.5397668645501568, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0672850608825684, + "step": 5680, + "unclipped_grad_norm": 0.7952145338058472 + }, + { + "bits_per_byte": 1.5636772685343114, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.0838584899902344, + "step": 5690, + "unclipped_grad_norm": 0.8005351424217224 + }, + { + "bits_per_byte": 1.5056737117737171, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.0436534881591797, + "step": 5700, + "unclipped_grad_norm": 0.7949317693710327 + }, + { + "bits_per_byte": 1.5501639037254171, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.0744917392730713, + "step": 5710, + "unclipped_grad_norm": 0.8340266346931458 + }, + { + "bits_per_byte": 1.5761724960500978, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0925195217132568, + "step": 5720, + "unclipped_grad_norm": 0.866409182548523 + }, + { + "bits_per_byte": 1.4710087166831585, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.0196255445480347, + "step": 5730, + "unclipped_grad_norm": 0.7789786458015442 + }, + { + "bits_per_byte": 1.5530638751842658, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.0765018463134766, + "step": 5740, + "unclipped_grad_norm": 0.8428030610084534 + }, + { + "bits_per_byte": 1.5069880031916107, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.0445644855499268, + "step": 5750, + "unclipped_grad_norm": 0.8580909967422485 + }, + { + "bits_per_byte": 1.4955731706885442, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.0366523265838623, + "step": 5760, + "unclipped_grad_norm": 0.846638560295105 + }, + { + "bits_per_byte": 1.4561774488207024, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.009345293045044, + "step": 5770, + "unclipped_grad_norm": 0.7746208310127258 + }, + { + "bits_per_byte": 1.5437431034381048, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.0700411796569824, + "step": 5780, + "unclipped_grad_norm": 0.7821170687675476 + }, + { + "bits_per_byte": 1.4957042214685017, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 1.0367431640625, + "step": 5790, + "unclipped_grad_norm": 0.7897992730140686 + }, + { + "bits_per_byte": 1.4886228358192257, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.0318347215652466, + "step": 5800, + "unclipped_grad_norm": 0.8172131180763245 + }, + { + "bits_per_byte": 1.5320221418165028, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.0619168281555176, + "step": 5810, + "unclipped_grad_norm": 0.8061510920524597 + }, + { + "bits_per_byte": 1.521096943920443, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.0543440580368042, + "step": 5820, + "unclipped_grad_norm": 0.8244926333427429 + }, + { + "bits_per_byte": 1.4864051195363497, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.0302975177764893, + "step": 5830, + "unclipped_grad_norm": 0.8219233751296997 + }, + { + "bits_per_byte": 1.4737029968915754, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.0214930772781372, + "step": 5840, + "unclipped_grad_norm": 0.8423903584480286 + }, + { + "bits_per_byte": 1.4586333610750233, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.011047601699829, + "step": 5850, + "unclipped_grad_norm": 0.7926213145256042 + }, + { + "bits_per_byte": 1.4926749190562043, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.0346434116363525, + "step": 5860, + "unclipped_grad_norm": 0.872589647769928 + }, + { + "bits_per_byte": 1.5390300908738606, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0667743682861328, + "step": 5870, + "unclipped_grad_norm": 0.8347393274307251 + }, + { + "bits_per_byte": 1.5461890406986758, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.0717365741729736, + "step": 5880, + "unclipped_grad_norm": 0.8236801624298096 + }, + { + "bits_per_byte": 1.5302139162253312, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.0606634616851807, + "step": 5890, + "unclipped_grad_norm": 0.8220352530479431 + }, + { + "bits_per_byte": 1.5066418021154238, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.044324517250061, + "step": 5900, + "unclipped_grad_norm": 0.8733406662940979 + }, + { + "bits_per_byte": 1.5066249478156393, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.044312834739685, + "step": 5910, + "unclipped_grad_norm": 0.886996328830719 + }, + { + "bits_per_byte": 1.5011601990844736, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.040524959564209, + "step": 5920, + "unclipped_grad_norm": 0.8211999535560608 + }, + { + "bits_per_byte": 1.557414692285793, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.0795176029205322, + "step": 5930, + "unclipped_grad_norm": 0.8763678073883057 + }, + { + "bits_per_byte": 1.5382152370740725, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.0662095546722412, + "step": 5940, + "unclipped_grad_norm": 0.8679670095443726 + }, + { + "bits_per_byte": 1.5161003319649258, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.0508806705474854, + "step": 5950, + "unclipped_grad_norm": 0.8319846391677856 + }, + { + "bits_per_byte": 1.5077700083050838, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.0451065301895142, + "step": 5960, + "unclipped_grad_norm": 0.8072320222854614 + }, + { + "bits_per_byte": 1.5292109134054994, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.0599682331085205, + "step": 5970, + "unclipped_grad_norm": 0.8743261694908142 + }, + { + "bits_per_byte": 1.484084385645607, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.028688907623291, + "step": 5980, + "unclipped_grad_norm": 0.8455321192741394 + }, + { + "bits_per_byte": 1.450148253029405, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0051661729812622, + "step": 5990, + "unclipped_grad_norm": 0.8320293426513672 + }, + { + "bits_per_byte": 1.520576696401583, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.053983449935913, + "step": 6000, + "unclipped_grad_norm": 0.8345706462860107 + }, + { + "bits_per_byte": 1.5028198316652996, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.041675329208374, + "step": 6010, + "unclipped_grad_norm": 0.8575472235679626 + }, + { + "bits_per_byte": 1.4869836691738523, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.030698537826538, + "step": 6020, + "unclipped_grad_norm": 0.823765218257904 + }, + { + "bits_per_byte": 1.5394364858778493, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.0670560598373413, + "step": 6030, + "unclipped_grad_norm": 0.8210551142692566 + }, + { + "bits_per_byte": 1.4928266077542653, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.0347485542297363, + "step": 6040, + "unclipped_grad_norm": 0.8822175860404968 + }, + { + "bits_per_byte": 1.5083255122673707, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.0454915761947632, + "step": 6050, + "unclipped_grad_norm": 0.8693289160728455 + }, + { + "bits_per_byte": 1.5452926671223788, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.071115255355835, + "step": 6060, + "unclipped_grad_norm": 0.8629904389381409 + }, + { + "bits_per_byte": 1.5035423307815743, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 1.0421761274337769, + "step": 6070, + "unclipped_grad_norm": 0.8153941631317139 + }, + { + "bits_per_byte": 1.4807776752174677, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.0263968706130981, + "step": 6080, + "unclipped_grad_norm": 0.8348871469497681 + }, + { + "bits_per_byte": 1.495174858869146, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.0363762378692627, + "step": 6090, + "unclipped_grad_norm": 0.8315237760543823 + }, + { + "bits_per_byte": 1.492264052503293, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.0343586206436157, + "step": 6100, + "unclipped_grad_norm": 0.8456729650497437 + }, + { + "bits_per_byte": 1.4632765486830175, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.014266014099121, + "step": 6110, + "unclipped_grad_norm": 0.8281521797180176 + }, + { + "bits_per_byte": 1.4707294168581573, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0194319486618042, + "step": 6120, + "unclipped_grad_norm": 0.8127527832984924 + }, + { + "bits_per_byte": 1.5076700863849326, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.0450372695922852, + "step": 6130, + "unclipped_grad_norm": 0.8982784152030945 + }, + { + "bits_per_byte": 1.5358587307919505, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.0645761489868164, + "step": 6140, + "unclipped_grad_norm": 0.8576988577842712 + }, + { + "bits_per_byte": 1.462926048040559, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.0140230655670166, + "step": 6150, + "unclipped_grad_norm": 0.8157727718353271 + }, + { + "bits_per_byte": 1.5647535359634108, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 1.0846045017242432, + "step": 6160, + "unclipped_grad_norm": 0.8942285180091858 + }, + { + "bits_per_byte": 1.429317198408949, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 0.9907271862030029, + "step": 6170, + "unclipped_grad_norm": 0.8000110387802124 + }, + { + "bits_per_byte": 1.4986380735095703, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0387767553329468, + "step": 6180, + "unclipped_grad_norm": 0.8287771940231323 + }, + { + "bits_per_byte": 1.5022734427835085, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.0412966012954712, + "step": 6190, + "unclipped_grad_norm": 0.8627387881278992 + }, + { + "bits_per_byte": 1.462083677016633, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.0134391784667969, + "step": 6200, + "unclipped_grad_norm": 0.8199469447135925 + }, + { + "bits_per_byte": 1.4800539722226371, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.0258952379226685, + "step": 6210, + "unclipped_grad_norm": 0.8438522219657898 + }, + { + "bits_per_byte": 1.4620126481818267, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0133899450302124, + "step": 6220, + "unclipped_grad_norm": 0.8124659657478333 + }, + { + "bits_per_byte": 1.4590605659797666, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.0113437175750732, + "step": 6230, + "unclipped_grad_norm": 0.8244684338569641 + }, + { + "bits_per_byte": 1.4679795862535119, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.0175259113311768, + "step": 6240, + "unclipped_grad_norm": 0.8646737933158875 + }, + { + "bits_per_byte": 1.4557873921685456, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.0090749263763428, + "step": 6250, + "unclipped_grad_norm": 0.8324311971664429 + }, + { + "bits_per_byte": 1.5158126049900322, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 1.050681233406067, + "step": 6260, + "unclipped_grad_norm": 0.9003724455833435 + }, + { + "bits_per_byte": 1.4846065249736264, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.0290508270263672, + "step": 6270, + "unclipped_grad_norm": 0.8243275284767151 + }, + { + "bits_per_byte": 1.5953544090314304, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.1058154106140137, + "step": 6280, + "unclipped_grad_norm": 0.9166515469551086 + }, + { + "bits_per_byte": 1.4653227982629833, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 1.0156843662261963, + "step": 6290, + "unclipped_grad_norm": 0.8469663858413696 + }, + { + "bits_per_byte": 1.4947307996646182, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.0360684394836426, + "step": 6300, + "unclipped_grad_norm": 0.8578129410743713 + }, + { + "bits_per_byte": 1.5386338428462727, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.0664997100830078, + "step": 6310, + "unclipped_grad_norm": 0.9026638865470886 + }, + { + "bits_per_byte": 1.4953267195498579, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 1.036481499671936, + "step": 6320, + "unclipped_grad_norm": 0.8539533019065857 + }, + { + "bits_per_byte": 1.446193168007513, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 1.002424716949463, + "step": 6330, + "unclipped_grad_norm": 0.8302637338638306 + }, + { + "bits_per_byte": 1.513790604963838, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 1.0492796897888184, + "step": 6340, + "unclipped_grad_norm": 0.9018753170967102 + }, + { + "bits_per_byte": 1.497909039052563, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.038271427154541, + "step": 6350, + "unclipped_grad_norm": 0.9401493668556213 + }, + { + "bits_per_byte": 1.4529796034105593, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.0071287155151367, + "step": 6360, + "unclipped_grad_norm": 0.8621690273284912 + }, + { + "bits_per_byte": 1.506204278251629, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 1.0440212488174438, + "step": 6370, + "unclipped_grad_norm": 0.8795078992843628 + }, + { + "bits_per_byte": 1.4540510553254347, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.007871389389038, + "step": 6380, + "unclipped_grad_norm": 0.8092584609985352 + }, + { + "bits_per_byte": 1.5016250681897556, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0408471822738647, + "step": 6390, + "unclipped_grad_norm": 0.8602833151817322 + }, + { + "bits_per_byte": 1.4795925427703722, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.0255753993988037, + "step": 6400, + "unclipped_grad_norm": 0.8351550102233887 + }, + { + "bits_per_byte": 1.4637653233767696, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.0146048069000244, + "step": 6410, + "unclipped_grad_norm": 0.8905119299888611 + }, + { + "bits_per_byte": 1.444658738796515, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0013611316680908, + "step": 6420, + "unclipped_grad_norm": 0.864800214767456 + }, + { + "bits_per_byte": 1.4844902647016431, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.0289702415466309, + "step": 6430, + "unclipped_grad_norm": 0.8982930779457092 + }, + { + "bits_per_byte": 1.4770083314585076, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.0237841606140137, + "step": 6440, + "unclipped_grad_norm": 0.8630130887031555 + }, + { + "bits_per_byte": 1.4631460138510126, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.014175534248352, + "step": 6450, + "unclipped_grad_norm": 0.8504552841186523 + }, + { + "bits_per_byte": 1.4514069940510699, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.0060386657714844, + "step": 6460, + "unclipped_grad_norm": 0.8505173921585083 + }, + { + "bits_per_byte": 1.4858781646941062, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.0299322605133057, + "step": 6470, + "unclipped_grad_norm": 0.8629505038261414 + }, + { + "bits_per_byte": 1.4917465567068469, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 1.0339999198913574, + "step": 6480, + "unclipped_grad_norm": 0.8834547996520996 + }, + { + "bits_per_byte": 1.4793987183228499, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.02544105052948, + "step": 6490, + "unclipped_grad_norm": 0.9070925712585449 + }, + { + "bits_per_byte": 1.4808681380918216, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.0264595746994019, + "step": 6500, + "unclipped_grad_norm": 0.8845940828323364 + }, + { + "bits_per_byte": 1.4247765124608691, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 0.9875798225402832, + "step": 6510, + "unclipped_grad_norm": 0.8376188278198242 + }, + { + "bits_per_byte": 1.4013865279782254, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 0.9713671207427979, + "step": 6520, + "unclipped_grad_norm": 0.8534479737281799 + }, + { + "bits_per_byte": 1.5087006064289026, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.0457515716552734, + "step": 6530, + "unclipped_grad_norm": 0.9148337244987488 + }, + { + "bits_per_byte": 1.4696770550175278, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.018702507019043, + "step": 6540, + "unclipped_grad_norm": 0.8229706883430481 + }, + { + "bits_per_byte": 1.5014071661711126, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 1.040696144104004, + "step": 6550, + "unclipped_grad_norm": 0.8751465678215027 + }, + { + "bits_per_byte": 1.4696712076073983, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.0186984539031982, + "step": 6560, + "unclipped_grad_norm": 0.8710578083992004 + }, + { + "bits_per_byte": 1.3961035649090217, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 0.967705249786377, + "step": 6570, + "unclipped_grad_norm": 0.8840856552124023 + }, + { + "bits_per_byte": 1.4996073677298332, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.039448618888855, + "step": 6580, + "unclipped_grad_norm": 0.8965790867805481 + }, + { + "bits_per_byte": 1.4646056306088853, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.0151872634887695, + "step": 6590, + "unclipped_grad_norm": 0.8638942241668701 + }, + { + "bits_per_byte": 1.4059718434242041, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.974545419216156, + "step": 6600, + "unclipped_grad_norm": 0.8657469153404236 + }, + { + "bits_per_byte": 1.5139821936368993, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.049412488937378, + "step": 6610, + "unclipped_grad_norm": 0.8749237060546875 + }, + { + "bits_per_byte": 1.5330705480561624, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 1.0626435279846191, + "step": 6620, + "unclipped_grad_norm": 0.9075427651405334 + }, + { + "bits_per_byte": 1.494598373023454, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.0359766483306885, + "step": 6630, + "unclipped_grad_norm": 0.9141798615455627 + }, + { + "bits_per_byte": 1.4761069703853356, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.023159384727478, + "step": 6640, + "unclipped_grad_norm": 0.9060668349266052 + }, + { + "bits_per_byte": 1.4529606853189645, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.0071156024932861, + "step": 6650, + "unclipped_grad_norm": 0.86590975522995 + }, + { + "bits_per_byte": 1.5066096413597125, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.044302225112915, + "step": 6660, + "unclipped_grad_norm": 0.887394905090332 + }, + { + "bits_per_byte": 1.4867124525334416, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.0305105447769165, + "step": 6670, + "unclipped_grad_norm": 0.9190967082977295 + }, + { + "bits_per_byte": 1.507951793967046, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.0452325344085693, + "step": 6680, + "unclipped_grad_norm": 0.9080004692077637 + }, + { + "bits_per_byte": 1.4695153913257168, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.0185904502868652, + "step": 6690, + "unclipped_grad_norm": 0.8927342295646667 + }, + { + "bits_per_byte": 1.5133348509390518, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.0489637851715088, + "step": 6700, + "unclipped_grad_norm": 0.9144639372825623 + }, + { + "bits_per_byte": 1.4453965443687165, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.0018725395202637, + "step": 6710, + "unclipped_grad_norm": 0.8875970840454102 + }, + { + "bits_per_byte": 1.4502357921986941, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.0052268505096436, + "step": 6720, + "unclipped_grad_norm": 0.8778177499771118 + }, + { + "bits_per_byte": 1.4290980065204244, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 0.9905752539634705, + "step": 6730, + "unclipped_grad_norm": 0.891217827796936 + }, + { + "bits_per_byte": 1.4631644159946549, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 1.014188289642334, + "step": 6740, + "unclipped_grad_norm": 0.9166605472564697 + }, + { + "bits_per_byte": 1.4281306901006465, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 0.9899047613143921, + "step": 6750, + "unclipped_grad_norm": 0.8902378678321838 + }, + { + "bits_per_byte": 1.4114853492368895, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 0.9783670902252197, + "step": 6760, + "unclipped_grad_norm": 0.8585396409034729 + }, + { + "bits_per_byte": 1.4293125548773757, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 0.9907239675521851, + "step": 6770, + "unclipped_grad_norm": 0.839362621307373 + }, + { + "bits_per_byte": 1.4443323157251773, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 1.0011348724365234, + "step": 6780, + "unclipped_grad_norm": 0.8865506052970886 + }, + { + "bits_per_byte": 1.5321463132904258, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.0620028972625732, + "step": 6790, + "unclipped_grad_norm": 0.9070268273353577 + }, + { + "bits_per_byte": 1.5086359409521781, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0457067489624023, + "step": 6800, + "unclipped_grad_norm": 0.9465430378913879 + }, + { + "bits_per_byte": 1.4761700880182023, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0232031345367432, + "step": 6810, + "unclipped_grad_norm": 0.9031977653503418 + }, + { + "bits_per_byte": 1.4240491978303704, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 0.987075686454773, + "step": 6820, + "unclipped_grad_norm": 0.8830646872520447 + }, + { + "bits_per_byte": 1.4450329730447924, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.0016205310821533, + "step": 6830, + "unclipped_grad_norm": 0.8898956179618835 + }, + { + "bits_per_byte": 1.4835576027860142, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 1.028323769569397, + "step": 6840, + "unclipped_grad_norm": 0.8833323121070862 + }, + { + "bits_per_byte": 1.4698094816586922, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.018794298171997, + "step": 6850, + "unclipped_grad_norm": 0.9023848176002502 + }, + { + "bits_per_byte": 1.4589838617174817, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.0112905502319336, + "step": 6860, + "unclipped_grad_norm": 0.87961345911026 + }, + { + "bits_per_byte": 1.4633828339612507, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 1.0143396854400635, + "step": 6870, + "unclipped_grad_norm": 0.9281585812568665 + }, + { + "bits_per_byte": 1.5043229600338406, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 1.0427172183990479, + "step": 6880, + "unclipped_grad_norm": 0.8951436281204224 + }, + { + "bits_per_byte": 1.3983493143639882, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 0.969261884689331, + "step": 6890, + "unclipped_grad_norm": 0.8639072775840759 + }, + { + "bits_per_byte": 1.4340412178628485, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 0.9940016269683838, + "step": 6900, + "unclipped_grad_norm": 0.8780765533447266 + }, + { + "bits_per_byte": 1.5785962475487074, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.094199538230896, + "step": 6910, + "unclipped_grad_norm": 0.9642757773399353 + }, + { + "bits_per_byte": 1.4159894028803357, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 0.9814890623092651, + "step": 6920, + "unclipped_grad_norm": 0.8638912439346313 + }, + { + "bits_per_byte": 1.425586550746433, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.9881412982940674, + "step": 6930, + "unclipped_grad_norm": 0.9177617430686951 + }, + { + "bits_per_byte": 1.4596446190620966, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 1.0117485523223877, + "step": 6940, + "unclipped_grad_norm": 0.8722310662269592 + }, + { + "bits_per_byte": 1.4445414466286266, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 1.0012798309326172, + "step": 6950, + "unclipped_grad_norm": 0.8999669551849365 + }, + { + "bits_per_byte": 1.4954775483346645, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 1.036586046218872, + "step": 6960, + "unclipped_grad_norm": 0.9472109079360962 + }, + { + "bits_per_byte": 1.4678593703805587, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 1.0174425840377808, + "step": 6970, + "unclipped_grad_norm": 0.9188230037689209 + }, + { + "bits_per_byte": 1.5123616011178191, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.048289179801941, + "step": 6980, + "unclipped_grad_norm": 0.956657350063324 + }, + { + "bits_per_byte": 1.4579877382036843, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.0106000900268555, + "step": 6990, + "unclipped_grad_norm": 0.8828865885734558 + }, + { + "bits_per_byte": 1.4102709797391473, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 0.9775253534317017, + "step": 7000, + "unclipped_grad_norm": 0.9071143269538879 + }, + { + "bits_per_byte": 1.450688278553114, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 1.0055404901504517, + "step": 7010, + "unclipped_grad_norm": 0.923660159111023 + }, + { + "bits_per_byte": 1.4221000324568167, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 0.9857246279716492, + "step": 7020, + "unclipped_grad_norm": 0.8817518949508667 + }, + { + "bits_per_byte": 1.4361036338119955, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 0.9954311847686768, + "step": 7030, + "unclipped_grad_norm": 0.8942034244537354 + }, + { + "bits_per_byte": 1.5146297083173978, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.0498613119125366, + "step": 7040, + "unclipped_grad_norm": 0.9504315853118896 + }, + { + "bits_per_byte": 1.4749856435017101, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.022382140159607, + "step": 7050, + "unclipped_grad_norm": 0.9024695754051208 + }, + { + "bits_per_byte": 1.4841951424727626, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 1.0287656784057617, + "step": 7060, + "unclipped_grad_norm": 0.9987031817436218 + }, + { + "bits_per_byte": 1.4343887948002423, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 0.9942425489425659, + "step": 7070, + "unclipped_grad_norm": 0.8814687132835388 + }, + { + "bits_per_byte": 1.3483379633716395, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 0.9345966577529907, + "step": 7080, + "unclipped_grad_norm": 0.9086679816246033 + }, + { + "bits_per_byte": 1.4180791640709696, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 0.9829375743865967, + "step": 7090, + "unclipped_grad_norm": 0.8847902417182922 + }, + { + "bits_per_byte": 1.4043928707066329, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 0.9734509587287903, + "step": 7100, + "unclipped_grad_norm": 0.9200909733772278 + }, + { + "bits_per_byte": 1.4728762762888779, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 1.0209200382232666, + "step": 7110, + "unclipped_grad_norm": 0.9179381728172302 + }, + { + "bits_per_byte": 1.456272039278677, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.0094108581542969, + "step": 7120, + "unclipped_grad_norm": 0.9354409575462341 + }, + { + "bits_per_byte": 1.413579409993795, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 0.97981858253479, + "step": 7130, + "unclipped_grad_norm": 0.8862469792366028 + }, + { + "bits_per_byte": 1.522903277702455, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.055596113204956, + "step": 7140, + "unclipped_grad_norm": 0.9388699531555176 + }, + { + "bits_per_byte": 1.5318888552620842, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 1.0618244409561157, + "step": 7150, + "unclipped_grad_norm": 0.9473410248756409 + }, + { + "bits_per_byte": 1.4570297948383786, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0099360942840576, + "step": 7160, + "unclipped_grad_norm": 0.9446573853492737 + }, + { + "bits_per_byte": 1.4438824091105202, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.0008230209350586, + "step": 7170, + "unclipped_grad_norm": 0.9158006310462952 + }, + { + "bits_per_byte": 1.3822597354364945, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 0.958109438419342, + "step": 7180, + "unclipped_grad_norm": 0.8682859539985657 + }, + { + "bits_per_byte": 1.383616678551802, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 0.9590499997138977, + "step": 7190, + "unclipped_grad_norm": 0.8774144649505615 + }, + { + "bits_per_byte": 1.4863700350755737, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 1.030273199081421, + "step": 7200, + "unclipped_grad_norm": 0.9574132561683655 + }, + { + "bits_per_byte": 1.4696534933943595, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 1.0186861753463745, + "step": 7210, + "unclipped_grad_norm": 0.8893125653266907 + }, + { + "bits_per_byte": 1.431243920046565, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 0.9920626878738403, + "step": 7220, + "unclipped_grad_norm": 0.8840544819831848 + }, + { + "bits_per_byte": 1.4359567606281587, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 0.9953293800354004, + "step": 7230, + "unclipped_grad_norm": 0.920866847038269 + }, + { + "bits_per_byte": 1.4126603347075843, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 0.9791815280914307, + "step": 7240, + "unclipped_grad_norm": 0.9085155129432678 + }, + { + "bits_per_byte": 1.4389281048871154, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 0.9973889589309692, + "step": 7250, + "unclipped_grad_norm": 0.9654173254966736 + }, + { + "bits_per_byte": 1.430908209912081, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 0.9918299913406372, + "step": 7260, + "unclipped_grad_norm": 0.9169784784317017 + }, + { + "bits_per_byte": 1.4964650727159188, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 1.0372705459594727, + "step": 7270, + "unclipped_grad_norm": 0.9460402727127075 + }, + { + "bits_per_byte": 1.4553687863963456, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.0087847709655762, + "step": 7280, + "unclipped_grad_norm": 0.9709736108779907 + }, + { + "bits_per_byte": 1.4400710155934242, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 0.998181164264679, + "step": 7290, + "unclipped_grad_norm": 0.9269301295280457 + }, + { + "bits_per_byte": 1.4390281987899176, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 0.9974583387374878, + "step": 7300, + "unclipped_grad_norm": 0.9335241913795471 + }, + { + "bits_per_byte": 1.4888771981598516, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 1.0320110321044922, + "step": 7310, + "unclipped_grad_norm": 0.9333628416061401 + }, + { + "bits_per_byte": 1.458887035485046, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 1.0112234354019165, + "step": 7320, + "unclipped_grad_norm": 0.9484428763389587 + }, + { + "bits_per_byte": 1.465335524979147, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 1.015693187713623, + "step": 7330, + "unclipped_grad_norm": 0.996306836605072 + }, + { + "bits_per_byte": 1.4293175423742508, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.990727424621582, + "step": 7340, + "unclipped_grad_norm": 0.9288363456726074 + }, + { + "bits_per_byte": 1.4790635241363181, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 1.0252087116241455, + "step": 7350, + "unclipped_grad_norm": 0.9366341233253479 + }, + { + "bits_per_byte": 1.4634435438370053, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 1.014381766319275, + "step": 7360, + "unclipped_grad_norm": 0.911174476146698 + }, + { + "bits_per_byte": 1.472038548796525, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 1.0203393697738647, + "step": 7370, + "unclipped_grad_norm": 0.9289841651916504 + }, + { + "bits_per_byte": 1.516696595815467, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 1.051293969154358, + "step": 7380, + "unclipped_grad_norm": 0.9735171794891357 + }, + { + "bits_per_byte": 1.4907805301569503, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 1.0333303213119507, + "step": 7390, + "unclipped_grad_norm": 0.9834519624710083 + }, + { + "bits_per_byte": 1.4227901128434037, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.9862029552459717, + "step": 7400, + "unclipped_grad_norm": 0.911080002784729 + }, + { + "bits_per_byte": 1.4410250893490857, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 0.9988424777984619, + "step": 7410, + "unclipped_grad_norm": 0.9081923365592957 + }, + { + "bits_per_byte": 1.5121206534239604, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0481221675872803, + "step": 7420, + "unclipped_grad_norm": 0.9652585387229919 + }, + { + "bits_per_byte": 1.424262800282742, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 0.987223744392395, + "step": 7430, + "unclipped_grad_norm": 0.971235990524292 + }, + { + "bits_per_byte": 1.4423605346330353, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 0.9997681379318237, + "step": 7440, + "unclipped_grad_norm": 0.9377946257591248 + }, + { + "bits_per_byte": 1.3811993764025998, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 0.9573744535446167, + "step": 7450, + "unclipped_grad_norm": 0.9864786863327026 + }, + { + "bits_per_byte": 1.4081080399305712, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 0.9760261178016663, + "step": 7460, + "unclipped_grad_norm": 0.8965867161750793 + }, + { + "bits_per_byte": 1.4674344012502767, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 1.0171480178833008, + "step": 7470, + "unclipped_grad_norm": 0.9695210456848145 + }, + { + "bits_per_byte": 1.3825076484277132, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 0.9582812786102295, + "step": 7480, + "unclipped_grad_norm": 0.9450381994247437 + }, + { + "bits_per_byte": 1.416268530722686, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 0.981682538986206, + "step": 7490, + "unclipped_grad_norm": 0.9229742288589478 + }, + { + "bits_per_byte": 1.4874254926039188, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.031004786491394, + "step": 7500, + "unclipped_grad_norm": 0.931046187877655 + }, + { + "bits_per_byte": 1.4623161975605996, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 1.0136003494262695, + "step": 7510, + "unclipped_grad_norm": 0.9589980244636536 + }, + { + "bits_per_byte": 1.4426217762796958, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 0.9999492168426514, + "step": 7520, + "unclipped_grad_norm": 0.9353710412979126 + }, + { + "bits_per_byte": 1.4634922149271994, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.0144155025482178, + "step": 7530, + "unclipped_grad_norm": 0.9122790098190308 + }, + { + "bits_per_byte": 1.495648843054924, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.0367047786712646, + "step": 7540, + "unclipped_grad_norm": 0.9431754946708679 + }, + { + "bits_per_byte": 1.4333156230588584, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 0.993498682975769, + "step": 7550, + "unclipped_grad_norm": 0.9254633784294128 + }, + { + "bits_per_byte": 1.4703515709742117, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 1.0191700458526611, + "step": 7560, + "unclipped_grad_norm": 0.9234781265258789 + }, + { + "bits_per_byte": 1.4767751229839376, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 1.0236225128173828, + "step": 7570, + "unclipped_grad_norm": 0.9604462385177612 + }, + { + "bits_per_byte": 1.4300699664717755, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 0.9912489652633667, + "step": 7580, + "unclipped_grad_norm": 0.940305769443512 + }, + { + "bits_per_byte": 1.4335085875931266, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 0.993632435798645, + "step": 7590, + "unclipped_grad_norm": 0.9437996745109558 + }, + { + "bits_per_byte": 1.4131444658697632, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 0.9795171022415161, + "step": 7600, + "unclipped_grad_norm": 0.9260700345039368 + }, + { + "bits_per_byte": 1.3642425749581304, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 0.9456208944320679, + "step": 7610, + "unclipped_grad_norm": 0.9076235294342041 + }, + { + "bits_per_byte": 1.470742143574321, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 1.019440770149231, + "step": 7620, + "unclipped_grad_norm": 0.9942763447761536 + }, + { + "bits_per_byte": 1.4260536556261765, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 0.9884650707244873, + "step": 7630, + "unclipped_grad_norm": 0.9157193303108215 + }, + { + "bits_per_byte": 1.422580294009351, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.9860575199127197, + "step": 7640, + "unclipped_grad_norm": 0.9327218532562256 + }, + { + "bits_per_byte": 1.5098085186657608, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 1.0465195178985596, + "step": 7650, + "unclipped_grad_norm": 1.0226681232452393 + }, + { + "bits_per_byte": 1.4502154982458924, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 1.0052127838134766, + "step": 7660, + "unclipped_grad_norm": 0.9471380710601807 + }, + { + "bits_per_byte": 1.413664283431996, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 0.9798774123191833, + "step": 7670, + "unclipped_grad_norm": 0.9374066591262817 + }, + { + "bits_per_byte": 1.389169396408884, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 0.962898850440979, + "step": 7680, + "unclipped_grad_norm": 0.9010422229766846 + }, + { + "bits_per_byte": 1.4406357206275322, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 0.998572587966919, + "step": 7690, + "unclipped_grad_norm": 0.94871586561203 + }, + { + "bits_per_byte": 1.446458709220445, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 1.0026087760925293, + "step": 7700, + "unclipped_grad_norm": 1.0057995319366455 + }, + { + "bits_per_byte": 1.3678712369086847, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 0.9481360912322998, + "step": 7710, + "unclipped_grad_norm": 0.9059723019599915 + }, + { + "bits_per_byte": 1.3764043281138938, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 0.9540507793426514, + "step": 7720, + "unclipped_grad_norm": 0.89348965883255 + }, + { + "bits_per_byte": 1.4205549403214652, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 0.984653651714325, + "step": 7730, + "unclipped_grad_norm": 0.905714750289917 + }, + { + "bits_per_byte": 1.4181357463631035, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 0.9829767942428589, + "step": 7740, + "unclipped_grad_norm": 0.9062879085540771 + }, + { + "bits_per_byte": 1.4299636811935423, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9911752939224243, + "step": 7750, + "unclipped_grad_norm": 0.9408755302429199 + }, + { + "bits_per_byte": 1.4095965497737888, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 0.9770578742027283, + "step": 7760, + "unclipped_grad_norm": 0.9354599714279175 + }, + { + "bits_per_byte": 1.4129720532622732, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 0.9793975949287415, + "step": 7770, + "unclipped_grad_norm": 0.9595562815666199 + }, + { + "bits_per_byte": 1.5389698969460586, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 1.06673264503479, + "step": 7780, + "unclipped_grad_norm": 1.0378655195236206 + }, + { + "bits_per_byte": 1.4196591686844462, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 0.9840327501296997, + "step": 7790, + "unclipped_grad_norm": 0.9534812569618225 + }, + { + "bits_per_byte": 1.46563511875695, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 1.0159008502960205, + "step": 7800, + "unclipped_grad_norm": 1.0107316970825195 + }, + { + "bits_per_byte": 1.3875758051659892, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9617942571640015, + "step": 7810, + "unclipped_grad_norm": 0.9311396479606628 + }, + { + "bits_per_byte": 1.4088830797466845, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 0.9765633344650269, + "step": 7820, + "unclipped_grad_norm": 0.9659230709075928 + }, + { + "bits_per_byte": 1.451390655699238, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 1.006027340888977, + "step": 7830, + "unclipped_grad_norm": 0.9494246244430542 + }, + { + "bits_per_byte": 1.4084317972708205, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.9762505292892456, + "step": 7840, + "unclipped_grad_norm": 0.9109260439872742 + }, + { + "bits_per_byte": 1.3780774613328126, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.9552105069160461, + "step": 7850, + "unclipped_grad_norm": 0.9282220602035522 + }, + { + "bits_per_byte": 1.397101666223304, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 0.9683970808982849, + "step": 7860, + "unclipped_grad_norm": 0.9663445949554443 + }, + { + "bits_per_byte": 1.4514924694285487, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0060979127883911, + "step": 7870, + "unclipped_grad_norm": 0.9978665709495544 + }, + { + "bits_per_byte": 1.421875853071417, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9855692386627197, + "step": 7880, + "unclipped_grad_norm": 0.9516252875328064 + }, + { + "bits_per_byte": 1.499389637693841, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 1.0392976999282837, + "step": 7890, + "unclipped_grad_norm": 0.979440450668335 + }, + { + "bits_per_byte": 1.4682203619647198, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 1.0176928043365479, + "step": 7900, + "unclipped_grad_norm": 0.9775227904319763 + }, + { + "bits_per_byte": 1.4497874334278948, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.0049160718917847, + "step": 7910, + "unclipped_grad_norm": 1.0392301082611084 + }, + { + "bits_per_byte": 1.373789331907525, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 0.9522382020950317, + "step": 7920, + "unclipped_grad_norm": 0.979443371295929 + }, + { + "bits_per_byte": 1.3771509048012893, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.9545682668685913, + "step": 7930, + "unclipped_grad_norm": 0.9486931562423706 + }, + { + "bits_per_byte": 1.4403031061807634, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 0.9983420372009277, + "step": 7940, + "unclipped_grad_norm": 1.0302218198776245 + }, + { + "bits_per_byte": 1.4070514645149954, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 0.975293755531311, + "step": 7950, + "unclipped_grad_norm": 0.9500374794006348 + }, + { + "bits_per_byte": 1.4169921477261913, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 0.982184112071991, + "step": 7960, + "unclipped_grad_norm": 0.9470998048782349 + }, + { + "bits_per_byte": 1.4089996839839694, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 0.9766441583633423, + "step": 7970, + "unclipped_grad_norm": 0.9724183082580566 + }, + { + "bits_per_byte": 1.4438361457774382, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 1.0007909536361694, + "step": 7980, + "unclipped_grad_norm": 0.9879897236824036 + }, + { + "bits_per_byte": 1.446100297376047, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 1.0023603439331055, + "step": 7990, + "unclipped_grad_norm": 0.9398271441459656 + }, + { + "bits_per_byte": 1.3767236139052206, + "learning_rate": 3e-05, + "loss_nats": 0.9542720913887024, + "step": 8000, + "unclipped_grad_norm": 0.9364242553710938 + } + ], + "variant": "uniform_group_7_mlp_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073001.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073001.json new file mode 100644 index 0000000..d149f33 --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073001.json @@ -0,0 +1,14077 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "d85887f72fb1577b4ce446757bab5154ce794490cf1c1f966e050208d90c4823", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.0002677410375326872, + 0.0002473295899108052, + 0.00023034414334688336, + 0.00021438254043459892, + 0.00020044577831868082, + 0.00018819731485564262, + 0.00017973134526982903, + 0.000171217747265473, + 0.0001640921109355986, + 0.00016061548376455903, + 0.00015463090676348656, + 0.00015024171443656087, + 0.0001478576596127823, + 0.00014403100067283958, + 0.00013951722939964384, + 0.00013661097909789532, + 0.00013379570737015456, + 0.0001311724045081064, + 0.0001286817860091105, + 0.00012687484559137374, + 0.0001249351626029238, + 0.00012369138130452484, + 0.00012227990373503417, + 0.00012101328320568427, + 0.00011953213106608018, + 0.0001181897969217971, + 0.0001171672047348693, + 0.00011609455395955592, + 0.00011492414341773838, + 0.0001140913664130494, + 0.00011346775136189535, + 0.00011285000800853595 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00021242368711682502, + "first_to_last_ratio": 1.8345658968465297, + "imbalance_abs_log_ratio": 0.6068078850437921, + "last_quartile_mean": 0.0001157896194854402, + "mean": 0.0001511171253696375, + "normalized": [ + 1.7717451736709768, + 1.636674793183293, + 1.5242755762025908, + 1.4186515254985965, + 1.3264266232459345, + 1.2453738409548603, + 1.1893512719369177, + 1.1330135273991528, + 1.085860457802012, + 1.0628542818802846, + 1.023252039669589, + 0.9942070699734702, + 0.9784308644775869, + 0.9531083940389613, + 0.9232390376562555, + 0.9040072643238835, + 0.8853775311228679, + 0.8680181295617843, + 0.8515367513400655, + 0.8395795332993111, + 0.8267439067367663, + 0.8185133286645813, + 0.8091730400239778, + 0.800791325997512, + 0.7909899739933552, + 0.7821072339266707, + 0.7753403490721149, + 0.7682422073314642, + 0.7604971517068638, + 0.7549863467424895, + 0.7508596466770359, + 0.7467718018887739 + ], + "population_cv": 0.2714186609524326 + }, + "activation_output_rms_by_block": [ + 0.0034885562490671873, + 0.004831995815038681, + 0.005952076520770788, + 0.006925336085259914, + 0.00385080324485898, + 0.005542940925806761, + 0.006729288958013058, + 0.007927131839096546, + 0.004081308841705322, + 0.005811207927763462, + 0.007534808944910765, + 0.008581922389566898, + 0.00499920267611742, + 0.007102519739419222, + 0.008542952127754688, + 0.010447698645293713, + 0.005041283555328846, + 0.007636935915797949, + 0.009387043304741383, + 0.011020516976714134, + 0.005807018838822842, + 0.008367395959794521, + 0.01024632342159748, + 0.012378672137856483, + 0.006453148555010557, + 0.009127611294388771, + 0.011037957854568958, + 0.012611051090061665, + 0.007010402157902718, + 0.009794782847166061, + 0.012445803731679916, + 0.014087481424212456 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.0056560162047389895, + "first_to_last_ratio": 0.5480088979804595, + "imbalance_abs_log_ratio": 0.6014637549753238, + "last_quartile_mean": 0.010321029869373888, + "mean": 0.007962599374877755, + "normalized": [ + 0.4381177659237371, + 0.606836484864952, + 0.747504205668035, + 0.8697330807712821, + 0.4836113263475722, + 0.6961220406611074, + 0.845112084785299, + 0.9955457339856744, + 0.512559862622497, + 0.7298129234151854, + 0.9462750278110585, + 1.077778999737589, + 0.6278355146046978, + 0.8919850673170736, + 1.072884836415099, + 1.3120964842531853, + 0.6331203314377778, + 0.9591008609440675, + 1.1788918244911069, + 1.384035094304029, + 0.7292868277593074, + 1.0508372411895934, + 1.2868063479276561, + 1.554601902603762, + 0.8104324041933391, + 1.1463105029730196, + 1.3862254440922965, + 1.5837857082010076, + 0.8804162846646224, + 1.2300986632667847, + 1.5630327667805075, + 1.7692063559870779 + ], + "population_cv": 0.3451071044649734 + }, + "bits_per_byte": 7.959341947457508, + "branch_output_rms_by_sublayer": [ + 0.0022265110164880753, + 0.0027073738165199757, + 0.0021613112185150385, + 0.002722575329244137, + 0.0025601957459002733, + 0.0026850115973502398, + 0.0025323182344436646, + 0.0026861224323511124, + 0.0027253651060163975, + 0.00271838391199708, + 0.002904455177485943, + 0.0026645169127732515, + 0.0029208383057266474, + 0.0026859452482312918, + 0.00299624796025455, + 0.0027359756641089916, + 0.0030937574338167906, + 0.0026767742820084095, + 0.0033804215490818024, + 0.0027451682835817337, + 0.0034370317589491606, + 0.0026999011170119047, + 0.0038969130255281925, + 0.002663193503394723, + 0.004255065228790045, + 0.002626648638397455, + 0.004119740333408117, + 0.0026531771291047335, + 0.003970506135374308, + 0.0026517712976783514, + 0.004385598469525576, + 0.002676903735846281, + 0.0041999719105660915, + 0.002670086920261383, + 0.005232699681073427, + 0.0026355870068073273, + 0.004664790350943804, + 0.0026610493659973145, + 0.004898969549685717, + 0.0026388936676084995, + 0.005131533369421959, + 0.0026469926815479994, + 0.005069608334451914, + 0.00264862016774714, + 0.0051318020559847355, + 0.002641494618728757, + 0.0058372304774820805, + 0.002674198243767023, + 0.0059690922498703, + 0.002557571977376938, + 0.00554921617731452, + 0.00252733426168561, + 0.0058550527319312096, + 0.0025468147359788418, + 0.006255472078919411, + 0.0026856842450797558, + 0.006502739619463682, + 0.002551089273765683, + 0.006122888531535864, + 0.0025778315030038357, + 0.007071305997669697, + 0.0026001501828432083, + 0.006241494789719582, + 0.0025662274565547705 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0056993408498569815, + 0.005586029321860134, + 0.005830344684528519, + 0.00540288712216694, + 0.00538025811909819, + 0.0052226633069546126, + 0.005297605878918106, + 0.0052023753769565, + 0.0048107019260181325, + 0.005248672753719287, + 0.004962353375385338, + 0.005501319898161136, + 0.005588445553926757, + 0.005229606380765812, + 0.004884302670286647, + 0.005031817046498285, + 0.005000470961685071, + 0.0053937604587735305, + 0.004980398173225761, + 0.004658922297714218, + 0.00496598185096497, + 0.004578385934052906, + 0.004596161734371138, + 0.004990442293511754, + 0.005201771295872185, + 0.00459813521112035, + 0.004706502430387731, + 0.004781878207244352, + 0.0048378722082161795, + 0.004473110720597102, + 0.005084432926394394, + 0.004685705425265166 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.005452688082542498, + "first_to_last_ratio": 1.1368823875795573, + "imbalance_abs_log_ratio": 0.12828976841498158, + "last_quartile_mean": 0.004796176053137182, + "mean": 0.005075395512328069, + "normalized": [ + 1.122935313319594, + 1.1006096585560166, + 1.1487468652180286, + 1.064525337787646, + 1.060066768398564, + 1.0290160233362764, + 1.0437818818356701, + 1.025018713185997, + 0.947847692723257, + 1.0341406380981206, + 0.9777274230809889, + 1.0839194472230789, + 1.1010857262951226, + 1.0303840100861434, + 0.9623491722808084, + 0.9914137793352396, + 0.9852376922229987, + 1.0627271206100406, + 0.9812827711906272, + 0.9179427074003902, + 0.9784423379227619, + 0.9020747098294245, + 0.9055770576316904, + 0.9832617539638114, + 1.024899691706223, + 0.905965889742295, + 0.9273173724009683, + 0.9421685848185097, + 0.9532010257062827, + 0.8813324419214177, + 1.00178063247374, + 0.9232197596982638 + ], + "population_cv": 0.06921897980134169 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.02802751213312149, + 0.014061377383768559, + 0.01413465291261673, + 0.014165290631353855, + 0.014225807972252369, + 0.014281988143920898, + 0.01433455292135477, + 0.014350161887705326, + 0.01440478302538395, + 0.00966146495193243, + 0.009703479707241058, + 0.009741888381540775, + 0.00977881159633398, + 0.009820535778999329, + 0.009858808480203152, + 0.00991273857653141, + 0.009950309991836548, + 0.0074927168898284435, + 0.00752287870272994, + 0.007546746172010899, + 0.007571155205368996, + 0.00766373984515667, + 0.007693085819482803, + 0.007762390188872814, + 0.007788818795233965, + 0.006325984839349985, + 0.006353048607707024, + 0.006375366821885109, + 0.006389216054230928, + 0.006445344537496567, + 0.0064707002602517605, + 0.006555322092026472, + 0.00656962301582098, + 0.005531130358576775, + 0.005553483963012695, + 0.005611324217170477, + 0.0056249531917274, + 0.0056777093559503555, + 0.005688921548426151, + 0.00574630219489336, + 0.005768724251538515, + 0.004946626722812653, + 0.004966245498508215, + 0.0050392793491482735, + 0.005057469941675663, + 0.005084683652967215, + 0.005098348017781973, + 0.005203457549214363, + 0.005216081161051989, + 0.004667261149734259, + 0.0046758754178881645, + 0.004717131145298481, + 0.004727107938379049, + 0.004768412560224533, + 0.004777040798217058, + 0.004826710093766451, + 0.004837613552808762, + 0.0044052014127373695, + 0.0044186655431985855, + 0.004504534415900707, + 0.004510571248829365, + 0.004574235528707504, + 0.0045849340967834, + 0.004644096828997135 + ], + "loss_nats": 5.516995429992676, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0022265110164880753, + 0.0034885562490671873, + 0.0040087271481752396, + 0.004831995815038681, + 0.005379301495850086, + 0.005952076520770788, + 0.006410556845366955, + 0.006925336085259914, + 0.0027253651060163975, + 0.00385080324485898, + 0.004838948138058186, + 0.005542940925806761, + 0.006175771821290255, + 0.006729288958013058, + 0.00742313964292407, + 0.007927131839096546, + 0.0030937574338167906, + 0.004081308841705322, + 0.005150026176124811, + 0.005811207927763462, + 0.007019025273621082, + 0.007534808944910765, + 0.008143599145114422, + 0.008581922389566898, + 0.004255065228790045, + 0.00499920267611742, + 0.0066224560141563416, + 0.007102519739419222, + 0.008129785768687725, + 0.008542952127754688, + 0.010099578648805618, + 0.010447698645293713, + 0.0041999719105660915, + 0.005041283555328846, + 0.00719039561226964, + 0.007636935915797949, + 0.009030314162373543, + 0.009387043304741383, + 0.010736005380749702, + 0.011020516976714134, + 0.005131533369421959, + 0.005807018838822842, + 0.00787449348717928, + 0.008367395959794521, + 0.009912566281855106, + 0.01024632342159748, + 0.012113703414797783, + 0.012378672137856483, + 0.0059690922498703, + 0.006453148555010557, + 0.00876442901790142, + 0.009127611294388771, + 0.010806876234710217, + 0.011037957854568958, + 0.012353328987956047, + 0.012611051090061665, + 0.006502739619463682, + 0.007010402157902718, + 0.009470963850617409, + 0.009794782847166061, + 0.012123330496251583, + 0.012445803731679916, + 0.01384514570236206, + 0.014087481424212456 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.941999163245782e-05, + 6.621817010454834e-05, + 6.276910426095128e-05, + 5.9923673688899726e-05, + 5.571635483647697e-05, + 5.268222594168037e-05, + 4.969063229509629e-05, + 4.642214116756804e-05, + 4.2499988921917975e-05, + 3.978446693508886e-05, + 3.700290835695341e-05, + 3.4550401323940605e-05, + 3.204427412129007e-05, + 2.98763407045044e-05, + 2.8043992642778903e-05, + 2.6406481993035413e-05, + 2.5202029064530507e-05, + 2.3980048354133032e-05, + 2.2970654754317366e-05, + 2.2038069801055826e-05, + 2.1245627067401074e-05, + 2.046030931524001e-05, + 1.9785346012213267e-05, + 1.923364652611781e-05, + 1.899822564155329e-05, + 1.857742972788401e-05, + 1.8227287000627257e-05, + 1.7915044736582786e-05, + 1.7522566849947907e-05, + 1.7321037375950254e-05, + 1.712825905997306e-05, + 1.6981100998236798e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.7855286740959855e-05, + "first_to_last_ratio": 3.244124255259362, + "imbalance_abs_log_ratio": 1.1768454386205345, + "last_quartile_mean": 1.783386892384442e-05, + "mean": 3.283243222540477e-05, + "normalized": [ + 2.114372494729241, + 2.016852411357775, + 1.9118018375861412, + 1.8251365990038515, + 1.6969913911332248, + 1.6045788377784698, + 1.513461809772568, + 1.41391112449013, + 1.2944514323563494, + 1.2117429090222809, + 1.1270230637473653, + 1.052325368000197, + 0.9759945258181376, + 0.9099642846863768, + 0.8541551978314688, + 0.80428040821791, + 0.7675955558671622, + 0.7303768477919212, + 0.6996330517525091, + 0.6712286695593455, + 0.6470926954647555, + 0.6231737318384967, + 0.6026159096706807, + 0.585812418466987, + 0.5786420424513363, + 0.5658255715063759, + 0.5551610333188632, + 0.545650855641473, + 0.5336968863485374, + 0.5275587643655515, + 0.5216871824293211, + 0.5172050879951964 + ], + "population_cv": 0.4964530316937415 + }, + "activation_output_rms_by_block": [ + 0.0058692265301942825, + 0.010152247734367847, + 0.014839091338217258, + 0.019591957330703735, + 0.0067398143000900745, + 0.011955589056015015, + 0.01716931350529194, + 0.02306033857166767, + 0.007508765906095505, + 0.013779806904494762, + 0.020409800112247467, + 0.02685040980577469, + 0.0091559374704957, + 0.016716359183192253, + 0.02419254370033741, + 0.03126180171966553, + 0.009785215370357037, + 0.01850821264088154, + 0.02761545218527317, + 0.037425313144922256, + 0.012438409961760044, + 0.023153970018029213, + 0.03432171046733856, + 0.04660208150744438, + 0.017327340319752693, + 0.02931905910372734, + 0.044478677213191986, + 0.06014470010995865, + 0.019144421443343163, + 0.03802234306931496, + 0.05717705935239792, + 0.07674983143806458 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013672197295818478, + "first_to_last_ratio": 0.3194779819553082, + "imbalance_abs_log_ratio": 1.141066921607124, + "last_quartile_mean": 0.04279542900621891, + "mean": 0.02535833751608152, + "normalized": [ + 0.23145155026319017, + 0.40035147130325827, + 0.585176032490566, + 0.7726041708483092, + 0.26578297160907655, + 0.4714658067956201, + 0.6770677886278491, + 0.9093789589732953, + 0.29610639504004016, + 0.5434034031511731, + 0.804855606141537, + 1.058839515356515, + 0.36106221335248306, + 0.6592056428222561, + 0.9540271985494003, + 1.2328017047584527, + 0.38587763744967657, + 0.7298669571356627, + 1.0890087793712917, + 1.4758583097645188, + 0.49050573421352905, + 0.9130712927590635, + 1.3534684774020669, + 1.8377419843824816, + 0.6832995384166726, + 1.1561901124288627, + 1.7540060417992664, + 2.3717919233394786, + 0.7549556997260695, + 1.4994020426300538, + 2.254763716909198, + 3.0266113221890856 + ], + "population_cv": 0.6561149265484539 + }, + "bits_per_byte": 5.151485772272156, + "branch_output_rms_by_sublayer": [ + 0.0029127895832061768, + 0.005085675045847893, + 0.0026882688980549574, + 0.005228722468018532, + 0.0028465059585869312, + 0.005698997061699629, + 0.0029055613558739424, + 0.0058838739059865475, + 0.003085965756326914, + 0.005918111652135849, + 0.0032896376214921474, + 0.005925621837377548, + 0.0038608317263424397, + 0.005900311749428511, + 0.0034083875361829996, + 0.006688539404422045, + 0.003751161741092801, + 0.006295192055404186, + 0.004302044399082661, + 0.006429112516343594, + 0.004344411194324493, + 0.006875772960484028, + 0.004838971421122551, + 0.006519583985209465, + 0.005485833622515202, + 0.006734788883477449, + 0.005453970283269882, + 0.007042968180030584, + 0.00535221304744482, + 0.006980822887271643, + 0.0056844716891646385, + 0.006854009814560413, + 0.005563640967011452, + 0.007032194174826145, + 0.007088648620992899, + 0.007161186542361975, + 0.006791478954255581, + 0.007514364551752806, + 0.00717391399666667, + 0.00770252151414752, + 0.0077114407904446125, + 0.0078731095418334, + 0.008122370578348637, + 0.008188363164663315, + 0.00878527294844389, + 0.008078450337052345, + 0.009222123771905899, + 0.00825345516204834, + 0.01271942537277937, + 0.007952263578772545, + 0.00897612702101469, + 0.008032192476093769, + 0.012183033861219883, + 0.008434710092842579, + 0.012307075783610344, + 0.00904827006161213, + 0.013551505282521248, + 0.00846139620989561, + 0.014996928162872791, + 0.008829282596707344, + 0.015547323040664196, + 0.008883194997906685, + 0.014805778861045837, + 0.009233531542122364 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.000395177475103743, + 0.0003332755344654961, + 0.000315893459816185, + 0.000310100133167921, + 0.00028528332620524554, + 0.00028335812413191985, + 0.00029518873948296287, + 0.00026293067142410537, + 0.0002424619071402874, + 0.00026201411214491806, + 0.00023865248769426136, + 0.0002448734927312214, + 0.0002522633348151854, + 0.00023781239740430576, + 0.00021251402376915363, + 0.0002122590435785732, + 0.00020152644174535755, + 0.0002078757425859546, + 0.00020034669926835373, + 0.00018987579939082946, + 0.0001906458879908057, + 0.0001799030931959508, + 0.00017670666675783845, + 0.00017209476706835628, + 0.00018834348956749082, + 0.00015120127877278264, + 0.00017384489513614726, + 0.00016007662150962894, + 0.00016217445507190053, + 0.00017021204806660572, + 0.00017090328227710973, + 0.00016319949033805103 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0003101509329746973, + "first_to_last_ratio": 1.8517087704220867, + "imbalance_abs_log_ratio": 0.6161088724667294, + "last_quartile_mean": 0.00016749444509246459, + "mean": 0.00022634340380683275, + "normalized": [ + 1.745919997920494, + 1.47243316509423, + 1.39563801950145, + 1.3700427224844969, + 1.260400442014708, + 1.251894773013774, + 1.3041632074018137, + 1.1616449474644162, + 1.0712126046633574, + 1.1575955284675652, + 1.0543823397563354, + 1.081867147938637, + 1.1145159548385752, + 1.0506707657681995, + 0.9389008921617108, + 0.9377743729599494, + 0.8903570342935435, + 0.9184086617490346, + 0.8851448546710675, + 0.8388837324054484, + 0.8422860343370455, + 0.794823662497736, + 0.7807016408953734, + 0.7603259656518481, + 0.8321138871280206, + 0.6680171643164899, + 0.7680581462162288, + 0.7072290105093693, + 0.7164973767484046, + 0.7520079896468688, + 0.7550619077150706, + 0.7210260497687383 + ], + "population_cv": 0.26063317009417175 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6930886507034302, + "mean_weights": [ + 0.502555787563324, + 0.497444212436676 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929051876068115, + "mean_weights": [ + 0.5103161931037903, + 0.4896838068962097 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928611993789673, + "mean_weights": [ + 0.5053715109825134, + 0.4946284890174866 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927852630615234, + "mean_weights": [ + 0.5126319527626038, + 0.48736804723739624 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928048729896545, + "mean_weights": [ + 0.5031334757804871, + 0.49686652421951294 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927545666694641, + "mean_weights": [ + 0.5128438472747803, + 0.4871561527252197 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6925621032714844, + "mean_weights": [ + 0.5113073587417603, + 0.48869261145591736 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929246187210083, + "mean_weights": [ + 0.5095134973526001, + 0.4904865026473999 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0983444452285767, + "mean_weights": [ + 0.3399057686328888, + 0.3283795118331909, + 0.3317147493362427 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984482765197754, + "mean_weights": [ + 0.3407744765281677, + 0.33049434423446655, + 0.3287311792373657 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0980393886566162, + "mean_weights": [ + 0.3472622334957123, + 0.3314278721809387, + 0.3213099241256714 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985229015350342, + "mean_weights": [ + 0.3373940885066986, + 0.33421406149864197, + 0.3283918499946594 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982658863067627, + "mean_weights": [ + 0.3445737361907959, + 0.33073461055755615, + 0.32469165325164795 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985300540924072, + "mean_weights": [ + 0.3371811807155609, + 0.3333006501197815, + 0.32951819896698 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0983130931854248, + "mean_weights": [ + 0.34229734539985657, + 0.33283179998397827, + 0.32487088441848755 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984888076782227, + "mean_weights": [ + 0.33794093132019043, + 0.33444374799728394, + 0.32761532068252563 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3861021995544434, + "mean_weights": [ + 0.2562229633331299, + 0.25146764516830444, + 0.24576731026172638, + 0.24654211103916168 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861823081970215, + "mean_weights": [ + 0.2537113130092621, + 0.25214654207229614, + 0.24717503786087036, + 0.2469671070575714 + ], + "sources": 4 + }, + { + "entropy_mean": 1.386094093322754, + "mean_weights": [ + 0.2571265399456024, + 0.2500755786895752, + 0.24622224271297455, + 0.24657560884952545 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862230777740479, + "mean_weights": [ + 0.2514685392379761, + 0.2528022825717926, + 0.24948549270629883, + 0.24624371528625488 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3860026597976685, + "mean_weights": [ + 0.25493288040161133, + 0.25381627678871155, + 0.24853172898292542, + 0.2427191287279129 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862276077270508, + "mean_weights": [ + 0.25173667073249817, + 0.25301432609558105, + 0.24867257475852966, + 0.2465764284133911 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859740495681763, + "mean_weights": [ + 0.2546749711036682, + 0.2548622786998749, + 0.24870140850543976, + 0.24176132678985596 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862535953521729, + "mean_weights": [ + 0.2502487599849701, + 0.25214675068855286, + 0.24993090331554413, + 0.24767357110977173 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6090558767318726, + "mean_weights": [ + 0.20371176302433014, + 0.2064536213874817, + 0.19979709386825562, + 0.19547516107559204, + 0.1945623755455017 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609304666519165, + "mean_weights": [ + 0.20178237557411194, + 0.20363003015518188, + 0.20010292530059814, + 0.19683599472045898, + 0.19764868915081024 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6086277961730957, + "mean_weights": [ + 0.20440815389156342, + 0.20845328271389008, + 0.2041691243648529, + 0.19542333483695984, + 0.18754610419273376 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093450784683228, + "mean_weights": [ + 0.20030716061592102, + 0.20382772386074066, + 0.20014742016792297, + 0.19763237237930298, + 0.19808533787727356 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085963249206543, + "mean_weights": [ + 0.20527315139770508, + 0.21035820245742798, + 0.20175111293792725, + 0.19382911920547485, + 0.18878838419914246 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609370470046997, + "mean_weights": [ + 0.2003176212310791, + 0.2033592313528061, + 0.20006224513053894, + 0.19803357124328613, + 0.19822733104228973 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6083728075027466, + "mean_weights": [ + 0.20376041531562805, + 0.21104899048805237, + 0.2038523405790329, + 0.19410842657089233, + 0.18722981214523315 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093378067016602, + "mean_weights": [ + 0.2008873075246811, + 0.20332497358322144, + 0.20115891098976135, + 0.19900420308113098, + 0.19562460482120514 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7908473014831543, + "mean_weights": [ + 0.17166128754615784, + 0.17726510763168335, + 0.16891533136367798, + 0.16100504994392395, + 0.15850719809532166, + 0.16264605522155762 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7916767597198486, + "mean_weights": [ + 0.16709068417549133, + 0.16979600489139557, + 0.1685849130153656, + 0.1652715802192688, + 0.16460880637168884, + 0.16464802622795105 + ], + "sources": 6 + }, + { + "entropy_mean": 1.789864420890808, + "mean_weights": [ + 0.17288848757743835, + 0.18083803355693817, + 0.17338326573371887, + 0.16238118708133698, + 0.15700742602348328, + 0.15350158512592316 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791719913482666, + "mean_weights": [ + 0.16619735956192017, + 0.168890118598938, + 0.16766680777072906, + 0.16658610105514526, + 0.16604891419410706, + 0.16461071372032166 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7891879081726074, + "mean_weights": [ + 0.17246663570404053, + 0.18376711010932922, + 0.17454420030117035, + 0.16218964755535126, + 0.15576335787773132, + 0.15126904845237732 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917189598083496, + "mean_weights": [ + 0.16667470335960388, + 0.16900496184825897, + 0.166001096367836, + 0.16513821482658386, + 0.16654902696609497, + 0.16663199663162231 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7891875505447388, + "mean_weights": [ + 0.17235296964645386, + 0.18330925703048706, + 0.1743936687707901, + 0.1641043871641159, + 0.15562792122364044, + 0.15021178126335144 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079333782196045, + "mean_weights": [ + 0.12415941059589386, + 0.12247946858406067, + 0.12275533378124237, + 0.1243581771850586, + 0.1256372034549713, + 0.12609991431236267, + 0.12752515077590942, + 0.1269853413105011 + ], + "sources": 8 + }, + { + "entropy_mean": 2.19067645072937, + "mean_weights": [ + 0.11858826875686646, + 0.13251525163650513, + 0.12347469478845596, + 0.11762590706348419, + 0.11183994263410568, + 0.10465474426746368, + 0.0985296368598938, + 0.09454642236232758, + 0.09822513163089752 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197180986404419, + "mean_weights": [ + 0.11080950498580933, + 0.11013305932283401, + 0.11013011634349823, + 0.11041422188282013, + 0.11128482222557068, + 0.11163365840911865, + 0.1125442311167717, + 0.11194804310798645, + 0.11110235750675201 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1899502277374268, + "mean_weights": [ + 0.11909156292676926, + 0.1331091821193695, + 0.12361659109592438, + 0.11795277148485184, + 0.11277814209461212, + 0.1057085245847702, + 0.09961889684200287, + 0.0943179503083229, + 0.09380637109279633 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1971864700317383, + "mean_weights": [ + 0.11064367741346359, + 0.11015763878822327, + 0.11035164445638657, + 0.11099889129400253, + 0.1101146712899208, + 0.11141451448202133, + 0.11247441172599792, + 0.11148296296596527, + 0.11236158758401871 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1897201538085938, + "mean_weights": [ + 0.11971956491470337, + 0.13387462496757507, + 0.12342339754104614, + 0.1180819645524025, + 0.11173982918262482, + 0.10558837652206421, + 0.10011725127696991, + 0.09448471665382385, + 0.09297028183937073 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1971964836120605, + "mean_weights": [ + 0.11113918572664261, + 0.11165578663349152, + 0.11136722564697266, + 0.11112368106842041, + 0.11216380447149277, + 0.11136430501937866, + 0.1116628348827362, + 0.10975746810436249, + 0.10976571589708328 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1886677742004395, + "mean_weights": [ + 0.11989010870456696, + 0.13561508059501648, + 0.1242879331111908, + 0.1177452951669693, + 0.11283841729164124, + 0.1049613356590271, + 0.09941282868385315, + 0.09397745132446289, + 0.0912715420126915 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028571274131536484, + 0.014370470307767391, + 0.015031064860522747, + 0.014924541115760803, + 0.01576058566570282, + 0.015609913505613804, + 0.01673153042793274, + 0.01674462854862213, + 0.01786557026207447, + 0.012058191932737827, + 0.013009775429964066, + 0.013223569840192795, + 0.014060634188354015, + 0.01422955747693777, + 0.015225647017359734, + 0.015402527526021004, + 0.016711384057998657, + 0.012696896679699421, + 0.013651188462972641, + 0.01374440360814333, + 0.014770817942917347, + 0.01499935146421194, + 0.01605224795639515, + 0.016234049573540688, + 0.017371296882629395, + 0.014139331877231598, + 0.0150116803124547, + 0.015255129896104336, + 0.01619553007185459, + 0.01641164906322956, + 0.017412541434168816, + 0.017549853771924973, + 0.01851520501077175, + 0.0156314168125391, + 0.01649715006351471, + 0.016789749264717102, + 0.01767360419034958, + 0.017833195626735687, + 0.018907228484749794, + 0.019026586785912514, + 0.02030135691165924, + 0.017719922587275505, + 0.018491322174668312, + 0.018898772075772285, + 0.019703106954693794, + 0.020232124254107475, + 0.021055491641163826, + 0.021651869639754295, + 0.02249983325600624, + 0.02048179879784584, + 0.021194644272327423, + 0.021763430908322334, + 0.02249440737068653, + 0.023286480456590652, + 0.024062326177954674, + 0.02487023174762726, + 0.02590898610651493, + 0.022659940645098686, + 0.024567829445004463, + 0.024109894409775734, + 0.02627130225300789, + 0.02551914006471634, + 0.02790611796081066, + 0.02684609591960907 + ], + "loss_nats": 3.570737838745117, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1932034492492676, + "mean_weights": [ + 0.10075974464416504, + 0.09453918039798737, + 0.10217029601335526, + 0.10980333387851715, + 0.11425476521253586, + 0.11821146309375763, + 0.11953863501548767, + 0.12103148549795151, + 0.11969110369682312 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.0029127895832061768, + 0.0058692265301942825, + 0.006536925211548805, + 0.010152247734367847, + 0.010624829679727554, + 0.014839091338217258, + 0.0151302395388484, + 0.019591957330703735, + 0.003085965756326914, + 0.0067398143000900745, + 0.007633335888385773, + 0.011955589056015015, + 0.012725084088742733, + 0.01716931350529194, + 0.01780773513019085, + 0.02306033857166767, + 0.003751161741092801, + 0.007508765906095505, + 0.0091110710054636, + 0.013779806904494762, + 0.015157311223447323, + 0.020409800112247467, + 0.02179115265607834, + 0.02685040980577469, + 0.005485833622515202, + 0.0091559374704957, + 0.011829552240669727, + 0.016716359183192253, + 0.018926752731204033, + 0.02419254370033741, + 0.02609706111252308, + 0.03126180171966553, + 0.005563640967011452, + 0.009785215370357037, + 0.01357173826545477, + 0.01850821264088154, + 0.021842118352651596, + 0.02761545218527317, + 0.03138817846775055, + 0.037425313144922256, + 0.0077114407904446125, + 0.012438409961760044, + 0.017470013350248337, + 0.023153970018029213, + 0.028209656476974487, + 0.03432171046733856, + 0.04007846489548683, + 0.04660208150744438, + 0.01271942537277937, + 0.017327340319752693, + 0.02364915981888771, + 0.02931905910372734, + 0.03804082050919533, + 0.044478677213191986, + 0.05299994722008705, + 0.06014470010995865, + 0.013551505282521248, + 0.019144421443343163, + 0.031610529869794846, + 0.03802234306931496, + 0.05028318613767624, + 0.05717705935239792, + 0.06944417208433151, + 0.07674983143806458 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00013502020738087595, + 0.0001386140938848257, + 0.00013852481788489968, + 0.00013219378888607025, + 0.00013201223919168115, + 0.00011907621228601784, + 0.00010749710054369643, + 9.981176117435098e-05, + 9.781593689695e-05, + 8.850422454997897e-05, + 8.08831537142396e-05, + 7.543387619080022e-05, + 7.733904203632846e-05, + 7.133937469916418e-05, + 6.59702272969298e-05, + 6.017514533596113e-05, + 5.727267125621438e-05, + 5.206368223298341e-05, + 4.759432704304345e-05, + 4.3677500798366964e-05, + 3.97721778426785e-05, + 3.6940233258064836e-05, + 3.430873766774312e-05, + 3.225093678338453e-05, + 2.9220358555903658e-05, + 2.7307431082590483e-05, + 2.608104477985762e-05, + 2.5369541617692448e-05, + 2.8449645469663665e-05, + 2.820042209350504e-05, + 2.805530675686896e-05, + 2.801550544972997e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00012534377765405225, + "first_to_last_ratio": 4.543514284047766, + "imbalance_abs_log_ratio": 1.5137007839906835, + "last_quartile_mean": 2.758740697572648e-05, + "mean": 6.827471014503317e-05, + "normalized": [ + 1.9776020592992347, + 2.03024067902117, + 2.0289330791831572, + 1.9362043222923453, + 1.9335452162484903, + 1.7440749588400906, + 1.574479046712095, + 1.4619140961905945, + 1.432681833275654, + 1.2962958665364244, + 1.1846722387018975, + 1.1048582415151837, + 1.13276265650986, + 1.0448872583658118, + 0.9662469039676911, + 0.8813680088591155, + 0.8388563076218469, + 0.7625617468369571, + 0.6971003896163128, + 0.6397317645960655, + 0.5825316249339191, + 0.5410529488824517, + 0.5025101914729843, + 0.4723701750600653, + 0.42798216929565935, + 0.39996407197602785, + 0.38200154529334107, + 0.3715803635606943, + 0.41669375687175014, + 0.413043453917271, + 0.41091799140958957, + 0.41033503313624886 + ], + "population_cv": 0.5682770161254035 + }, + "activation_output_rms_by_block": [ + 0.021690133959054947, + 0.038645509630441666, + 0.050416164100170135, + 0.059038665145635605, + 0.015957212075591087, + 0.0259401835501194, + 0.03326375409960747, + 0.03896600753068924, + 0.01569177769124508, + 0.02718697488307953, + 0.03437623754143715, + 0.03927858546376228, + 0.0174946840852499, + 0.027429770678281784, + 0.036497704684734344, + 0.042413465678691864, + 0.01978003978729248, + 0.02902580052614212, + 0.04184097424149513, + 0.049806781113147736, + 0.02072436735033989, + 0.035371601581573486, + 0.05015818774700165, + 0.06557824462652206, + 0.024349043145775795, + 0.0433179996907711, + 0.06327643990516663, + 0.08553080260753632, + 0.032196998596191406, + 0.06154516339302063, + 0.0982988178730011, + 0.1370837539434433 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.03548970376141369, + "first_to_last_ratio": 0.5203778234995319, + "imbalance_abs_log_ratio": 0.6532001475878298, + "last_quartile_mean": 0.06819987739436328, + "mean": 0.043192870216444135, + "normalized": [ + 0.5021693129065826, + 0.894719647867457, + 1.1672334773662711, + 1.366861355816052, + 0.3694408821554987, + 0.600566330047768, + 0.770121409689127, + 0.9021398053758959, + 0.36329555347009557, + 0.6294320045609996, + 0.7958775920464214, + 0.9093766000484119, + 0.4050363867365644, + 0.6350532053282925, + 0.8449937339621194, + 0.9819552501640527, + 0.45794687151311236, + 0.6720044391745837, + 0.9687009460548811, + 1.1531250612325734, + 0.4798099141475861, + 0.8189222296254584, + 1.1612608167888254, + 1.5182654984006019, + 0.5637282964470782, + 1.002896993732992, + 1.4649741864358992, + 1.9802065058175629, + 0.7454239191526002, + 1.4248917259865153, + 2.2758112017195224, + 3.1737588462285977 + ], + "population_cv": 0.5898779055306158 + }, + "bits_per_byte": 3.0783016453895327, + "branch_output_rms_by_sublayer": [ + 0.010510989464819431, + 0.01773845963180065, + 0.01288721151649952, + 0.015410542488098145, + 0.013293960131704807, + 0.011829170398414135, + 0.012563494965434074, + 0.010719931684434414, + 0.012978821992874146, + 0.008852306753396988, + 0.012484307400882244, + 0.008082501590251923, + 0.017479149624705315, + 0.007394776213914156, + 0.013811159878969193, + 0.008539069443941116, + 0.012748918496072292, + 0.008203585632145405, + 0.015691176056861877, + 0.00825700256973505, + 0.019360050559043884, + 0.008359342813491821, + 0.01773277297616005, + 0.008825630880892277, + 0.014515543356537819, + 0.008455797098577023, + 0.014256616123020649, + 0.009114319458603859, + 0.017845245078206062, + 0.009323488920927048, + 0.011321012862026691, + 0.00932974461466074, + 0.01630145125091076, + 0.010338325053453445, + 0.014692036435008049, + 0.01085656601935625, + 0.022584766149520874, + 0.012223388068377972, + 0.021950112655758858, + 0.014179684221744537, + 0.015464341267943382, + 0.013388797640800476, + 0.016217676922678947, + 0.014884707517921925, + 0.019293656572699547, + 0.01591324806213379, + 0.015093626454472542, + 0.018570836633443832, + 0.016100121662020683, + 0.01750165969133377, + 0.011022309772670269, + 0.01947851851582527, + 0.008948881179094315, + 0.021226298063993454, + 0.007377106696367264, + 0.02357090450823307, + 0.007519193924963474, + 0.02960866689682007, + 0.008151640184223652, + 0.02826736681163311, + 0.009839511476457119, + 0.034673310816287994, + 0.011650451458990574, + 0.03381374478340149 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.000534027543251826, + 0.0003551799374971097, + 0.0003169502727458517, + 0.00031266717764008215, + 0.0002927028358434817, + 0.00025320354741983973, + 0.00021793521292911338, + 0.0002553923951558317, + 0.0002247035985789251, + 0.0001890734727443061, + 0.00017489844348413705, + 0.00016649772261059797, + 0.0001703301962196103, + 0.00015541374025934292, + 0.00014978136198409063, + 0.00013453045687611686, + 0.00014882006004230677, + 0.00013684889755545492, + 0.00014149569788357677, + 0.00013256541433128886, + 0.00011810236114568028, + 0.00011399650055164618, + 0.00011714792127563267, + 0.0001104246640026569, + 0.0001026434139331144, + 9.221143034651731e-05, + 8.781994013379414e-05, + 7.90832897193947e-05, + 0.00010462192413175276, + 9.52877304114312e-05, + 0.00011873406905510886, + 0.0001114991774094402 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.000317257365310392, + "first_to_last_ratio": 3.20502057979239, + "imbalance_abs_log_ratio": 1.1647185115040897, + "last_quartile_mean": 9.89876218925692e-05, + "mean": 0.00017858095022403313, + "normalized": [ + 2.9903947906082844, + 1.9889015992552947, + 1.774826891380249, + 1.7508428376477747, + 1.6390484846019722, + 1.417864263249754, + 1.2203721206305018, + 1.4301211570183558, + 1.2582730593438451, + 1.058754993223577, + 0.979379061790878, + 0.9323375332123807, + 0.9537982411109802, + 0.8702705415352169, + 0.8387309049267972, + 0.753330389984743, + 0.8333479010813261, + 0.7663129655418197, + 0.7923336599232325, + 0.7423267384622104, + 0.6613379590461282, + 0.6383463656601415, + 0.6559933807534813, + 0.6183451474758485, + 0.5747724704362158, + 0.5163564771653211, + 0.49176544319885407, + 0.44284280949442395, + 0.5858515367988725, + 0.533582838997614, + 0.6648753347216193, + 0.6243621017222857 + ], + "population_cv": 0.5428104396751369 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6472191214561462, + "mean_weights": [ + 0.3721027076244354, + 0.627897322177887 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6685923337936401, + "mean_weights": [ + 0.4305521249771118, + 0.5694478750228882 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6887905597686768, + "mean_weights": [ + 0.5073041915893555, + 0.49269580841064453 + ], + "sources": 2 + }, + { + "entropy_mean": 0.674967885017395, + "mean_weights": [ + 0.4601387083530426, + 0.539861261844635 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6842271089553833, + "mean_weights": [ + 0.46463143825531006, + 0.5353685021400452 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6764289140701294, + "mean_weights": [ + 0.45219624042510986, + 0.5478037595748901 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6803587675094604, + "mean_weights": [ + 0.4517097771167755, + 0.5482901930809021 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6795487403869629, + "mean_weights": [ + 0.4564407467842102, + 0.543559193611145 + ], + "sources": 2 + }, + { + "entropy_mean": 1.09000825881958, + "mean_weights": [ + 0.3023155927658081, + 0.3300391733646393, + 0.367645263671875 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0830318927764893, + "mean_weights": [ + 0.3015379011631012, + 0.3218569755554199, + 0.3766051232814789 + ], + "sources": 3 + }, + { + "entropy_mean": 1.092214822769165, + "mean_weights": [ + 0.30797675251960754, + 0.3204318583011627, + 0.37159138917922974 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0806152820587158, + "mean_weights": [ + 0.31027084589004517, + 0.3089178800582886, + 0.38081130385398865 + ], + "sources": 3 + }, + { + "entropy_mean": 1.088780164718628, + "mean_weights": [ + 0.29294297099113464, + 0.3224499821662903, + 0.38460707664489746 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0694758892059326, + "mean_weights": [ + 0.2983662486076355, + 0.3019666075706482, + 0.3996671736240387 + ], + "sources": 3 + }, + { + "entropy_mean": 1.090681791305542, + "mean_weights": [ + 0.3049081563949585, + 0.31428080797195435, + 0.38081103563308716 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0703673362731934, + "mean_weights": [ + 0.29188719391822815, + 0.30882728099823, + 0.39928552508354187 + ], + "sources": 3 + }, + { + "entropy_mean": 1.375261664390564, + "mean_weights": [ + 0.22141581773757935, + 0.22935809195041656, + 0.2638857364654541, + 0.2853403687477112 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3544485569000244, + "mean_weights": [ + 0.22965481877326965, + 0.2214069962501526, + 0.2419331669807434, + 0.30700498819351196 + ], + "sources": 4 + }, + { + "entropy_mean": 1.375445008277893, + "mean_weights": [ + 0.22887249290943146, + 0.22569704055786133, + 0.24806134402751923, + 0.297369122505188 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3552662134170532, + "mean_weights": [ + 0.2435493767261505, + 0.22663038969039917, + 0.22198355197906494, + 0.3078366816043854 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3799614906311035, + "mean_weights": [ + 0.2553842067718506, + 0.22196413576602936, + 0.24428126215934753, + 0.27837038040161133 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3527708053588867, + "mean_weights": [ + 0.25185543298721313, + 0.22270265221595764, + 0.2234145700931549, + 0.3020273447036743 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3809083700180054, + "mean_weights": [ + 0.2529875934123993, + 0.2247350960969925, + 0.24474212527275085, + 0.27753520011901855 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3513743877410889, + "mean_weights": [ + 0.24927198886871338, + 0.2232775092124939, + 0.2234012484550476, + 0.3040492534637451 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6052519083023071, + "mean_weights": [ + 0.20885375142097473, + 0.18223194777965546, + 0.1860320121049881, + 0.20962435007095337, + 0.21325793862342834 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5736613273620605, + "mean_weights": [ + 0.20247238874435425, + 0.1819916069507599, + 0.1644536554813385, + 0.20117707550525665, + 0.2499052882194519 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6082277297973633, + "mean_weights": [ + 0.19722914695739746, + 0.19568710029125214, + 0.1940370351076126, + 0.20102860033512115, + 0.21201811730861664 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5787183046340942, + "mean_weights": [ + 0.20195573568344116, + 0.1856016367673874, + 0.17212294042110443, + 0.1921299397945404, + 0.24818973243236542 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6036648750305176, + "mean_weights": [ + 0.20087790489196777, + 0.19327759742736816, + 0.18064075708389282, + 0.1987924724817276, + 0.22641125321388245 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5773308277130127, + "mean_weights": [ + 0.19965466856956482, + 0.19064198434352875, + 0.17014144361019135, + 0.18579250574111938, + 0.2537693977355957 + ], + "sources": 5 + }, + { + "entropy_mean": 1.596920132637024, + "mean_weights": [ + 0.20649376511573792, + 0.19138765335083008, + 0.17314665019512177, + 0.18983504176139832, + 0.23913687467575073 + ], + "sources": 5 + }, + { + "entropy_mean": 1.574884295463562, + "mean_weights": [ + 0.20621415972709656, + 0.19019779562950134, + 0.16562730073928833, + 0.17967543005943298, + 0.2582853138446808 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7829394340515137, + "mean_weights": [ + 0.15515035390853882, + 0.167438805103302, + 0.150580033659935, + 0.15603643655776978, + 0.18237966299057007, + 0.18841469287872314 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7675386667251587, + "mean_weights": [ + 0.16129285097122192, + 0.15975908935070038, + 0.14229260385036469, + 0.14818856120109558, + 0.18669070303440094, + 0.2017761766910553 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7799508571624756, + "mean_weights": [ + 0.15204058587551117, + 0.16960464417934418, + 0.15014684200286865, + 0.15069472789764404, + 0.17520248889923096, + 0.2023106813430786 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7687162160873413, + "mean_weights": [ + 0.15768879652023315, + 0.16014868021011353, + 0.14229220151901245, + 0.14887431263923645, + 0.17495375871658325, + 0.21604226529598236 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7769694328308105, + "mean_weights": [ + 0.1434900015592575, + 0.17719288170337677, + 0.15652287006378174, + 0.15137143433094025, + 0.17447316646575928, + 0.19694967567920685 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7790402173995972, + "mean_weights": [ + 0.1672755777835846, + 0.16318361461162567, + 0.14764192700386047, + 0.15013208985328674, + 0.1656753122806549, + 0.20609146356582642 + ], + "sources": 6 + }, + { + "entropy_mean": 1.772436261177063, + "mean_weights": [ + 0.137189120054245, + 0.1825355887413025, + 0.15700370073318481, + 0.15137675404548645, + 0.17622745037078857, + 0.1956673562526703 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0557358264923096, + "mean_weights": [ + 0.11572219431400299, + 0.14279131591320038, + 0.13401660323143005, + 0.1240810751914978, + 0.12964043021202087, + 0.13645057380199432, + 0.10849803686141968, + 0.1087997779250145 + ], + "sources": 8 + }, + { + "entropy_mean": 2.128070592880249, + "mean_weights": [ + 0.08253289759159088, + 0.16997724771499634, + 0.15496571362018585, + 0.12397155165672302, + 0.11783402413129807, + 0.10891544073820114, + 0.08614566922187805, + 0.07161472737789154, + 0.08404272049665451 + ], + "sources": 9 + }, + { + "entropy_mean": 2.190647602081299, + "mean_weights": [ + 0.111246258020401, + 0.11339963972568512, + 0.10967698693275452, + 0.10811547189950943, + 0.11347523331642151, + 0.11441484093666077, + 0.10874959081411362, + 0.1088113933801651, + 0.11211058497428894 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1061489582061768, + "mean_weights": [ + 0.0818711519241333, + 0.18386806547641754, + 0.16306090354919434, + 0.1236560046672821, + 0.11738234013319016, + 0.10990595817565918, + 0.08861973881721497, + 0.07051397860050201, + 0.061121866106987 + ], + "sources": 9 + }, + { + "entropy_mean": 2.174285888671875, + "mean_weights": [ + 0.10868722200393677, + 0.129492849111557, + 0.11783719062805176, + 0.11122089624404907, + 0.11352718621492386, + 0.12009987980127335, + 0.09868897497653961, + 0.10157625377178192, + 0.09886953234672546 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0911030769348145, + "mean_weights": [ + 0.07921186089515686, + 0.1853443682193756, + 0.17000551521778107, + 0.12975995242595673, + 0.12159739434719086, + 0.11123217642307281, + 0.08325955271720886, + 0.0661969929933548, + 0.05339217931032181 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1573359966278076, + "mean_weights": [ + 0.10092437267303467, + 0.1396944522857666, + 0.12192441523075104, + 0.12009409070014954, + 0.13081315159797668, + 0.12593936920166016, + 0.09386305510997772, + 0.08600614964962006, + 0.08074095100164413 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0948352813720703, + "mean_weights": [ + 0.08049570769071579, + 0.187056303024292, + 0.1619722694158554, + 0.1283482015132904, + 0.12387267500162125, + 0.11433333158493042, + 0.08365806937217712, + 0.06816720962524414, + 0.05209621414542198 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.043896060436964035, + 0.017306910827755928, + 0.0239819698035717, + 0.027399448677897453, + 0.030875300988554955, + 0.03204025700688362, + 0.03548785299062729, + 0.03623313829302788, + 0.03888553008437157, + 0.025523660704493523, + 0.027128782123327255, + 0.027281278744339943, + 0.028869546949863434, + 0.029556388035416603, + 0.03074541874229908, + 0.03038080967962742, + 0.03226381912827492, + 0.02304573729634285, + 0.024761522188782692, + 0.02376183681190014, + 0.025758439674973488, + 0.024560648947954178, + 0.026881994679570198, + 0.025268739089369774, + 0.027781862765550613, + 0.020820777863264084, + 0.02243473008275032, + 0.02128320187330246, + 0.022841596975922585, + 0.02157461643218994, + 0.023407133296132088, + 0.02214578352868557, + 0.023906925693154335, + 0.018881559371948242, + 0.019972745329141617, + 0.019260240718722343, + 0.020448695868253708, + 0.020204225555062294, + 0.021090438589453697, + 0.020976155996322632, + 0.021342279389500618, + 0.018290864303708076, + 0.019086189568042755, + 0.019081290811300278, + 0.020024435594677925, + 0.020085478201508522, + 0.021220820024609566, + 0.02111699804663658, + 0.022582143545150757, + 0.019801219925284386, + 0.02100391499698162, + 0.021013697609305382, + 0.022442955523729324, + 0.022457340732216835, + 0.02410208247601986, + 0.024221593514084816, + 0.025193551555275917, + 0.021587364375591278, + 0.02587711811065674, + 0.02333785593509674, + 0.027406372129917145, + 0.02415860816836357, + 0.028257466852664948, + 0.02561909332871437 + ], + "loss_nats": 2.133716106414795, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.174569606781006, + "mean_weights": [ + 0.12282125651836395, + 0.09209376573562622, + 0.09000569581985474, + 0.09502357244491577, + 0.09669246524572372, + 0.10431371629238129, + 0.12377016246318817, + 0.1270047426223755, + 0.14827463030815125 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.010510989464819431, + 0.021690133959054947, + 0.02805406041443348, + 0.038645509630441666, + 0.04236667975783348, + 0.050416164100170135, + 0.05213770642876625, + 0.059038665145635605, + 0.012978821992874146, + 0.015957212075591087, + 0.02230519987642765, + 0.0259401835501194, + 0.029941538348793983, + 0.03326375409960747, + 0.03444092720746994, + 0.03896600753068924, + 0.012748918496072292, + 0.01569177769124508, + 0.02370862103998661, + 0.02718697488307953, + 0.03030574880540371, + 0.03437623754143715, + 0.034412682056427, + 0.03927858546376228, + 0.014515543356537819, + 0.0174946840852499, + 0.02315252088010311, + 0.027429770678281784, + 0.03182410076260567, + 0.036497704684734344, + 0.03731905296444893, + 0.042413465678691864, + 0.01630145125091076, + 0.01978003978729248, + 0.023382632061839104, + 0.02902580052614212, + 0.035387299954891205, + 0.04184097424149513, + 0.04106847196817398, + 0.049806781113147736, + 0.015464341267943382, + 0.02072436735033989, + 0.026616448536515236, + 0.035371601581573486, + 0.03925501927733421, + 0.05015818774700165, + 0.05158501863479614, + 0.06557824462652206, + 0.016100121662020683, + 0.024349043145775795, + 0.029167983680963516, + 0.0433179996907711, + 0.04585959017276764, + 0.06327643990516663, + 0.06569604575634003, + 0.08553080260753632, + 0.007519193924963474, + 0.032196998596191406, + 0.03536655753850937, + 0.06154516339302063, + 0.06592655926942825, + 0.0982988178730011, + 0.10538331419229507, + 0.1370837539434433 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.912946603028104e-05, + 8.98468351806514e-05, + 8.289486868306994e-05, + 7.69269245211035e-05, + 8.701786282472312e-05, + 8.263266499852762e-05, + 7.924547389848158e-05, + 7.185961294453591e-05, + 6.937592843314633e-05, + 6.650048453593627e-05, + 6.397806282620877e-05, + 6.214729364728555e-05, + 6.411123467842117e-05, + 6.199828203534707e-05, + 5.916267400607467e-05, + 5.662695912178606e-05, + 5.848028013133444e-05, + 5.5864726164145395e-05, + 5.7232024119002745e-05, + 5.329703708412126e-05, + 6.049815419828519e-05, + 5.729254553443752e-05, + 5.153644451638684e-05, + 4.557902502710931e-05, + 5.965012314845808e-05, + 5.673339182976633e-05, + 5.410864105215296e-05, + 5.2005798352183774e-05, + 4.5126816985430196e-05, + 3.954961721319705e-05, + 3.792164716287516e-05, + 3.6557423300109804e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 8.119421363517176e-05, + "first_to_last_ratio": 1.7019463434397786, + "imbalance_abs_log_ratio": 0.5317725040606909, + "last_quartile_mean": 4.770668238052167e-05, + "mean": 6.1715260130768e-05, + "normalized": [ + 1.282170177402059, + 1.4558285096793826, + 1.3431826829770244, + 1.2464814108877387, + 1.4099894035987472, + 1.3389340792445479, + 1.2840499048463692, + 1.1643734919414277, + 1.1241292394481721, + 1.077537134171174, + 1.0366652055042163, + 1.007000432560798, + 1.038823048668617, + 1.0045859306754825, + 0.9586393038077669, + 0.9175519798798485, + 0.9475821702350604, + 0.9052011778897804, + 0.9273561190171481, + 0.863595761748238, + 0.9802786874769078, + 0.9283367746168578, + 0.8350680918655556, + 0.7385373557614803, + 0.9665376605731854, + 0.9192765567147311, + 0.8767465443312168, + 0.8426732422741001, + 0.731210026334027, + 0.6408401605923019, + 0.6144614327562302, + 0.5923563025198072 + ], + "population_cv": 0.22289232151468405 + }, + "activation_output_rms_by_block": [ + 0.07933542132377625, + 0.17356063425540924, + 0.19592563807964325, + 0.20413769781589508, + 0.0703900009393692, + 0.08005841821432114, + 0.0889945924282074, + 0.10390866547822952, + 0.043775007128715515, + 0.06786487251520157, + 0.09043840318918228, + 0.11542254686355591, + 0.03245418146252632, + 0.05422379821538925, + 0.0745156928896904, + 0.09330973774194717, + 0.03882657364010811, + 0.06532005220651627, + 0.0864439532160759, + 0.11046812683343887, + 0.0477026104927063, + 0.08639359474182129, + 0.11987065523862839, + 0.16759569942951202, + 0.06834843754768372, + 0.10255371779203415, + 0.1335299015045166, + 0.16770142316818237, + 0.060653913766145706, + 0.1185198500752449, + 0.19719740748405457, + 0.280917763710022 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.12453888356685638, + "first_to_last_ratio": 0.8821421066737113, + "imbalance_abs_log_ratio": 0.12540211727588868, + "last_quartile_mean": 0.1411778018809855, + "mean": 0.10688621841836721, + "normalized": [ + 0.7422418202994759, + 1.6237887056315277, + 1.8330299357468485, + 1.909859856926291, + 0.65855076529935, + 0.7490059934664504, + 0.8326105436705642, + 0.972142779638034, + 0.4095477207115157, + 0.6349263124790253, + 0.8461184662291453, + 1.0798636959142511, + 0.3036329841467142, + 0.5073039257797476, + 0.6971496792788181, + 0.8729819346468051, + 0.3632514482656278, + 0.6111176274460819, + 0.8087474184718791, + 1.033511414923967, + 0.4462933687670733, + 0.8082762775240584, + 1.121479055133547, + 1.5679823078174555, + 0.6394504226930232, + 0.9594662371777898, + 1.2492714540789758, + 1.568971431955579, + 0.5674624349487043, + 1.1088412690524991, + 1.844928283571574, + 2.628194428307601 + ], + "population_cv": 0.5270692473404438 + }, + "bits_per_byte": 2.066011762411041, + "branch_output_rms_by_sublayer": [ + 0.04519475996494293, + 0.06201421841979027, + 0.11058512330055237, + 0.024957187473773956, + 0.04467621445655823, + 0.007390463724732399, + 0.03156944364309311, + 0.015626417472958565, + 0.06896122545003891, + 0.009649131447076797, + 0.02159368060529232, + 0.00823317188769579, + 0.02192658744752407, + 0.01007626112550497, + 0.03447022661566734, + 0.017795119434595108, + 0.039527297019958496, + 0.016687674447894096, + 0.027605466544628143, + 0.01717737689614296, + 0.024575263261795044, + 0.01972619630396366, + 0.02325247973203659, + 0.019825972616672516, + 0.02363201417028904, + 0.01986200176179409, + 0.02095704898238182, + 0.022178664803504944, + 0.02059737965464592, + 0.02332259714603424, + 0.019563060253858566, + 0.025621045380830765, + 0.027532115578651428, + 0.02510117180645466, + 0.030061373487114906, + 0.027030576020479202, + 0.02861795388162136, + 0.02360380068421364, + 0.0402015745639801, + 0.030928917229175568, + 0.043986596167087555, + 0.023246120661497116, + 0.06151795759797096, + 0.023922353982925415, + 0.06635378301143646, + 0.030496269464492798, + 0.07754725962877274, + 0.05318857729434967, + 0.04700809717178345, + 0.04513707384467125, + 0.0281364768743515, + 0.04732036218047142, + 0.01650506816804409, + 0.05235907435417175, + 0.011390995234251022, + 0.060677725821733475, + 0.013558072037994862, + 0.05817825347185135, + 0.010303095914423466, + 0.06757662445306778, + 0.0185127891600132, + 0.0921226441860199, + 0.017877288162708282, + 0.09247501194477081 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0002499867152056645, + 0.00022023184426304317, + 0.00012862347080347888, + 0.00013874508711477122, + 0.00016561428986663383, + 0.00010237510824701164, + 0.00011183016649318567, + 0.00018676622342607996, + 0.00015642584072286578, + 0.0001222540107587513, + 0.00012523380872062526, + 0.00012743167233473566, + 0.00013086282613597257, + 0.00013887000222485716, + 0.0001429926281348834, + 0.00015194658148812596, + 0.0001557433490681304, + 0.00016747093812728608, + 0.00015864942110219157, + 0.00019577301560792646, + 0.00019437764302629505, + 0.00022437945703715723, + 0.00027218714798262134, + 0.0002948071371943306, + 0.00028787668401285504, + 0.00025299422855138884, + 0.0002433303889943037, + 0.00023362869890304787, + 0.00021691172073067734, + 0.0001988072477910395, + 0.00019662747032228586, + 0.00019539784029560395 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0001630216131774836, + "first_to_last_ratio": 0.7143904907034329, + "imbalance_abs_log_ratio": 0.3363255603938907, + "last_quartile_mean": 0.00022819678495015027, + "mean": 0.00018403602077149458, + "normalized": [ + 1.358357533257343, + 1.1966779293521597, + 0.6989037812504226, + 0.7539017988605713, + 0.8999014930466586, + 0.5562775581532708, + 0.6076536866227826, + 1.0148351537005644, + 0.849974043489505, + 0.6642939259730358, + 0.6804853104062701, + 0.6924278834139714, + 0.7110718085915166, + 0.7545805524521904, + 0.7769817426797547, + 0.8256350080334982, + 0.8462655757021973, + 0.9099899977473634, + 0.8620563541697966, + 1.063775530394699, + 1.0561934680583047, + 1.2192148911747795, + 1.4789884439013068, + 1.6018991062644918, + 1.5642409719904375, + 1.3746995152949713, + 1.3221889278753263, + 1.269472671293677, + 1.1786373114424287, + 1.080262695083401, + 1.0684183971051258, + 1.06173693321818 + ], + "population_cv": 0.28586106954832347 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.4698029160499573, + "mean_weights": [ + 0.8109943866729736, + 0.18900558352470398 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6435889005661011, + "mean_weights": [ + 0.5100059509277344, + 0.4899940490722656 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5176190733909607, + "mean_weights": [ + 0.7797738909721375, + 0.22022610902786255 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6866215467453003, + "mean_weights": [ + 0.4749618172645569, + 0.5250381827354431 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6593912839889526, + "mean_weights": [ + 0.55280601978302, + 0.4471939504146576 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6897172927856445, + "mean_weights": [ + 0.5164542198181152, + 0.48354578018188477 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6356666684150696, + "mean_weights": [ + 0.6334895491600037, + 0.36651045083999634 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6826819777488708, + "mean_weights": [ + 0.5319589376449585, + 0.4680410921573639 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0528697967529297, + "mean_weights": [ + 0.35035771131515503, + 0.23919537663459778, + 0.4104468822479248 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0909645557403564, + "mean_weights": [ + 0.3614423871040344, + 0.28591710329055786, + 0.3526404798030853 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0500558614730835, + "mean_weights": [ + 0.3499464988708496, + 0.22412750124931335, + 0.4259260296821594 + ], + "sources": 3 + }, + { + "entropy_mean": 1.086782693862915, + "mean_weights": [ + 0.3487471342086792, + 0.2704853415489197, + 0.3807675242424011 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0477923154830933, + "mean_weights": [ + 0.29735344648361206, + 0.26350510120391846, + 0.4391414523124695 + ], + "sources": 3 + }, + { + "entropy_mean": 1.083518147468567, + "mean_weights": [ + 0.3840525150299072, + 0.25850173830986023, + 0.35744574666023254 + ], + "sources": 3 + }, + { + "entropy_mean": 1.041725993156433, + "mean_weights": [ + 0.3748757243156433, + 0.20431557297706604, + 0.42080870270729065 + ], + "sources": 3 + }, + { + "entropy_mean": 1.088074803352356, + "mean_weights": [ + 0.37115857005119324, + 0.27379852533340454, + 0.35504287481307983 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3461657762527466, + "mean_weights": [ + 0.24980032444000244, + 0.17032209038734436, + 0.26108914613723755, + 0.31878846883773804 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3758656978607178, + "mean_weights": [ + 0.28765738010406494, + 0.2253328114748001, + 0.21813109517097473, + 0.2688787281513214 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3665999174118042, + "mean_weights": [ + 0.2572351098060608, + 0.2049572765827179, + 0.26986706256866455, + 0.26794055104255676 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3734302520751953, + "mean_weights": [ + 0.2780992090702057, + 0.21354396641254425, + 0.21350377798080444, + 0.2948530316352844 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3616993427276611, + "mean_weights": [ + 0.3108273148536682, + 0.20910179615020752, + 0.2097688913345337, + 0.2703019976615906 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3674063682556152, + "mean_weights": [ + 0.29994261264801025, + 0.20192782580852509, + 0.21637186408042908, + 0.2817577123641968 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3652420043945312, + "mean_weights": [ + 0.31152600049972534, + 0.20496675372123718, + 0.22167494893074036, + 0.2618322968482971 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3603029251098633, + "mean_weights": [ + 0.2523220181465149, + 0.19280630350112915, + 0.22273331880569458, + 0.33213838934898376 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5933597087860107, + "mean_weights": [ + 0.23010152578353882, + 0.1523759663105011, + 0.1853415071964264, + 0.21938833594322205, + 0.21279269456863403 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5951242446899414, + "mean_weights": [ + 0.20269069075584412, + 0.16687172651290894, + 0.1650846004486084, + 0.22331196069717407, + 0.24204102158546448 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6055580377578735, + "mean_weights": [ + 0.19624704122543335, + 0.18407827615737915, + 0.19682693481445312, + 0.21266505122184753, + 0.21018269658088684 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5947179794311523, + "mean_weights": [ + 0.19966183602809906, + 0.18184176087379456, + 0.15833750367164612, + 0.2062276005744934, + 0.25393128395080566 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5722358226776123, + "mean_weights": [ + 0.17954319715499878, + 0.15006548166275024, + 0.1787385493516922, + 0.22400569915771484, + 0.2676470875740051 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5913262367248535, + "mean_weights": [ + 0.19785037636756897, + 0.18112456798553467, + 0.15975478291511536, + 0.19423411786556244, + 0.2670361399650574 + ], + "sources": 5 + }, + { + "entropy_mean": 1.568877935409546, + "mean_weights": [ + 0.18879836797714233, + 0.15164746344089508, + 0.17191189527511597, + 0.2049403190612793, + 0.2827019691467285 + ], + "sources": 5 + }, + { + "entropy_mean": 1.573760747909546, + "mean_weights": [ + 0.19746656715869904, + 0.16800090670585632, + 0.13806992769241333, + 0.21452516317367554, + 0.28193745017051697 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7648463249206543, + "mean_weights": [ + 0.14480476081371307, + 0.13240447640419006, + 0.16316933929920197, + 0.17283213138580322, + 0.22918567061424255, + 0.15760360658168793 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7635750770568848, + "mean_weights": [ + 0.16815733909606934, + 0.14792580902576447, + 0.11194154620170593, + 0.14750641584396362, + 0.20471270382404327, + 0.21975621581077576 + ], + "sources": 6 + }, + { + "entropy_mean": 1.763267159461975, + "mean_weights": [ + 0.13464021682739258, + 0.13171157240867615, + 0.1645362675189972, + 0.15497304499149323, + 0.2008623480796814, + 0.21327653527259827 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7614943981170654, + "mean_weights": [ + 0.15073809027671814, + 0.14079315960407257, + 0.11966855823993683, + 0.16534580290317535, + 0.18966889381408691, + 0.2337855100631714 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7334396839141846, + "mean_weights": [ + 0.12032384425401688, + 0.125158429145813, + 0.1547280251979828, + 0.15251007676124573, + 0.1917954981327057, + 0.25548410415649414 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7752734422683716, + "mean_weights": [ + 0.16270102560520172, + 0.14989414811134338, + 0.12619557976722717, + 0.1607460379600525, + 0.18175114691257477, + 0.21871207654476166 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7268621921539307, + "mean_weights": [ + 0.12063982337713242, + 0.13393959403038025, + 0.16096177697181702, + 0.14662346243858337, + 0.17392438650131226, + 0.26391100883483887 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0249087810516357, + "mean_weights": [ + 0.10775190591812134, + 0.12611693143844604, + 0.1291588544845581, + 0.13433250784873962, + 0.11416041851043701, + 0.10145977884531021, + 0.10813163220882416, + 0.1788879632949829 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1254401206970215, + "mean_weights": [ + 0.08790647983551025, + 0.11165842413902283, + 0.11382804811000824, + 0.10206783562898636, + 0.09298259764909744, + 0.08998951315879822, + 0.1232423186302185, + 0.20247796177864075, + 0.07584682106971741 + ], + "sources": 9 + }, + { + "entropy_mean": 2.164797782897949, + "mean_weights": [ + 0.09679380059242249, + 0.10930618643760681, + 0.10940437018871307, + 0.1065194308757782, + 0.10231906175613403, + 0.0894792377948761, + 0.09792773425579071, + 0.15008091926574707, + 0.13816925883293152 + ], + "sources": 9 + }, + { + "entropy_mean": 2.097749948501587, + "mean_weights": [ + 0.08595120906829834, + 0.11955319344997406, + 0.1062607616186142, + 0.09907947480678558, + 0.07950928807258606, + 0.07379093766212463, + 0.1002209484577179, + 0.20458786189556122, + 0.131046324968338 + ], + "sources": 9 + }, + { + "entropy_mean": 2.149176597595215, + "mean_weights": [ + 0.09992527961730957, + 0.11369793117046356, + 0.12063031643629074, + 0.11137115955352783, + 0.09962157905101776, + 0.0898594781756401, + 0.09756124764680862, + 0.14056888222694397, + 0.12676411867141724 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1050100326538086, + "mean_weights": [ + 0.076517753303051, + 0.1342213898897171, + 0.1254764199256897, + 0.12790073454380035, + 0.0898401141166687, + 0.09525885432958603, + 0.1059754490852356, + 0.16228562593460083, + 0.08252367377281189 + ], + "sources": 9 + }, + { + "entropy_mean": 2.140446662902832, + "mean_weights": [ + 0.11036726832389832, + 0.12699949741363525, + 0.11147990077733994, + 0.12464220076799393, + 0.11014757305383682, + 0.10495586693286896, + 0.08858617395162582, + 0.10959435999393463, + 0.11322715878486633 + ], + "sources": 9 + }, + { + "entropy_mean": 2.140577793121338, + "mean_weights": [ + 0.08426377177238464, + 0.12632998824119568, + 0.1283624768257141, + 0.11601665616035461, + 0.09646609425544739, + 0.08777236193418503, + 0.10991427302360535, + 0.15994463860988617, + 0.09092973917722702 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05581815540790558, + 0.04701259359717369, + 0.05504828691482544, + 0.06247949227690697, + 0.10042016953229904, + 0.09893567860126495, + 0.10539961606264114, + 0.08913368731737137, + 0.10551442950963974, + 0.0704219788312912, + 0.07794100791215897, + 0.07123440504074097, + 0.07851417362689972, + 0.07974004000425339, + 0.07635531574487686, + 0.07136041671037674, + 0.07898814976215363, + 0.051502104848623276, + 0.05948945879936218, + 0.05734645202755928, + 0.05630601570010185, + 0.054764918982982635, + 0.05502825230360031, + 0.054255250841379166, + 0.057338930666446686, + 0.04283996671438217, + 0.04555618762969971, + 0.04723348096013069, + 0.04675706475973129, + 0.04486935958266258, + 0.04799116402864456, + 0.04580603912472725, + 0.0505371131002903, + 0.041227150708436966, + 0.04136831313371658, + 0.04064388945698738, + 0.04442180320620537, + 0.043157923966646194, + 0.04518991708755493, + 0.045875295996665955, + 0.04627705737948418, + 0.04033881053328514, + 0.04083601385354996, + 0.041378311812877655, + 0.04169000685214996, + 0.041933998465538025, + 0.04239153489470482, + 0.04077022150158882, + 0.04231491684913635, + 0.03644668683409691, + 0.0375138483941555, + 0.037383291870355606, + 0.038913119584321976, + 0.03884527459740639, + 0.04077959060668945, + 0.0407528392970562, + 0.04941263794898987, + 0.048238202929496765, + 0.04445720463991165, + 0.05065429210662842, + 0.049143001437187195, + 0.04975641518831253, + 0.05452987924218178, + 0.0528714694082737 + ], + "loss_nats": 1.4320502281188965, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.933801293373108, + "mean_weights": [ + 0.11528845131397247, + 0.09230738878250122, + 0.0814325213432312, + 0.06491375714540482, + 0.06580930203199387, + 0.05624303221702576, + 0.05220801383256912, + 0.15835918486118317, + 0.313438355922699 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.04519475996494293, + 0.07933542132377625, + 0.1634819060564041, + 0.17356063425540924, + 0.19396348297595978, + 0.19592563807964325, + 0.19977088272571564, + 0.20413769781589508, + 0.06896122545003891, + 0.0703900009393692, + 0.07872548699378967, + 0.08005841821432114, + 0.0875031054019928, + 0.0889945924282074, + 0.10048835724592209, + 0.10390866547822952, + 0.039527297019958496, + 0.043775007128715515, + 0.061073824763298035, + 0.06786487251520157, + 0.08160728216171265, + 0.09043840318918228, + 0.10582270473241806, + 0.11542254686355591, + 0.02363201417028904, + 0.03245418146252632, + 0.04137130454182625, + 0.05422379821538925, + 0.06148746609687805, + 0.0745156928896904, + 0.07912598550319672, + 0.09330973774194717, + 0.027532115578651428, + 0.03882657364010811, + 0.051289595663547516, + 0.06532005220651627, + 0.0751892700791359, + 0.0864439532160759, + 0.09719052165746689, + 0.11046812683343887, + 0.043986596167087555, + 0.0477026104927063, + 0.08199729770421982, + 0.08639359474182129, + 0.1128886342048645, + 0.11987065523862839, + 0.15118840336799622, + 0.16759569942951202, + 0.04700809717178345, + 0.06834843754768372, + 0.07857025414705276, + 0.10255371779203415, + 0.10558044165372849, + 0.1335299015045166, + 0.13421936333179474, + 0.16770142316818237, + 0.013558072037994862, + 0.060653913766145706, + 0.06223326548933983, + 0.1185198500752449, + 0.12151430547237396, + 0.19719740748405457, + 0.2012815624475479, + 0.280917763710022 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.722850568825379e-05, + 8.501399133820087e-05, + 7.900782657088712e-05, + 7.458877371391281e-05, + 8.851825259625912e-05, + 8.496154623571783e-05, + 8.129017805913463e-05, + 7.308541535167024e-05, + 7.228730828501284e-05, + 6.910465162945911e-05, + 6.638290506089106e-05, + 6.404728628695011e-05, + 7.102679228410125e-05, + 6.824541924288496e-05, + 6.386093446053565e-05, + 6.002810187055729e-05, + 6.220424256753176e-05, + 5.8480411098571494e-05, + 5.7781468058237806e-05, + 5.275792864267714e-05, + 5.794492972199805e-05, + 5.5180109484354034e-05, + 4.93703773827292e-05, + 4.439146141521633e-05, + 6.059826409909874e-05, + 5.768534538219683e-05, + 5.5251708545256406e-05, + 5.3433937864610925e-05, + 5.2880252042086795e-05, + 4.163408812019043e-05, + 3.907483187504113e-05, + 3.544480205164291e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 8.046181119425455e-05, + "first_to_last_ratio": 1.6254778770020237, + "imbalance_abs_log_ratio": 0.4858018507042873, + "last_quartile_mean": 4.950040374751552e-05, + "mean": 6.28997514695584e-05, + "normalized": [ + 1.2278030339377428, + 1.3515791295192177, + 1.2560912360540017, + 1.18583574610802, + 1.4072909753721254, + 1.3507453408116674, + 1.2923767766948435, + 1.1619348827958629, + 1.1492463260367212, + 1.09864745114142, + 1.0553762695392908, + 1.018243868864046, + 1.1292062468398798, + 1.0849871048522937, + 1.0152811890114135, + 0.954345612949029, + 0.988942581078985, + 0.9297399390659639, + 0.9186279231357903, + 0.838762116066713, + 0.9212266879948113, + 0.8772707076761593, + 0.7849057624118434, + 0.7057493929320289, + 0.9634102310948954, + 0.9170997346486308, + 0.8784090120291904, + 0.8495095229505216, + 0.8407068518812711, + 0.6619118064455324, + 0.6212239470286648, + 0.563512593031423 + ], + "population_cv": 0.21279267414040023 + }, + "activation_output_rms_by_block": [ + 0.11307977139949799, + 0.24579407274723053, + 0.2624119520187378, + 0.2694927752017975, + 0.08423247188329697, + 0.09299755096435547, + 0.1023314967751503, + 0.11912773549556732, + 0.047540031373500824, + 0.07420121878385544, + 0.10375737398862839, + 0.13788114488124847, + 0.03878077492117882, + 0.06997464597225189, + 0.0936177521944046, + 0.1194845587015152, + 0.04525832086801529, + 0.07861223816871643, + 0.10294434428215027, + 0.13677391409873962, + 0.06066577881574631, + 0.10652344673871994, + 0.15397457778453827, + 0.2117626667022705, + 0.0858822911977768, + 0.13233982026576996, + 0.1727868914604187, + 0.21057960391044617, + 0.07290095835924149, + 0.13914754986763, + 0.22899514436721802, + 0.3399814963340759 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.16118347831070423, + "first_to_last_ratio": 0.9326305492848443, + "imbalance_abs_log_ratio": 0.06974613803006748, + "last_quartile_mean": 0.17282671947032213, + "mean": 0.13293232407886535, + "normalized": [ + 0.8506566945479013, + 1.8490165913401708, + 1.9740266623417755, + 2.0272930385382737, + 0.6336492833249797, + 0.6995856847367116, + 0.7698014571267079, + 0.8961532593448971, + 0.35762581977650926, + 0.5581879298208444, + 0.7805277964377708, + 1.037228122179276, + 0.2917332198162064, + 0.5263930083005073, + 0.7042512259009598, + 0.8988375064489812, + 0.3404613677044019, + 0.5913703737104438, + 0.7744116789914545, + 1.028898840417438, + 0.45636589321762594, + 0.8013359239512152, + 1.1582929799165258, + 1.593011091693824, + 0.6460602550424457, + 0.9955428161118746, + 1.2998109576261425, + 1.5841113541768417, + 0.5484065584830309, + 1.0467548117572767, + 1.7226445558219614, + 2.5575532413950244 + ], + "population_cv": 0.5475104567779164 + }, + "bits_per_byte": 1.94431735460847, + "branch_output_rms_by_sublayer": [ + 0.06706089526414871, + 0.08734722435474396, + 0.16490890085697174, + 0.01817459426820278, + 0.05402721092104912, + 0.006570694502443075, + 0.03852631151676178, + 0.02017863281071186, + 0.08292478322982788, + 0.007697225548326969, + 0.027649108320474625, + 0.00738769443705678, + 0.027229653671383858, + 0.010129167698323727, + 0.04429584741592407, + 0.02217743918299675, + 0.04215651750564575, + 0.020272625610232353, + 0.029113495722413063, + 0.02010158821940422, + 0.028859032317996025, + 0.02570836804807186, + 0.03370937705039978, + 0.025632938370108604, + 0.0246309582144022, + 0.027024641633033752, + 0.025226473808288574, + 0.029663097113370895, + 0.022274626418948174, + 0.032867297530174255, + 0.025765124708414078, + 0.03647798299789429, + 0.02585558034479618, + 0.036331918090581894, + 0.035017866641283035, + 0.03689613565802574, + 0.03337937593460083, + 0.034316401928663254, + 0.05114835500717163, + 0.046151019632816315, + 0.05665922909975052, + 0.030194124206900597, + 0.07309310138225555, + 0.034518927335739136, + 0.08364450931549072, + 0.046253010630607605, + 0.08640947192907333, + 0.07801470160484314, + 0.05644892156124115, + 0.061151664704084396, + 0.04434261471033096, + 0.06492369621992111, + 0.03316858038306236, + 0.06979183107614517, + 0.014417458325624466, + 0.07597717642784119, + 0.016864154487848282, + 0.07042016088962555, + 0.013014601543545723, + 0.08321022987365723, + 0.01720009371638298, + 0.1123834028840065, + 0.02818106859922409, + 0.1276838332414627 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00028301765084205076, + 0.00018640727245129113, + 0.00010306090203068865, + 0.00014222708269657445, + 0.00015838688665849668, + 9.970135661519603e-05, + 0.00012138718434767938, + 0.00021048722070999366, + 0.00017274196027943569, + 0.00013106145358904433, + 0.00013638371514977, + 0.0001547949615264904, + 0.00016587079599548126, + 0.00017931423985143718, + 0.00019278834870391096, + 0.00020784376871445687, + 0.00020123613260398355, + 0.00021726968219813378, + 0.0002071508396997652, + 0.0002559233234473849, + 0.0002295854626557122, + 0.0002641860573878995, + 0.0003281630733843104, + 0.0003582708629313403, + 0.000355412897731539, + 0.00032680906284868124, + 0.0003121616071976194, + 0.00029556208910195224, + 0.0002908363635073086, + 0.0002515380376246955, + 0.00023503421639413057, + 0.00023619860105820258 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00016308444454399634, + "first_to_last_ratio": 0.5663753457749909, + "imbalance_abs_log_ratio": 0.56849826548963, + "last_quartile_mean": 0.00028794410943301614, + "mean": 0.000219087909685458, + "normalized": [ + 1.291799493857858, + 0.8508332236083402, + 0.47040889740858816, + 0.6491781445209288, + 0.7229375956249293, + 0.45507466276133685, + 0.5540569743074986, + 0.960743205830882, + 0.7884595755532076, + 0.5982139944518515, + 0.622506807178792, + 0.7065426921491367, + 0.7570969855599063, + 0.8184579427905291, + 0.8799588666517243, + 0.9486774921211115, + 0.9185177442830869, + 0.9917009227486143, + 0.9455147022816972, + 1.1681307463054946, + 1.0479147981526018, + 1.2058449860021414, + 1.4978602600912592, + 1.6352835875138234, + 1.6222387544881012, + 1.4916800444071845, + 1.4248235224197732, + 1.3490570498677328, + 1.3274870526851938, + 1.1481146448739494, + 1.0727849689723477, + 1.0780996605303788 + ], + "population_cv": 0.3276215791506264 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.298600435256958, + "mean_weights": [ + 0.9062461256980896, + 0.09375390410423279 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6085544228553772, + "mean_weights": [ + 0.5902345180511475, + 0.40976545214653015 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5791133642196655, + "mean_weights": [ + 0.7206420302391052, + 0.2793579697608948 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6809916496276855, + "mean_weights": [ + 0.4366615414619446, + 0.5633384585380554 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6643537878990173, + "mean_weights": [ + 0.43601688742637634, + 0.5639830827713013 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6867029070854187, + "mean_weights": [ + 0.5107938051223755, + 0.4892061948776245 + ], + "sources": 2 + }, + { + "entropy_mean": 0.630885124206543, + "mean_weights": [ + 0.6297253370285034, + 0.3702746331691742 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6814190149307251, + "mean_weights": [ + 0.5384589433670044, + 0.461541086435318 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0193029642105103, + "mean_weights": [ + 0.2793715000152588, + 0.22541894018650055, + 0.49520957469940186 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0885753631591797, + "mean_weights": [ + 0.3747631013393402, + 0.27809539437294006, + 0.3471415042877197 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0166544914245605, + "mean_weights": [ + 0.29693084955215454, + 0.2092972993850708, + 0.49377185106277466 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0730276107788086, + "mean_weights": [ + 0.3801417946815491, + 0.2446514070034027, + 0.3752068281173706 + ], + "sources": 3 + }, + { + "entropy_mean": 1.01689612865448, + "mean_weights": [ + 0.25329312682151794, + 0.25301480293273926, + 0.4936920702457428 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0735528469085693, + "mean_weights": [ + 0.4058016240596771, + 0.23908020555973053, + 0.35511815547943115 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0387647151947021, + "mean_weights": [ + 0.3897400498390198, + 0.19914782047271729, + 0.41111212968826294 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0789580345153809, + "mean_weights": [ + 0.4085901975631714, + 0.2620368003845215, + 0.3293730318546295 + ], + "sources": 3 + }, + { + "entropy_mean": 1.330599308013916, + "mean_weights": [ + 0.25999969244003296, + 0.1543980836868286, + 0.24968576431274414, + 0.3359164595603943 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3653013706207275, + "mean_weights": [ + 0.3057574927806854, + 0.2060212939977646, + 0.20908181369304657, + 0.2791393995285034 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3513377904891968, + "mean_weights": [ + 0.2617591619491577, + 0.19308003783226013, + 0.2539125084877014, + 0.29124826192855835 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3677287101745605, + "mean_weights": [ + 0.29450443387031555, + 0.20663464069366455, + 0.20358902215957642, + 0.2952719032764435 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3450267314910889, + "mean_weights": [ + 0.3230239152908325, + 0.18811741471290588, + 0.2109987735748291, + 0.2778598666191101 + ], + "sources": 4 + }, + { + "entropy_mean": 1.348590612411499, + "mean_weights": [ + 0.3295133709907532, + 0.18519240617752075, + 0.19538277387619019, + 0.2899114787578583 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3515987396240234, + "mean_weights": [ + 0.33015355467796326, + 0.19018611311912537, + 0.21045629680156708, + 0.2692040503025055 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3623924255371094, + "mean_weights": [ + 0.2646353542804718, + 0.19441348314285278, + 0.2166558802127838, + 0.3242952823638916 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5844863653182983, + "mean_weights": [ + 0.23668400943279266, + 0.13988922536373138, + 0.18485453724861145, + 0.23012810945510864, + 0.20844411849975586 + ], + "sources": 5 + }, + { + "entropy_mean": 1.593141794204712, + "mean_weights": [ + 0.20686954259872437, + 0.17310559749603271, + 0.1597880721092224, + 0.2032153457403183, + 0.257021427154541 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6000027656555176, + "mean_weights": [ + 0.18989244103431702, + 0.17239899933338165, + 0.1975756287574768, + 0.20976364612579346, + 0.23036929965019226 + ], + "sources": 5 + }, + { + "entropy_mean": 1.582854986190796, + "mean_weights": [ + 0.19638973474502563, + 0.1686723232269287, + 0.15907755494117737, + 0.19003787636756897, + 0.2858225107192993 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5503523349761963, + "mean_weights": [ + 0.17854918539524078, + 0.139242485165596, + 0.17552749812602997, + 0.2097170352935791, + 0.29696381092071533 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5834622383117676, + "mean_weights": [ + 0.19760790467262268, + 0.17277662456035614, + 0.15623365342617035, + 0.19005371630191803, + 0.283328115940094 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5523425340652466, + "mean_weights": [ + 0.191162571310997, + 0.13771341741085052, + 0.18561774492263794, + 0.18527957797050476, + 0.30022668838500977 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5726368427276611, + "mean_weights": [ + 0.20087841153144836, + 0.16328519582748413, + 0.1421341896057129, + 0.19957204163074493, + 0.2941301763057709 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7497403621673584, + "mean_weights": [ + 0.14710332453250885, + 0.11305859684944153, + 0.15883512794971466, + 0.16542699933052063, + 0.2511061429977417, + 0.16446979343891144 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7714827060699463, + "mean_weights": [ + 0.17014864087104797, + 0.14715781807899475, + 0.12150244414806366, + 0.1485304832458496, + 0.20396047830581665, + 0.20870016515254974 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7444989681243896, + "mean_weights": [ + 0.13433504104614258, + 0.11546684801578522, + 0.15696288645267487, + 0.141643226146698, + 0.2109500616788864, + 0.24064195156097412 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7677668333053589, + "mean_weights": [ + 0.15450707077980042, + 0.1484912931919098, + 0.12180650234222412, + 0.162429541349411, + 0.18306714296340942, + 0.22969844937324524 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6891989707946777, + "mean_weights": [ + 0.11655238270759583, + 0.10491175204515457, + 0.1503290832042694, + 0.1321149468421936, + 0.194725900888443, + 0.3013659715652466 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7821013927459717, + "mean_weights": [ + 0.1637197732925415, + 0.1602180004119873, + 0.1371064931154251, + 0.15685361623764038, + 0.17574524879455566, + 0.20635688304901123 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7014548778533936, + "mean_weights": [ + 0.12046012282371521, + 0.11536401510238647, + 0.15961314737796783, + 0.13822567462921143, + 0.1780020147562027, + 0.28833502531051636 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.021324634552002, + "mean_weights": [ + 0.10935264825820923, + 0.12541475892066956, + 0.13264349102973938, + 0.15386636555194855, + 0.11865094304084778, + 0.10020800679922104, + 0.10042205452919006, + 0.159441739320755 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0959906578063965, + "mean_weights": [ + 0.08627711981534958, + 0.10818275809288025, + 0.10734419524669647, + 0.09354669600725174, + 0.09311383217573166, + 0.08255733549594879, + 0.10319340229034424, + 0.20966464281082153, + 0.11612001806497574 + ], + "sources": 9 + }, + { + "entropy_mean": 2.150697708129883, + "mean_weights": [ + 0.08980904519557953, + 0.10636033117771149, + 0.12191382050514221, + 0.11199057847261429, + 0.09992339462041855, + 0.08958576619625092, + 0.09268327802419662, + 0.13146400451660156, + 0.15626978874206543 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0447325706481934, + "mean_weights": [ + 0.08423449099063873, + 0.10816886276006699, + 0.1022123247385025, + 0.09324942529201508, + 0.07462728023529053, + 0.06389737129211426, + 0.08259779959917068, + 0.20694155991077423, + 0.1840709000825882 + ], + "sources": 9 + }, + { + "entropy_mean": 2.142486333847046, + "mean_weights": [ + 0.0978899598121643, + 0.10836011171340942, + 0.12591950595378876, + 0.1209651455283165, + 0.10007082670927048, + 0.09053842723369598, + 0.08933036029338837, + 0.11989083886146545, + 0.14703482389450073 + ], + "sources": 9 + }, + { + "entropy_mean": 2.067331314086914, + "mean_weights": [ + 0.07690603286027908, + 0.1283465027809143, + 0.11419844627380371, + 0.12572115659713745, + 0.09080883860588074, + 0.09361410140991211, + 0.09618563950061798, + 0.16779401898384094, + 0.1064252257347107 + ], + "sources": 9 + }, + { + "entropy_mean": 2.126595973968506, + "mean_weights": [ + 0.10529343783855438, + 0.12262256443500519, + 0.11240119487047195, + 0.1594175547361374, + 0.11083489656448364, + 0.1012311726808548, + 0.08822474628686905, + 0.0945647805929184, + 0.1054096445441246 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1250782012939453, + "mean_weights": [ + 0.08339430391788483, + 0.1166929081082344, + 0.1221323236823082, + 0.12451616674661636, + 0.09800273925065994, + 0.07886499911546707, + 0.09880783408880234, + 0.15631729364395142, + 0.12127144634723663 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05882728472352028, + 0.05434255301952362, + 0.06656001508235931, + 0.08629672974348068, + 0.1458369940519333, + 0.15356051921844482, + 0.13871629536151886, + 0.1133713349699974, + 0.1344493180513382, + 0.09049845486879349, + 0.09826874732971191, + 0.08978800475597382, + 0.09347957372665405, + 0.10162461549043655, + 0.09102483093738556, + 0.08678106218576431, + 0.09482892602682114, + 0.060551125556230545, + 0.07042072713375092, + 0.06859821081161499, + 0.06875573098659515, + 0.06323139369487762, + 0.06267102062702179, + 0.06210179254412651, + 0.06717807799577713, + 0.048520777374506, + 0.053565312176942825, + 0.05379631742835045, + 0.05197212100028992, + 0.049394212663173676, + 0.05453816428780556, + 0.051639657467603683, + 0.05769462510943413, + 0.04657750949263573, + 0.048166461288928986, + 0.04587066173553467, + 0.05144380033016205, + 0.049700506031513214, + 0.05281500518321991, + 0.05409446731209755, + 0.05452948063611984, + 0.04756034538149834, + 0.048005420714616776, + 0.048697564750909805, + 0.04912369325757027, + 0.049967169761657715, + 0.05079125985503197, + 0.05016079545021057, + 0.052426256239414215, + 0.04546017944812775, + 0.04680249094963074, + 0.04691625386476517, + 0.04869557544589043, + 0.04886418953537941, + 0.051024626940488815, + 0.051005132496356964, + 0.05881999060511589, + 0.05965002626180649, + 0.05215844139456749, + 0.06303735822439194, + 0.057204701006412506, + 0.06299907714128494, + 0.06047141179442406, + 0.06585761159658432 + ], + "loss_nats": 1.3476980924606323, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.8709112405776978, + "mean_weights": [ + 0.10687088966369629, + 0.08648081868886948, + 0.0809902623295784, + 0.06436935067176819, + 0.058983318507671356, + 0.0546012744307518, + 0.04820964112877846, + 0.13850103318691254, + 0.3609933853149414 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.06706089526414871, + 0.11307977139949799, + 0.239619642496109, + 0.24579407274723053, + 0.2610020935535431, + 0.2624119520187378, + 0.2643596827983856, + 0.2694927752017975, + 0.08292478322982788, + 0.08423247188329697, + 0.09236455708742142, + 0.09299755096435547, + 0.10117614269256592, + 0.1023314967751503, + 0.11657007038593292, + 0.11912773549556732, + 0.04215651750564575, + 0.047540031373500824, + 0.06610929220914841, + 0.07420121878385544, + 0.09169621020555496, + 0.10375737398862839, + 0.12501466274261475, + 0.13788114488124847, + 0.0246309582144022, + 0.03878077492117882, + 0.05176131799817085, + 0.06997464597225189, + 0.0750405415892601, + 0.0936177521944046, + 0.09971214830875397, + 0.1194845587015152, + 0.02585558034479618, + 0.04525832086801529, + 0.059157054871320724, + 0.07861223816871643, + 0.08684703707695007, + 0.10294434428215027, + 0.11631578952074051, + 0.13677391409873962, + 0.05665922909975052, + 0.06066577881574631, + 0.09965206682682037, + 0.10652344673871994, + 0.14260981976985931, + 0.15397457778453827, + 0.18696846067905426, + 0.2117626667022705, + 0.05644892156124115, + 0.0858822911977768, + 0.10220976173877716, + 0.13233982026576996, + 0.1401691436767578, + 0.1727868914604187, + 0.1740308403968811, + 0.21057960391044617, + 0.016864154487848282, + 0.07290095835924149, + 0.07443612068891525, + 0.13914754986763, + 0.1412491649389267, + 0.22899514436721802, + 0.23334667086601257, + 0.3399814963340759 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00012893932580482215, + 0.00013874190335627645, + 0.00013047004176769406, + 0.0001248678017873317, + 0.0001568332372698933, + 0.00015078592696227133, + 0.00014506626757793128, + 0.00013033850700594485, + 0.00012732735194731504, + 0.00012182179489172995, + 0.00011695339344441891, + 0.00011271165567450225, + 0.00013322403538040817, + 0.00012706179404631257, + 0.00011747308599296957, + 0.00010898760228883475, + 0.00011914948117919266, + 0.00010761221346911043, + 0.00010350234515499324, + 9.157581371255219e-05, + 0.00010028973338194191, + 9.595853771315888e-05, + 8.392572635784745e-05, + 7.49087703297846e-05, + 9.893974493024871e-05, + 9.361770207760856e-05, + 8.900452667148784e-05, + 8.53935198392719e-05, + 9.635069000069052e-05, + 6.989986286498606e-05, + 6.29150090389885e-05, + 5.117334512760863e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00013825537644152064, + "first_to_last_ratio": 1.7087171008907982, + "imbalance_abs_log_ratio": 0.5357428555514637, + "last_quartile_mean": 8.091180006886134e-05, + "mean": 0.00010924439834525401, + "normalized": [ + 1.1802831793473259, + 1.2700138904861653, + 1.1942950278819688, + 1.1430133139888945, + 1.4356181153951746, + 1.3802623223364756, + 1.3279057761796307, + 1.1930909866337074, + 1.1655275133184586, + 1.115130814366578, + 1.0705665024105082, + 1.0317385365452825, + 1.2195044999869864, + 1.163096652743226, + 1.0753236575271325, + 0.9976493434875472, + 1.0906690227048055, + 0.9850593266028592, + 0.947438465704084, + 0.8382655321432377, + 0.9180309004493582, + 0.8783840559942786, + 0.7682382587032985, + 0.6856989599873513, + 0.9056733931341846, + 0.8569565441858251, + 0.814728517156546, + 0.781674128218417, + 0.8819737346732008, + 0.6398484858150421, + 0.5759106186859398, + 0.4684299232065091 + ], + "population_cv": 0.23039819631834382 + }, + "activation_output_rms_by_block": [ + 0.09847520291805267, + 0.2336176484823227, + 0.2445971816778183, + 0.25124794244766235, + 0.06568645685911179, + 0.0732743963599205, + 0.08026880025863647, + 0.09760069102048874, + 0.03844119608402252, + 0.06061003357172012, + 0.08555752038955688, + 0.11737477779388428, + 0.032784610986709595, + 0.056893471628427505, + 0.08006435632705688, + 0.10290847718715668, + 0.042495425790548325, + 0.07159165292978287, + 0.09270426630973816, + 0.12490540742874146, + 0.05401705205440521, + 0.09582175314426422, + 0.13860832154750824, + 0.18824194371700287, + 0.07648079842329025, + 0.11732161790132523, + 0.1534760445356369, + 0.1852438896894455, + 0.0618281289935112, + 0.1136249303817749, + 0.17999152839183807, + 0.2661416530609131 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.1430960400030017, + "first_to_last_ratio": 0.991906938893357, + "imbalance_abs_log_ratio": 0.008125987697286065, + "last_quartile_mean": 0.1442635739222169, + "mean": 0.11505928682163358, + "normalized": [ + 0.855864881821406, + 2.030411059686819, + 2.1258360662099016, + 2.1836389689877898, + 0.5708922649671886, + 0.6368403488673751, + 0.6976299130297081, + 0.8482643488985867, + 0.33409902969079386, + 0.526772199324324, + 0.7435950869588587, + 1.0201243292585345, + 0.2849366782320903, + 0.494470921905022, + 0.6958530557483266, + 0.8943952290151663, + 0.36933503556671, + 0.6222153370441552, + 0.8057086817637722, + 1.085574323282322, + 0.46947146594210304, + 0.8328032946424252, + 1.2046687006011143, + 1.6360430254432048, + 0.6647077392531713, + 1.0196623075128106, + 1.3338866316245945, + 1.6099864235784203, + 0.5373588755973951, + 0.987533762119678, + 1.5643372505068913, + 2.3130827629193407 + ], + "population_cv": 0.5584622573890647 + }, + "bits_per_byte": 1.8658982533119042, + "branch_output_rms_by_sublayer": [ + 0.06009881943464279, + 0.0740160420536995, + 0.16605307161808014, + 0.01321887131780386, + 0.03974800556898117, + 0.0063072373159229755, + 0.0285708699375391, + 0.01633937656879425, + 0.06428490579128265, + 0.006369625683873892, + 0.019304754212498665, + 0.005901239812374115, + 0.02134516090154648, + 0.008837349712848663, + 0.04194158315658569, + 0.019798006862401962, + 0.032778121531009674, + 0.01838836632668972, + 0.02474922500550747, + 0.018625546246767044, + 0.02529645897448063, + 0.02233816310763359, + 0.031109536066651344, + 0.02339978888630867, + 0.019492624327540398, + 0.024492669850587845, + 0.021350549533963203, + 0.025974294170737267, + 0.021464798599481583, + 0.03068808652460575, + 0.025218263268470764, + 0.034959711134433746, + 0.023065272718667984, + 0.03464050218462944, + 0.03318355977535248, + 0.036559805274009705, + 0.028468552976846695, + 0.036484114825725555, + 0.04525674507021904, + 0.04875451326370239, + 0.04778723791241646, + 0.02997542731463909, + 0.06364032626152039, + 0.03515400364995003, + 0.07339563965797424, + 0.047455791383981705, + 0.07156641036272049, + 0.0753600150346756, + 0.045828815549612045, + 0.05875468626618385, + 0.03798433765769005, + 0.062369704246520996, + 0.031021421775221825, + 0.06693152338266373, + 0.013733532279729843, + 0.07132836431264877, + 0.014834110625088215, + 0.059611398726701736, + 0.01227854285389185, + 0.07063451409339905, + 0.015306126326322556, + 0.0886390432715416, + 0.023551054298877716, + 0.10633378475904465 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0004111790560190594, + 0.000318877405155091, + 0.00014742903942172142, + 0.00020755451403694258, + 0.0002463606891270598, + 0.00014456987080996807, + 0.00019212787762562836, + 0.00036555868126082633, + 0.0002991879962661554, + 0.00022983523357009925, + 0.00024435205456627054, + 0.0002669921054596226, + 0.0003006866868649958, + 0.0003291358425384917, + 0.00036871190632583635, + 0.00039276994518724825, + 0.00039201298161452774, + 0.0004400376459388244, + 0.00039498460757089043, + 0.0004741323980951389, + 0.00038284106289177163, + 0.0004510554379638166, + 0.0005610915320148874, + 0.0005919697796700657, + 0.000569093514379326, + 0.0005200314300946135, + 0.0005020604670293563, + 0.00047426498818203583, + 0.00048720678977716534, + 0.0004023984378573175, + 0.00036024062252050477, + 0.00034111967523395124 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00025420714168203714, + "first_to_last_ratio": 0.5561886763237934, + "imbalance_abs_log_ratio": 0.5866476963902508, + "last_quartile_mean": 0.0004570519906342838, + "mean": 0.00036905844609591283, + "normalized": [ + 1.1141299172765715, + 0.8640295555577653, + 0.3994734194036215, + 0.5623892807021743, + 0.6675384122303336, + 0.3917262220640156, + 0.5205892986816976, + 0.9905170444624455, + 0.8106791740742356, + 0.6227610721321047, + 0.6620958201909489, + 0.7234412551291004, + 0.8147400230121052, + 0.8918257962126523, + 0.9990610165578315, + 1.0642486287528918, + 1.062197561826425, + 1.192325092661516, + 1.0702494733537131, + 1.2847081623811933, + 1.0373453498806444, + 1.222178879078123, + 1.5203324512699758, + 1.6040000870654008, + 1.5420146060860698, + 1.409076084278166, + 1.3603820000338869, + 1.2850674282056165, + 1.3201345069624757, + 1.0903379725192364, + 0.9761072435309706, + 0.9242971644260901 + ], + "population_cv": 0.32188168615263085 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.30903440713882446, + "mean_weights": [ + 0.9020393490791321, + 0.09796065837144852 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5943206548690796, + "mean_weights": [ + 0.6391856670379639, + 0.36081427335739136 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6354048252105713, + "mean_weights": [ + 0.6418365240097046, + 0.3581635057926178 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6752665042877197, + "mean_weights": [ + 0.4143034815788269, + 0.5856965184211731 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6511313915252686, + "mean_weights": [ + 0.39545655250549316, + 0.6045434474945068 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6874249577522278, + "mean_weights": [ + 0.48418158292770386, + 0.5158184170722961 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6541168689727783, + "mean_weights": [ + 0.5819840431213379, + 0.4180159270763397 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6862896680831909, + "mean_weights": [ + 0.5171412229537964, + 0.4828587770462036 + ], + "sources": 2 + }, + { + "entropy_mean": 0.9821022748947144, + "mean_weights": [ + 0.22901444137096405, + 0.21936020255088806, + 0.5516253709793091 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0879732370376587, + "mean_weights": [ + 0.3412911891937256, + 0.2740474343299866, + 0.38466137647628784 + ], + "sources": 3 + }, + { + "entropy_mean": 0.966917097568512, + "mean_weights": [ + 0.23901140689849854, + 0.1963682770729065, + 0.564620316028595 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0720024108886719, + "mean_weights": [ + 0.3473166227340698, + 0.24676883220672607, + 0.4059145450592041 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0106569528579712, + "mean_weights": [ + 0.23335079848766327, + 0.2564984858036041, + 0.5101507306098938 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0801734924316406, + "mean_weights": [ + 0.3693524897098541, + 0.2494254857301712, + 0.3812220096588135 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0216292142868042, + "mean_weights": [ + 0.34189894795417786, + 0.1885349005460739, + 0.4695661664009094 + ], + "sources": 3 + }, + { + "entropy_mean": 1.085930585861206, + "mean_weights": [ + 0.37305235862731934, + 0.2697691321372986, + 0.3571785092353821 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3077852725982666, + "mean_weights": [ + 0.22748318314552307, + 0.13374337553977966, + 0.2605305314064026, + 0.3782429099082947 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3685073852539062, + "mean_weights": [ + 0.2904655933380127, + 0.2061430811882019, + 0.2159208357334137, + 0.2874704897403717 + ], + "sources": 4 + }, + { + "entropy_mean": 1.337707281112671, + "mean_weights": [ + 0.23166599869728088, + 0.1705857813358307, + 0.2691241204738617, + 0.3286241292953491 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3637475967407227, + "mean_weights": [ + 0.27879974246025085, + 0.19106647372245789, + 0.2128581553697586, + 0.31727564334869385 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3454006910324097, + "mean_weights": [ + 0.2911474108695984, + 0.18621329963207245, + 0.21682560443878174, + 0.30581367015838623 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3548822402954102, + "mean_weights": [ + 0.30711978673934937, + 0.18216806650161743, + 0.20504707098007202, + 0.30566510558128357 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3559226989746094, + "mean_weights": [ + 0.30007290840148926, + 0.1815715879201889, + 0.22795470058918, + 0.29040080308914185 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3654773235321045, + "mean_weights": [ + 0.2562333047389984, + 0.19376778602600098, + 0.2279227077960968, + 0.3220762014389038 + ], + "sources": 4 + }, + { + "entropy_mean": 1.581005334854126, + "mean_weights": [ + 0.21280249953269958, + 0.13066443800926208, + 0.1934673935174942, + 0.23154397308826447, + 0.23152171075344086 + ], + "sources": 5 + }, + { + "entropy_mean": 1.585540771484375, + "mean_weights": [ + 0.2017359882593155, + 0.1621151715517044, + 0.1603015810251236, + 0.1971220076084137, + 0.2787252962589264 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5922611951828003, + "mean_weights": [ + 0.18180608749389648, + 0.15913578867912292, + 0.20041614770889282, + 0.20758184790611267, + 0.2510601282119751 + ], + "sources": 5 + }, + { + "entropy_mean": 1.58207368850708, + "mean_weights": [ + 0.1940680742263794, + 0.16344153881072998, + 0.16575275361537933, + 0.18803098797798157, + 0.28870663046836853 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5402436256408691, + "mean_weights": [ + 0.17004631459712982, + 0.1271815001964569, + 0.18410228192806244, + 0.20228636264801025, + 0.3163835406303406 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5844833850860596, + "mean_weights": [ + 0.1959158182144165, + 0.16973957419395447, + 0.16372722387313843, + 0.1855185478925705, + 0.2850988507270813 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5373270511627197, + "mean_weights": [ + 0.18315774202346802, + 0.12609878182411194, + 0.18130838871002197, + 0.18354278802871704, + 0.32589229941368103 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5777552127838135, + "mean_weights": [ + 0.20232322812080383, + 0.16796565055847168, + 0.146327942609787, + 0.19361403584480286, + 0.28976911306381226 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7422258853912354, + "mean_weights": [ + 0.14358624815940857, + 0.10591697692871094, + 0.16278813779354095, + 0.15928007662296295, + 0.26065593957901, + 0.16777263581752777 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7709391117095947, + "mean_weights": [ + 0.16574758291244507, + 0.14620095491409302, + 0.12474840879440308, + 0.14619803428649902, + 0.19964303076267242, + 0.2174619883298874 + ], + "sources": 6 + }, + { + "entropy_mean": 1.729372501373291, + "mean_weights": [ + 0.13057398796081543, + 0.10316480696201324, + 0.16024847328662872, + 0.13796097040176392, + 0.20938652753829956, + 0.2586652338504791 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7713168859481812, + "mean_weights": [ + 0.15559720993041992, + 0.1570751816034317, + 0.12412820756435394, + 0.15546472370624542, + 0.17953522503376007, + 0.22819945216178894 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6778056621551514, + "mean_weights": [ + 0.11762547492980957, + 0.09853287786245346, + 0.15266761183738708, + 0.12690392136573792, + 0.18736225366592407, + 0.3169078230857849 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7828495502471924, + "mean_weights": [ + 0.1644119918346405, + 0.16709990799427032, + 0.13716958463191986, + 0.1527465432882309, + 0.17479197680950165, + 0.20378002524375916 + ], + "sources": 6 + }, + { + "entropy_mean": 1.691389560699463, + "mean_weights": [ + 0.12016548216342926, + 0.11117547750473022, + 0.15797217190265656, + 0.1312505006790161, + 0.17261159420013428, + 0.3068247437477112 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0278921127319336, + "mean_weights": [ + 0.11234381794929504, + 0.1314859390258789, + 0.1422475129365921, + 0.16262540221214294, + 0.12343616783618927, + 0.1039048433303833, + 0.09523597359657288, + 0.12872034311294556 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0601582527160645, + "mean_weights": [ + 0.08340039104223251, + 0.09872499108314514, + 0.10069122910499573, + 0.0878734141588211, + 0.08539019525051117, + 0.08006646484136581, + 0.1009698286652565, + 0.2190263718366623, + 0.14385712146759033 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1482632160186768, + "mean_weights": [ + 0.08904889225959778, + 0.10518711805343628, + 0.11570116132497787, + 0.12019158899784088, + 0.10622677206993103, + 0.08939772844314575, + 0.0865967720746994, + 0.12295456230640411, + 0.16469541192054749 + ], + "sources": 9 + }, + { + "entropy_mean": 1.9992003440856934, + "mean_weights": [ + 0.08175462484359741, + 0.09771331399679184, + 0.09001943469047546, + 0.08729717880487442, + 0.07171636074781418, + 0.06423543393611908, + 0.07570543885231018, + 0.20757217705249786, + 0.22398602962493896 + ], + "sources": 9 + }, + { + "entropy_mean": 2.137761354446411, + "mean_weights": [ + 0.09767191112041473, + 0.11270089447498322, + 0.12036637961864471, + 0.12793320417404175, + 0.1046464592218399, + 0.08895810693502426, + 0.08073794841766357, + 0.10764030367136002, + 0.15934479236602783 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0407042503356934, + "mean_weights": [ + 0.07716986536979675, + 0.12590588629245758, + 0.11102603375911713, + 0.11707732081413269, + 0.08211341500282288, + 0.09504722058773041, + 0.09234312176704407, + 0.17442387342453003, + 0.12489324063062668 + ], + "sources": 9 + }, + { + "entropy_mean": 2.128730297088623, + "mean_weights": [ + 0.10417187213897705, + 0.13345485925674438, + 0.11557041108608246, + 0.1635882556438446, + 0.10930082947015762, + 0.09852401167154312, + 0.08385953307151794, + 0.08640560507774353, + 0.10512463748455048 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1099185943603516, + "mean_weights": [ + 0.08170980215072632, + 0.11237261444330215, + 0.11802452802658081, + 0.11393916606903076, + 0.0935322567820549, + 0.07808645069599152, + 0.09232928603887558, + 0.15550142526626587, + 0.15450447797775269 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06159411370754242, + 0.05641147121787071, + 0.0601210743188858, + 0.0968460738658905, + 0.14369840919971466, + 0.15035921335220337, + 0.13629312813282013, + 0.11518421769142151, + 0.13050560653209686, + 0.07938458025455475, + 0.08858852833509445, + 0.07773812115192413, + 0.0843547061085701, + 0.0894903838634491, + 0.08370587974786758, + 0.07600551843643188, + 0.0868152379989624, + 0.04898906499147415, + 0.06335873156785965, + 0.055966150015592575, + 0.05818251520395279, + 0.05542164295911789, + 0.055110398679971695, + 0.053531844168901443, + 0.05831512063741684, + 0.04076668992638588, + 0.04529546573758125, + 0.04488367959856987, + 0.04510748013854027, + 0.04101870581507683, + 0.047199174761772156, + 0.043394725769758224, + 0.05005872622132301, + 0.039048757404088974, + 0.04125509411096573, + 0.037758756428956985, + 0.04443074017763138, + 0.04148557782173157, + 0.0460357666015625, + 0.04605577141046524, + 0.04641414061188698, + 0.04017103090882301, + 0.04042011499404907, + 0.04096098616719246, + 0.04127231240272522, + 0.041848864406347275, + 0.04266415163874626, + 0.042332492768764496, + 0.04439258202910423, + 0.03856244310736656, + 0.039745256304740906, + 0.03987186402082443, + 0.04138168320059776, + 0.04157072678208351, + 0.04337858036160469, + 0.04334103316068649, + 0.0472542941570282, + 0.05266488343477249, + 0.0427602119743824, + 0.05429914966225624, + 0.046690016984939575, + 0.055435240268707275, + 0.04864690452814102, + 0.0566694401204586 + ], + "loss_nats": 1.293342113494873, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.7431397438049316, + "mean_weights": [ + 0.0995081216096878, + 0.0776149332523346, + 0.06879925727844238, + 0.060406532138586044, + 0.05441401153802872, + 0.04954121261835098, + 0.03926881030201912, + 0.12004305422306061, + 0.43040403723716736 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.06009881943464279, + 0.09847520291805267, + 0.22995010018348694, + 0.2336176484823227, + 0.2430322766304016, + 0.2445971816778183, + 0.2476799637079239, + 0.25124794244766235, + 0.06428490579128265, + 0.06568645685911179, + 0.07276169210672379, + 0.0732743963599205, + 0.07930935174226761, + 0.08026880025863647, + 0.09504730254411697, + 0.09760069102048874, + 0.032778121531009674, + 0.03844119608402252, + 0.05227602273225784, + 0.06061003357172012, + 0.07424736768007278, + 0.08555752038955688, + 0.10497704148292542, + 0.11737477779388428, + 0.019492624327540398, + 0.032784610986709595, + 0.040958914905786514, + 0.056893471628427505, + 0.06350068747997284, + 0.08006435632705688, + 0.08542235195636749, + 0.10290847718715668, + 0.023065272718667984, + 0.042495425790548325, + 0.05381188914179802, + 0.07159165292978287, + 0.07674155384302139, + 0.09270426630973816, + 0.10431230813264847, + 0.12490540742874146, + 0.04778723791241646, + 0.05401705205440521, + 0.08787702769041061, + 0.09582175314426422, + 0.12634232640266418, + 0.13860832154750824, + 0.16462822258472443, + 0.18824194371700287, + 0.045828815549612045, + 0.07648079842329025, + 0.08903630077838898, + 0.11732161790132523, + 0.12432269752025604, + 0.1534760445356369, + 0.15425331890583038, + 0.1852438896894455, + 0.014834110625088215, + 0.0618281289935112, + 0.06328272074460983, + 0.1136249303817749, + 0.1152983158826828, + 0.17999152839183807, + 0.18247950077056885, + 0.2661416530609131 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 7.9557801867581395, + "cross_entropy_nats": 5.514526605606079, + "step": 0 + }, + { + "bits_per_byte": 5.165781185190186, + "cross_entropy_nats": 3.58064666390419, + "step": 100 + }, + { + "bits_per_byte": 3.1104177286073162, + "cross_entropy_nats": 2.15597727894783, + "step": 500 + }, + { + "bits_per_byte": 1.9543160174708378, + "cross_entropy_nats": 1.354628637433052, + "step": 2000 + }, + { + "bits_per_byte": 1.8041839773784627, + "cross_entropy_nats": 1.2505650371313095, + "step": 4000 + }, + { + "bits_per_byte": 1.7180484874899726, + "cross_entropy_nats": 1.190860465168953, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.40.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.40.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.48.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.48.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "40": true, + "41": true, + "42": true, + "43": true, + "44": true, + "45": true, + "46": true, + "47": true, + "48": true, + "49": true, + "50": true, + "51": true, + "52": true, + "53": true, + "54": true, + "55": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_groups_6_7_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "22b43094ce3c67188311a7ad094214b5f64646f275a5093592a80f382624f2f1", + "final_model_state": "1c164cc59ddd419b6b4aeb37945f715ddbf0a1a6582dbc6c28a517527da49371", + "final_optimizer_state": "c7f66bb2b9115ed25ee91eadc1084d5bd947b04bbd5ac6888fc156e75f84147b", + "final_public_parameters": "5141274aafe289f5c2ce5cb835abb664bbe5d21b114a2d39a8e5b603decb17b0", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "73bbe569e1a47981386ebaf59ca891174746bf81897fe5403676ad6d50423c58" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "65136111a29a042e61a7909132560d95cd4bcf0f9b52f64d0fb2e57773856434", + "1": "d995676b4e7dec8f661cd8c2345fe7fc7a513c17f528c02fc946a441a6995a94", + "7999": "2345e7ac3decca2bdaebf13094fdc92fcabef42e3e461a2fefd6e8e7e76baccc" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "685354d7b4dc15d92c29dabb7ed14cdd365cfa90d020551e4b3b96d1f5a0ff26", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073001, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 197.05204110038463, + "measured_steps": 7980, + "median_ms": 197.57537748955656, + "p95_ms": 199.21832635154715, + "peak_allocated_bytes": 10566180864, + "peak_reserved_bytes": 11068768256, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 7.958777844362678, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.516604423522949, + "step": 1, + "unclipped_grad_norm": 21.321983337402344 + }, + { + "bits_per_byte": 7.24805334829669, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.023967742919922, + "step": 10, + "unclipped_grad_norm": 11.155019760131836 + }, + { + "bits_per_byte": 6.733684884373789, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.6674346923828125, + "step": 20, + "unclipped_grad_norm": 3.4893624782562256 + }, + { + "bits_per_byte": 6.611928046868954, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.583039283752441, + "step": 30, + "unclipped_grad_norm": 3.0757408142089844 + }, + { + "bits_per_byte": 6.392097070813699, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.4306640625, + "step": 40, + "unclipped_grad_norm": 3.2317423820495605 + }, + { + "bits_per_byte": 6.094256140188279, + "learning_rate": 3.75e-05, + "loss_nats": 4.224216461181641, + "step": 50, + "unclipped_grad_norm": 2.873729705810547 + }, + { + "bits_per_byte": 5.886236932594281, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.080028533935547, + "step": 60, + "unclipped_grad_norm": 2.6709868907928467 + }, + { + "bits_per_byte": 5.715815884201307, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.9619016647338867, + "step": 70, + "unclipped_grad_norm": 3.1735682487487793 + }, + { + "bits_per_byte": 5.566368464046354, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.8583126068115234, + "step": 80, + "unclipped_grad_norm": 2.5606157779693604 + }, + { + "bits_per_byte": 5.355355038500834, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.7120492458343506, + "step": 90, + "unclipped_grad_norm": 2.426494598388672 + }, + { + "bits_per_byte": 5.181467851762371, + "learning_rate": 7.5e-05, + "loss_nats": 3.591519832611084, + "step": 100, + "unclipped_grad_norm": 2.2765960693359375 + }, + { + "bits_per_byte": 4.992471301215007, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.460517406463623, + "step": 110, + "unclipped_grad_norm": 2.09186053276062 + }, + { + "bits_per_byte": 4.7761191902214035, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.310553550720215, + "step": 120, + "unclipped_grad_norm": 1.8941689729690552 + }, + { + "bits_per_byte": 4.582826513055486, + "learning_rate": 9.75e-05, + "loss_nats": 3.1765732765197754, + "step": 130, + "unclipped_grad_norm": 1.6992738246917725 + }, + { + "bits_per_byte": 4.428815017311936, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.0698206424713135, + "step": 140, + "unclipped_grad_norm": 1.4459766149520874 + }, + { + "bits_per_byte": 4.272379254121823, + "learning_rate": 0.0001125, + "loss_nats": 2.9613876342773438, + "step": 150, + "unclipped_grad_norm": 1.2247486114501953 + }, + { + "bits_per_byte": 4.168162583213185, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.8891501426696777, + "step": 160, + "unclipped_grad_norm": 1.0640714168548584 + }, + { + "bits_per_byte": 4.028077554605902, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.79205060005188, + "step": 170, + "unclipped_grad_norm": 0.9820604920387268 + }, + { + "bits_per_byte": 3.9491010575378898, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.7373082637786865, + "step": 180, + "unclipped_grad_norm": 0.8641575574874878 + }, + { + "bits_per_byte": 3.839261929807245, + "learning_rate": 0.0001425, + "loss_nats": 2.6611735820770264, + "step": 190, + "unclipped_grad_norm": 0.7899060249328613 + }, + { + "bits_per_byte": 3.783438081163686, + "learning_rate": 0.00015, + "loss_nats": 2.6224794387817383, + "step": 200, + "unclipped_grad_norm": 1.0962475538253784 + }, + { + "bits_per_byte": 3.7344969782076562, + "learning_rate": 0.0001575, + "loss_nats": 2.5885560512542725, + "step": 210, + "unclipped_grad_norm": 0.7674247622489929 + }, + { + "bits_per_byte": 3.635042162936108, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.5196192264556885, + "step": 220, + "unclipped_grad_norm": 0.7568824291229248 + }, + { + "bits_per_byte": 3.6286846522643135, + "learning_rate": 0.0001725, + "loss_nats": 2.5152125358581543, + "step": 230, + "unclipped_grad_norm": 1.0865727663040161 + }, + { + "bits_per_byte": 3.5048798453530052, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.4293975830078125, + "step": 240, + "unclipped_grad_norm": 0.734052836894989 + }, + { + "bits_per_byte": 3.540276626622985, + "learning_rate": 0.0001875, + "loss_nats": 2.453932762145996, + "step": 250, + "unclipped_grad_norm": 1.3704512119293213 + }, + { + "bits_per_byte": 3.541249360496265, + "learning_rate": 0.000195, + "loss_nats": 2.4546070098876953, + "step": 260, + "unclipped_grad_norm": 1.0168581008911133 + }, + { + "bits_per_byte": 3.519328795782593, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.439412832260132, + "step": 270, + "unclipped_grad_norm": 0.8324092030525208 + }, + { + "bits_per_byte": 3.468906578237333, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.4044628143310547, + "step": 280, + "unclipped_grad_norm": 0.9137193560600281 + }, + { + "bits_per_byte": 3.367204981614913, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.3339686393737793, + "step": 290, + "unclipped_grad_norm": 0.7944438457489014 + }, + { + "bits_per_byte": 3.413081697697869, + "learning_rate": 0.000225, + "loss_nats": 2.3657679557800293, + "step": 300, + "unclipped_grad_norm": 1.5879573822021484 + }, + { + "bits_per_byte": 3.3687473220278505, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.3350377082824707, + "step": 310, + "unclipped_grad_norm": 1.0807567834854126 + }, + { + "bits_per_byte": 3.345624254619346, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.319010019302368, + "step": 320, + "unclipped_grad_norm": 1.2019305229187012 + }, + { + "bits_per_byte": 3.331462171251397, + "learning_rate": 0.0002475, + "loss_nats": 2.3091936111450195, + "step": 330, + "unclipped_grad_norm": 1.149749517440796 + }, + { + "bits_per_byte": 3.414775382843566, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.3669419288635254, + "step": 340, + "unclipped_grad_norm": 1.1368536949157715 + }, + { + "bits_per_byte": 3.351615098279501, + "learning_rate": 0.0002625, + "loss_nats": 2.32316255569458, + "step": 350, + "unclipped_grad_norm": 0.9630405306816101 + }, + { + "bits_per_byte": 3.2885696981262127, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.2794628143310547, + "step": 360, + "unclipped_grad_norm": 1.1213526725769043 + }, + { + "bits_per_byte": 3.276858367533059, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.2713451385498047, + "step": 370, + "unclipped_grad_norm": 0.8436188101768494 + }, + { + "bits_per_byte": 3.338454985835479, + "learning_rate": 0.000285, + "loss_nats": 2.3140406608581543, + "step": 380, + "unclipped_grad_norm": 1.504451036453247 + }, + { + "bits_per_byte": 3.2545759513219714, + "learning_rate": 0.0002925, + "loss_nats": 2.2559001445770264, + "step": 390, + "unclipped_grad_norm": 1.3384807109832764 + }, + { + "bits_per_byte": 3.2093125253719923, + "learning_rate": 0.0003, + "loss_nats": 2.2245259284973145, + "step": 400, + "unclipped_grad_norm": 1.2285854816436768 + }, + { + "bits_per_byte": 3.1867102214303005, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.2088592052459717, + "step": 410, + "unclipped_grad_norm": 0.8287063837051392 + }, + { + "bits_per_byte": 3.2421075530997197, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.247257709503174, + "step": 420, + "unclipped_grad_norm": 1.8037424087524414 + }, + { + "bits_per_byte": 3.205863241326288, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.222135066986084, + "step": 430, + "unclipped_grad_norm": 1.0546784400939941 + }, + { + "bits_per_byte": 3.252657656834248, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.254570484161377, + "step": 440, + "unclipped_grad_norm": 1.1148794889450073 + }, + { + "bits_per_byte": 3.155373262616588, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.187138080596924, + "step": 450, + "unclipped_grad_norm": 0.8904770612716675 + }, + { + "bits_per_byte": 3.184654684787188, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.2074344158172607, + "step": 460, + "unclipped_grad_norm": 1.0755161046981812 + }, + { + "bits_per_byte": 3.137023057734828, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.1744186878204346, + "step": 470, + "unclipped_grad_norm": 1.1856991052627563 + }, + { + "bits_per_byte": 3.137933189923194, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.1750495433807373, + "step": 480, + "unclipped_grad_norm": 0.9833663702011108 + }, + { + "bits_per_byte": 3.080759621435664, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.1354198455810547, + "step": 490, + "unclipped_grad_norm": 1.625687837600708 + }, + { + "bits_per_byte": 3.0477578705615995, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.1125447750091553, + "step": 500, + "unclipped_grad_norm": 1.4587886333465576 + }, + { + "bits_per_byte": 3.0781640592688424, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.1336207389831543, + "step": 510, + "unclipped_grad_norm": 0.9339342713356018 + }, + { + "bits_per_byte": 3.005261301498701, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.0830883979797363, + "step": 520, + "unclipped_grad_norm": 1.0892436504364014 + }, + { + "bits_per_byte": 3.074576501171844, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.131134033203125, + "step": 530, + "unclipped_grad_norm": 1.2632100582122803 + }, + { + "bits_per_byte": 3.0146178456362414, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.089573860168457, + "step": 540, + "unclipped_grad_norm": 0.8534988164901733 + }, + { + "bits_per_byte": 2.991983208956187, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.0738847255706787, + "step": 550, + "unclipped_grad_norm": 0.9598786234855652 + }, + { + "bits_per_byte": 2.9897278284727724, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.0723214149475098, + "step": 560, + "unclipped_grad_norm": 1.29025137424469 + }, + { + "bits_per_byte": 2.9395350357837633, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.0375304222106934, + "step": 570, + "unclipped_grad_norm": 0.9295416474342346 + }, + { + "bits_per_byte": 3.001635907218513, + "learning_rate": 0.000299626473885547, + "loss_nats": 2.080575466156006, + "step": 580, + "unclipped_grad_norm": 0.898005485534668 + }, + { + "bits_per_byte": 2.900255574187905, + "learning_rate": 0.00029958384005397226, + "loss_nats": 2.0103039741516113, + "step": 590, + "unclipped_grad_norm": 1.2257674932479858 + }, + { + "bits_per_byte": 2.9531189134795115, + "learning_rate": 0.0002995389065559004, + "loss_nats": 2.0469460487365723, + "step": 600, + "unclipped_grad_norm": 0.852761447429657 + }, + { + "bits_per_byte": 2.8686083586720357, + "learning_rate": 0.000299491674159121, + "loss_nats": 1.9883677959442139, + "step": 610, + "unclipped_grad_norm": 0.9190118908882141 + }, + { + "bits_per_byte": 2.921888927874636, + "learning_rate": 0.00029944214367070535, + "loss_nats": 2.025299072265625, + "step": 620, + "unclipped_grad_norm": 0.8095329999923706 + }, + { + "bits_per_byte": 2.845826504842843, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.97257661819458, + "step": 630, + "unclipped_grad_norm": 0.8052086234092712 + }, + { + "bits_per_byte": 2.797842313356214, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.9393165111541748, + "step": 640, + "unclipped_grad_norm": 1.0781192779541016 + }, + { + "bits_per_byte": 2.7552171012704707, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.9097709655761719, + "step": 650, + "unclipped_grad_norm": 1.3079663515090942 + }, + { + "bits_per_byte": 2.687411565306601, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.86277174949646, + "step": 660, + "unclipped_grad_norm": 0.8526151776313782 + }, + { + "bits_per_byte": 2.7299691882105557, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.8922704458236694, + "step": 670, + "unclipped_grad_norm": 1.1361027956008911 + }, + { + "bits_per_byte": 2.655181672569609, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.8404316902160645, + "step": 680, + "unclipped_grad_norm": 0.9145796895027161 + }, + { + "bits_per_byte": 2.7342496644078804, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.8952374458312988, + "step": 690, + "unclipped_grad_norm": 1.2074719667434692 + }, + { + "bits_per_byte": 2.640863428954677, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.8305070400238037, + "step": 700, + "unclipped_grad_norm": 0.8382254242897034 + }, + { + "bits_per_byte": 2.6551386769068936, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.8404018878936768, + "step": 710, + "unclipped_grad_norm": 1.0521663427352905 + }, + { + "bits_per_byte": 2.673591039483266, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.8531920909881592, + "step": 720, + "unclipped_grad_norm": 1.272436261177063 + }, + { + "bits_per_byte": 2.6318969414692934, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.8242919445037842, + "step": 730, + "unclipped_grad_norm": 1.0338865518569946 + }, + { + "bits_per_byte": 2.5773001890396405, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.786448359489441, + "step": 740, + "unclipped_grad_norm": 1.0496065616607666 + }, + { + "bits_per_byte": 2.5505205984911417, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.7678861618041992, + "step": 750, + "unclipped_grad_norm": 1.0196244716644287 + }, + { + "bits_per_byte": 2.481558823165522, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.7200855016708374, + "step": 760, + "unclipped_grad_norm": 1.0939509868621826 + }, + { + "bits_per_byte": 2.61914511583842, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.815453052520752, + "step": 770, + "unclipped_grad_norm": 0.9276040196418762 + }, + { + "bits_per_byte": 2.5086259687062635, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.738847017288208, + "step": 780, + "unclipped_grad_norm": 0.9036074876785278 + }, + { + "bits_per_byte": 2.4903447288674996, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.7261754274368286, + "step": 790, + "unclipped_grad_norm": 1.0674705505371094 + }, + { + "bits_per_byte": 2.43120591462856, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.6851835250854492, + "step": 800, + "unclipped_grad_norm": 0.9424428939819336 + }, + { + "bits_per_byte": 2.4865733213167287, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7235612869262695, + "step": 810, + "unclipped_grad_norm": 0.8431268930435181 + }, + { + "bits_per_byte": 2.4259095369125885, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.6815123558044434, + "step": 820, + "unclipped_grad_norm": 1.1989459991455078 + }, + { + "bits_per_byte": 2.4961998782161237, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.730233907699585, + "step": 830, + "unclipped_grad_norm": 1.552717685699463 + }, + { + "bits_per_byte": 2.5448967658079282, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.7639880180358887, + "step": 840, + "unclipped_grad_norm": 1.1160975694656372 + }, + { + "bits_per_byte": 2.436699728427721, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.6889915466308594, + "step": 850, + "unclipped_grad_norm": 1.051925778388977 + }, + { + "bits_per_byte": 2.4492720041710943, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.6977059841156006, + "step": 860, + "unclipped_grad_norm": 0.9328287839889526 + }, + { + "bits_per_byte": 2.3406037343275234, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.6223828792572021, + "step": 870, + "unclipped_grad_norm": 0.865578293800354 + }, + { + "bits_per_byte": 2.3792623385537675, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.6491789817810059, + "step": 880, + "unclipped_grad_norm": 0.9337467551231384 + }, + { + "bits_per_byte": 2.312986416286672, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.6032400131225586, + "step": 890, + "unclipped_grad_norm": 1.0183653831481934 + }, + { + "bits_per_byte": 2.3254228257358633, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.6118602752685547, + "step": 900, + "unclipped_grad_norm": 1.4078937768936157 + }, + { + "bits_per_byte": 2.347192389682078, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.6269497871398926, + "step": 910, + "unclipped_grad_norm": 0.9840359687805176 + }, + { + "bits_per_byte": 2.320075197189935, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.6081535816192627, + "step": 920, + "unclipped_grad_norm": 0.9145527482032776 + }, + { + "bits_per_byte": 2.346458711693497, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.626441240310669, + "step": 930, + "unclipped_grad_norm": 0.8572312593460083 + }, + { + "bits_per_byte": 2.351096739821965, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.6296560764312744, + "step": 940, + "unclipped_grad_norm": 1.0435532331466675 + }, + { + "bits_per_byte": 2.320259906556962, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.6082816123962402, + "step": 950, + "unclipped_grad_norm": 0.8369219899177551 + }, + { + "bits_per_byte": 2.3634616044883976, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.6382267475128174, + "step": 960, + "unclipped_grad_norm": 0.9031174778938293 + }, + { + "bits_per_byte": 2.266536997976343, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5710437297821045, + "step": 970, + "unclipped_grad_norm": 1.1165263652801514 + }, + { + "bits_per_byte": 2.3488327602060073, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.628086805343628, + "step": 980, + "unclipped_grad_norm": 0.8563165068626404 + }, + { + "bits_per_byte": 2.3144214395254714, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.6042346954345703, + "step": 990, + "unclipped_grad_norm": 1.1368024349212646 + }, + { + "bits_per_byte": 2.279426409727907, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.5799779891967773, + "step": 1000, + "unclipped_grad_norm": 1.021753191947937 + }, + { + "bits_per_byte": 2.2576482466491496, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.564882516860962, + "step": 1010, + "unclipped_grad_norm": 1.0570216178894043 + }, + { + "bits_per_byte": 2.22758757108749, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.5440460443496704, + "step": 1020, + "unclipped_grad_norm": 0.9315199255943298 + }, + { + "bits_per_byte": 2.231221736482872, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.546565055847168, + "step": 1030, + "unclipped_grad_norm": 1.043064832687378 + }, + { + "bits_per_byte": 2.2689536981862672, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.572718858718872, + "step": 1040, + "unclipped_grad_norm": 0.9336802959442139 + }, + { + "bits_per_byte": 2.259553470455408, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5662031173706055, + "step": 1050, + "unclipped_grad_norm": 0.7879506349563599 + }, + { + "bits_per_byte": 2.1748846916091384, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.5075151920318604, + "step": 1060, + "unclipped_grad_norm": 0.8336158394813538 + }, + { + "bits_per_byte": 2.2471471579701174, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.5576037168502808, + "step": 1070, + "unclipped_grad_norm": 0.8977189660072327 + }, + { + "bits_per_byte": 2.2372222111714755, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.5507242679595947, + "step": 1080, + "unclipped_grad_norm": 0.8431094288825989 + }, + { + "bits_per_byte": 2.191021135808992, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.518700122833252, + "step": 1090, + "unclipped_grad_norm": 0.7572782039642334 + }, + { + "bits_per_byte": 2.2616172622657613, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5676336288452148, + "step": 1100, + "unclipped_grad_norm": 1.0218000411987305 + }, + { + "bits_per_byte": 2.123016099970122, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.4715626239776611, + "step": 1110, + "unclipped_grad_norm": 0.8405666947364807 + }, + { + "bits_per_byte": 2.1662938142332395, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.5015604496002197, + "step": 1120, + "unclipped_grad_norm": 0.8585324287414551 + }, + { + "bits_per_byte": 2.1182848572448862, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.4682831764221191, + "step": 1130, + "unclipped_grad_norm": 1.0757169723510742 + }, + { + "bits_per_byte": 2.244305832595213, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.5556342601776123, + "step": 1140, + "unclipped_grad_norm": 0.8914085030555725 + }, + { + "bits_per_byte": 2.0951435596753902, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.4522428512573242, + "step": 1150, + "unclipped_grad_norm": 0.8124125599861145 + }, + { + "bits_per_byte": 2.121791583495979, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.4707138538360596, + "step": 1160, + "unclipped_grad_norm": 0.8673461675643921 + }, + { + "bits_per_byte": 2.1775493908016066, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.5093622207641602, + "step": 1170, + "unclipped_grad_norm": 1.0352803468704224 + }, + { + "bits_per_byte": 2.165567015550693, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.5010566711425781, + "step": 1180, + "unclipped_grad_norm": 0.9384984970092773 + }, + { + "bits_per_byte": 2.0844211293246975, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.4448106288909912, + "step": 1190, + "unclipped_grad_norm": 0.930260419845581 + }, + { + "bits_per_byte": 2.0516309171111944, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.4220821857452393, + "step": 1200, + "unclipped_grad_norm": 1.0661922693252563 + }, + { + "bits_per_byte": 2.101316188997507, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.4565213918685913, + "step": 1210, + "unclipped_grad_norm": 0.7982707619667053 + }, + { + "bits_per_byte": 2.11886168705588, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4686830043792725, + "step": 1220, + "unclipped_grad_norm": 0.7903804183006287 + }, + { + "bits_per_byte": 2.0872621107343003, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.4467798471450806, + "step": 1230, + "unclipped_grad_norm": 0.7807658910751343 + }, + { + "bits_per_byte": 2.020406778916448, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.4004392623901367, + "step": 1240, + "unclipped_grad_norm": 0.9420327544212341 + }, + { + "bits_per_byte": 2.1222521530349896, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.4710330963134766, + "step": 1250, + "unclipped_grad_norm": 0.8917564153671265 + }, + { + "bits_per_byte": 2.094018105208144, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.451462745666504, + "step": 1260, + "unclipped_grad_norm": 0.7565622329711914 + }, + { + "bits_per_byte": 2.0811376365544247, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.4425346851348877, + "step": 1270, + "unclipped_grad_norm": 0.8209865093231201 + }, + { + "bits_per_byte": 1.9807775046087346, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.3729703426361084, + "step": 1280, + "unclipped_grad_norm": 0.8037207722663879 + }, + { + "bits_per_byte": 2.092389601487124, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4503339529037476, + "step": 1290, + "unclipped_grad_norm": 0.7903739213943481 + }, + { + "bits_per_byte": 2.0465137453174225, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.4185352325439453, + "step": 1300, + "unclipped_grad_norm": 0.8105548620223999 + }, + { + "bits_per_byte": 2.049106899727132, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.420332670211792, + "step": 1310, + "unclipped_grad_norm": 0.77737957239151 + }, + { + "bits_per_byte": 2.021933812873459, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.401497721672058, + "step": 1320, + "unclipped_grad_norm": 0.8686265349388123 + }, + { + "bits_per_byte": 2.0978068830066516, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.4540889263153076, + "step": 1330, + "unclipped_grad_norm": 0.7300577163696289 + }, + { + "bits_per_byte": 2.0568973698459154, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.4257326126098633, + "step": 1340, + "unclipped_grad_norm": 0.919792652130127 + }, + { + "bits_per_byte": 2.0468033641014753, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.4187359809875488, + "step": 1350, + "unclipped_grad_norm": 0.7215532064437866 + }, + { + "bits_per_byte": 2.1111642595512135, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.4633475542068481, + "step": 1360, + "unclipped_grad_norm": 0.7694162130355835 + }, + { + "bits_per_byte": 2.0903565945712748, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4489247798919678, + "step": 1370, + "unclipped_grad_norm": 0.7001428604125977 + }, + { + "bits_per_byte": 2.04615842916074, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.4182889461517334, + "step": 1380, + "unclipped_grad_norm": 0.7102509140968323 + }, + { + "bits_per_byte": 2.0505161255683015, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.421309471130371, + "step": 1390, + "unclipped_grad_norm": 0.7400784492492676 + }, + { + "bits_per_byte": 1.9793060210479523, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.371950387954712, + "step": 1400, + "unclipped_grad_norm": 0.8035197854042053 + }, + { + "bits_per_byte": 2.0066942581978546, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.3909344673156738, + "step": 1410, + "unclipped_grad_norm": 0.7851306796073914 + }, + { + "bits_per_byte": 2.045760977254596, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.4180134534835815, + "step": 1420, + "unclipped_grad_norm": 0.7472310066223145 + }, + { + "bits_per_byte": 2.077360381593525, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.4399164915084839, + "step": 1430, + "unclipped_grad_norm": 0.771403431892395 + }, + { + "bits_per_byte": 2.025465304626276, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.4039455652236938, + "step": 1440, + "unclipped_grad_norm": 0.7264323830604553 + }, + { + "bits_per_byte": 1.9586063611727556, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.3576024770736694, + "step": 1450, + "unclipped_grad_norm": 0.6727918386459351 + }, + { + "bits_per_byte": 2.0233887860997584, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.4025062322616577, + "step": 1460, + "unclipped_grad_norm": 0.822909414768219 + }, + { + "bits_per_byte": 1.9480701880329474, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.35029935836792, + "step": 1470, + "unclipped_grad_norm": 0.6954723000526428 + }, + { + "bits_per_byte": 1.999692328533277, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.3860810995101929, + "step": 1480, + "unclipped_grad_norm": 0.7725124955177307 + }, + { + "bits_per_byte": 2.0372935834220165, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.4121443033218384, + "step": 1490, + "unclipped_grad_norm": 0.7179713845252991 + }, + { + "bits_per_byte": 2.0175219419308754, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.398439645767212, + "step": 1500, + "unclipped_grad_norm": 0.7370100617408752 + }, + { + "bits_per_byte": 2.0160968936858263, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.3974518775939941, + "step": 1510, + "unclipped_grad_norm": 0.743471622467041 + }, + { + "bits_per_byte": 1.9661135758655681, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.3628060817718506, + "step": 1520, + "unclipped_grad_norm": 0.783790647983551 + }, + { + "bits_per_byte": 2.0459152456924197, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.4181203842163086, + "step": 1530, + "unclipped_grad_norm": 0.8134282231330872 + }, + { + "bits_per_byte": 1.9944171046833623, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3824245929718018, + "step": 1540, + "unclipped_grad_norm": 0.7183938026428223 + }, + { + "bits_per_byte": 2.0539129549054933, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.4236639738082886, + "step": 1550, + "unclipped_grad_norm": 0.7512509226799011 + }, + { + "bits_per_byte": 1.9679570579001664, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.3640838861465454, + "step": 1560, + "unclipped_grad_norm": 0.7425466179847717 + }, + { + "bits_per_byte": 2.039061737055537, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.41336989402771, + "step": 1570, + "unclipped_grad_norm": 0.734654426574707 + }, + { + "bits_per_byte": 1.9997533823743334, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.3861234188079834, + "step": 1580, + "unclipped_grad_norm": 0.7791112065315247 + }, + { + "bits_per_byte": 1.9916653822695574, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3805172443389893, + "step": 1590, + "unclipped_grad_norm": 0.677095353603363 + }, + { + "bits_per_byte": 2.0273311444054865, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.4052388668060303, + "step": 1600, + "unclipped_grad_norm": 0.7420785427093506 + }, + { + "bits_per_byte": 1.9617567393712605, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.3597861528396606, + "step": 1610, + "unclipped_grad_norm": 0.7535207271575928 + }, + { + "bits_per_byte": 1.9265297049776824, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.3353686332702637, + "step": 1620, + "unclipped_grad_norm": 0.6667174100875854 + }, + { + "bits_per_byte": 1.9517581839980493, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.3528556823730469, + "step": 1630, + "unclipped_grad_norm": 0.725364089012146 + }, + { + "bits_per_byte": 1.9890120339320458, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.3786780834197998, + "step": 1640, + "unclipped_grad_norm": 0.6893826127052307 + }, + { + "bits_per_byte": 2.0536652998882508, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.4234923124313354, + "step": 1650, + "unclipped_grad_norm": 0.7862091660499573 + }, + { + "bits_per_byte": 1.9885088126956212, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.3783292770385742, + "step": 1660, + "unclipped_grad_norm": 0.7903512120246887 + }, + { + "bits_per_byte": 1.993925922232498, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.3820841312408447, + "step": 1670, + "unclipped_grad_norm": 0.6966707706451416 + }, + { + "bits_per_byte": 2.0060665215222055, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.3904993534088135, + "step": 1680, + "unclipped_grad_norm": 0.7420340776443481 + }, + { + "bits_per_byte": 2.062234679432792, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.4294321537017822, + "step": 1690, + "unclipped_grad_norm": 0.7326623201370239 + }, + { + "bits_per_byte": 1.9473220635016941, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.349780797958374, + "step": 1700, + "unclipped_grad_norm": 0.66156405210495 + }, + { + "bits_per_byte": 1.9027651422814593, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3188962936401367, + "step": 1710, + "unclipped_grad_norm": 0.7105585932731628 + }, + { + "bits_per_byte": 1.8903479948891646, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.3102893829345703, + "step": 1720, + "unclipped_grad_norm": 0.6643191576004028 + }, + { + "bits_per_byte": 1.8665820563317348, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.293816089630127, + "step": 1730, + "unclipped_grad_norm": 0.7081220149993896 + }, + { + "bits_per_byte": 1.96802155139424, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.364128589630127, + "step": 1740, + "unclipped_grad_norm": 0.6791313290596008 + }, + { + "bits_per_byte": 1.9944721391316382, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.382462739944458, + "step": 1750, + "unclipped_grad_norm": 0.7308006286621094 + }, + { + "bits_per_byte": 1.991308346286366, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.3802697658538818, + "step": 1760, + "unclipped_grad_norm": 0.7906081080436707 + }, + { + "bits_per_byte": 2.00654583717016, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.3908315896987915, + "step": 1770, + "unclipped_grad_norm": 0.7655326128005981 + }, + { + "bits_per_byte": 1.8592688380690956, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.288746953010559, + "step": 1780, + "unclipped_grad_norm": 0.7074741721153259 + }, + { + "bits_per_byte": 1.952668832134368, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3534868955612183, + "step": 1790, + "unclipped_grad_norm": 0.7222051620483398 + }, + { + "bits_per_byte": 1.8861160177993834, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.3073559999465942, + "step": 1800, + "unclipped_grad_norm": 0.6845945119857788 + }, + { + "bits_per_byte": 1.9459288320470545, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.3488150835037231, + "step": 1810, + "unclipped_grad_norm": 0.6448895931243896 + }, + { + "bits_per_byte": 1.9107855532044464, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.3244556188583374, + "step": 1820, + "unclipped_grad_norm": 0.6578379273414612 + }, + { + "bits_per_byte": 1.8921146006788274, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.311513900756836, + "step": 1830, + "unclipped_grad_norm": 0.6559426188468933 + }, + { + "bits_per_byte": 1.905526151758411, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.320810079574585, + "step": 1840, + "unclipped_grad_norm": 0.677651584148407 + }, + { + "bits_per_byte": 1.9092674623452806, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.3234033584594727, + "step": 1850, + "unclipped_grad_norm": 0.7678938508033752 + }, + { + "bits_per_byte": 1.897424221058915, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.3151942491531372, + "step": 1860, + "unclipped_grad_norm": 0.6985909342765808 + }, + { + "bits_per_byte": 1.886600492926864, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.3076918125152588, + "step": 1870, + "unclipped_grad_norm": 0.7078948020935059 + }, + { + "bits_per_byte": 1.898806789589201, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.316152572631836, + "step": 1880, + "unclipped_grad_norm": 0.683250367641449 + }, + { + "bits_per_byte": 1.9117479681186746, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3251227140426636, + "step": 1890, + "unclipped_grad_norm": 0.68494713306427 + }, + { + "bits_per_byte": 1.91050522148354, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.3242613077163696, + "step": 1900, + "unclipped_grad_norm": 0.6974118947982788 + }, + { + "bits_per_byte": 1.8642745651051085, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.2922166585922241, + "step": 1910, + "unclipped_grad_norm": 0.6859782934188843 + }, + { + "bits_per_byte": 1.8595417745360148, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.2889361381530762, + "step": 1920, + "unclipped_grad_norm": 0.6479495167732239 + }, + { + "bits_per_byte": 1.94510073558315, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.3482410907745361, + "step": 1930, + "unclipped_grad_norm": 0.6703650951385498 + }, + { + "bits_per_byte": 1.8199764777744223, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.2615115642547607, + "step": 1940, + "unclipped_grad_norm": 0.6317785382270813 + }, + { + "bits_per_byte": 1.9155513644425057, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.327759027481079, + "step": 1950, + "unclipped_grad_norm": 0.6845464706420898 + }, + { + "bits_per_byte": 1.823228669702238, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.263765811920166, + "step": 1960, + "unclipped_grad_norm": 0.662417471408844 + }, + { + "bits_per_byte": 1.9053677557369664, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.3207002878189087, + "step": 1970, + "unclipped_grad_norm": 0.6656482219696045 + }, + { + "bits_per_byte": 1.9521315583330725, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.3531144857406616, + "step": 1980, + "unclipped_grad_norm": 0.7318056225776672 + }, + { + "bits_per_byte": 1.9373622042249272, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.3428771495819092, + "step": 1990, + "unclipped_grad_norm": 0.6949125528335571 + }, + { + "bits_per_byte": 1.900603148377463, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.3173977136611938, + "step": 2000, + "unclipped_grad_norm": 0.6581599712371826 + }, + { + "bits_per_byte": 1.8758416022541884, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.300234317779541, + "step": 2010, + "unclipped_grad_norm": 0.6078773140907288 + }, + { + "bits_per_byte": 1.9067026850729634, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.3216255903244019, + "step": 2020, + "unclipped_grad_norm": 0.6958262920379639 + }, + { + "bits_per_byte": 1.983292750877603, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.3747137784957886, + "step": 2030, + "unclipped_grad_norm": 0.6581296324729919 + }, + { + "bits_per_byte": 1.9112309882701812, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.324764370918274, + "step": 2040, + "unclipped_grad_norm": 0.6548293232917786 + }, + { + "bits_per_byte": 1.9557756987222046, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.3556404113769531, + "step": 2050, + "unclipped_grad_norm": 0.6823968291282654 + }, + { + "bits_per_byte": 1.9199252272192486, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.3307907581329346, + "step": 2060, + "unclipped_grad_norm": 0.6643821597099304 + }, + { + "bits_per_byte": 1.8721622054216296, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.2976839542388916, + "step": 2070, + "unclipped_grad_norm": 0.714849591255188 + }, + { + "bits_per_byte": 1.8679991933748443, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.2947983741760254, + "step": 2080, + "unclipped_grad_norm": 0.7407627105712891 + }, + { + "bits_per_byte": 1.875685442007205, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.300126075744629, + "step": 2090, + "unclipped_grad_norm": 0.7046352028846741 + }, + { + "bits_per_byte": 1.8501363873256296, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.282416820526123, + "step": 2100, + "unclipped_grad_norm": 0.6226866245269775 + }, + { + "bits_per_byte": 1.8931537198553405, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.3122341632843018, + "step": 2110, + "unclipped_grad_norm": 0.638272225856781 + }, + { + "bits_per_byte": 1.8568935856780293, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.2871005535125732, + "step": 2120, + "unclipped_grad_norm": 0.6061417460441589 + }, + { + "bits_per_byte": 1.8254683997644243, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.2653182744979858, + "step": 2130, + "unclipped_grad_norm": 0.6564081907272339 + }, + { + "bits_per_byte": 1.948432727460966, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.350550651550293, + "step": 2140, + "unclipped_grad_norm": 0.6463961005210876 + }, + { + "bits_per_byte": 1.8285157603550626, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2674305438995361, + "step": 2150, + "unclipped_grad_norm": 0.6475794315338135 + }, + { + "bits_per_byte": 1.8751418048478277, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.2997492551803589, + "step": 2160, + "unclipped_grad_norm": 0.7653019428253174 + }, + { + "bits_per_byte": 1.9566493705885877, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.356245994567871, + "step": 2170, + "unclipped_grad_norm": 0.6685433387756348 + }, + { + "bits_per_byte": 1.8327747387210291, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.2703826427459717, + "step": 2180, + "unclipped_grad_norm": 0.6575022339820862 + }, + { + "bits_per_byte": 1.845147858554703, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.278959035873413, + "step": 2190, + "unclipped_grad_norm": 0.6102811098098755 + }, + { + "bits_per_byte": 1.8820553354298617, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.3045413494110107, + "step": 2200, + "unclipped_grad_norm": 0.6623853445053101 + }, + { + "bits_per_byte": 1.8328992541602538, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.2704689502716064, + "step": 2210, + "unclipped_grad_norm": 0.6560285687446594 + }, + { + "bits_per_byte": 1.8367568250191066, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2731428146362305, + "step": 2220, + "unclipped_grad_norm": 0.6610417366027832 + }, + { + "bits_per_byte": 1.9033600302707938, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.3193086385726929, + "step": 2230, + "unclipped_grad_norm": 0.6783599257469177 + }, + { + "bits_per_byte": 1.8456421366932827, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.279301643371582, + "step": 2240, + "unclipped_grad_norm": 0.7856665253639221 + }, + { + "bits_per_byte": 1.8491072431428666, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2817034721374512, + "step": 2250, + "unclipped_grad_norm": 0.6108865141868591 + }, + { + "bits_per_byte": 1.829634335516274, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.2682058811187744, + "step": 2260, + "unclipped_grad_norm": 0.6457281112670898 + }, + { + "bits_per_byte": 1.8200751958160175, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.261579990386963, + "step": 2270, + "unclipped_grad_norm": 0.6352243423461914 + }, + { + "bits_per_byte": 1.8463887133806782, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.279819130897522, + "step": 2280, + "unclipped_grad_norm": 0.6741878986358643 + }, + { + "bits_per_byte": 1.8767995456194944, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.3008983135223389, + "step": 2290, + "unclipped_grad_norm": 0.6832013726234436 + }, + { + "bits_per_byte": 1.7966688729815419, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.2453559637069702, + "step": 2300, + "unclipped_grad_norm": 0.6234115958213806 + }, + { + "bits_per_byte": 1.8154861827430453, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.2583991289138794, + "step": 2310, + "unclipped_grad_norm": 0.5922688245773315 + }, + { + "bits_per_byte": 1.8041798712926733, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.2505621910095215, + "step": 2320, + "unclipped_grad_norm": 0.6404216289520264 + }, + { + "bits_per_byte": 1.8787744223993519, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.3022671937942505, + "step": 2330, + "unclipped_grad_norm": 0.636642575263977 + }, + { + "bits_per_byte": 1.7922862390896048, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.2423181533813477, + "step": 2340, + "unclipped_grad_norm": 0.6358290314674377 + }, + { + "bits_per_byte": 1.8506316973601145, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2827601432800293, + "step": 2350, + "unclipped_grad_norm": 0.6418381333351135 + }, + { + "bits_per_byte": 1.858295588247863, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.2880723476409912, + "step": 2360, + "unclipped_grad_norm": 0.6279385089874268 + }, + { + "bits_per_byte": 1.8775554093700364, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.3014222383499146, + "step": 2370, + "unclipped_grad_norm": 0.6207043528556824 + }, + { + "bits_per_byte": 1.8383786214167428, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.2742669582366943, + "step": 2380, + "unclipped_grad_norm": 0.6515224575996399 + }, + { + "bits_per_byte": 1.8946633835646143, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.3132805824279785, + "step": 2390, + "unclipped_grad_norm": 0.7189726829528809 + }, + { + "bits_per_byte": 1.8317641686645592, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.2696821689605713, + "step": 2400, + "unclipped_grad_norm": 0.6402539014816284 + }, + { + "bits_per_byte": 1.8239792019906034, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2642860412597656, + "step": 2410, + "unclipped_grad_norm": 0.656789243221283 + }, + { + "bits_per_byte": 1.8057149884342747, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.2516262531280518, + "step": 2420, + "unclipped_grad_norm": 0.6494315266609192 + }, + { + "bits_per_byte": 1.8177274606490894, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.2599526643753052, + "step": 2430, + "unclipped_grad_norm": 0.6307070851325989 + }, + { + "bits_per_byte": 1.8891331094434698, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.3094472885131836, + "step": 2440, + "unclipped_grad_norm": 0.62800532579422 + }, + { + "bits_per_byte": 1.8127318805894774, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.2564899921417236, + "step": 2450, + "unclipped_grad_norm": 0.5990567207336426 + }, + { + "bits_per_byte": 1.8415953849184807, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.2764966487884521, + "step": 2460, + "unclipped_grad_norm": 0.6279588341712952 + }, + { + "bits_per_byte": 1.8055199601081964, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.2514910697937012, + "step": 2470, + "unclipped_grad_norm": 0.6423900723457336 + }, + { + "bits_per_byte": 1.837602463713399, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.2737289667129517, + "step": 2480, + "unclipped_grad_norm": 0.6430677771568298 + }, + { + "bits_per_byte": 1.8147010819418565, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.25785493850708, + "step": 2490, + "unclipped_grad_norm": 0.6383248567581177 + }, + { + "bits_per_byte": 1.823237268834781, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2637717723846436, + "step": 2500, + "unclipped_grad_norm": 0.6207119226455688 + }, + { + "bits_per_byte": 1.7868851239392585, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.2385743856430054, + "step": 2510, + "unclipped_grad_norm": 0.6664624810218811 + }, + { + "bits_per_byte": 1.8497139979351105, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.2821240425109863, + "step": 2520, + "unclipped_grad_norm": 0.6723573803901672 + }, + { + "bits_per_byte": 1.860183269823733, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.2893807888031006, + "step": 2530, + "unclipped_grad_norm": 0.6805514097213745 + }, + { + "bits_per_byte": 1.8088952916140297, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.2538306713104248, + "step": 2540, + "unclipped_grad_norm": 0.6090428233146667 + }, + { + "bits_per_byte": 1.807309095625122, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.252731204032898, + "step": 2550, + "unclipped_grad_norm": 0.6697689890861511 + }, + { + "bits_per_byte": 1.8339791332250213, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.2712174654006958, + "step": 2560, + "unclipped_grad_norm": 0.711447536945343 + }, + { + "bits_per_byte": 1.8207660501245335, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2620588541030884, + "step": 2570, + "unclipped_grad_norm": 0.7240896224975586 + }, + { + "bits_per_byte": 1.8499687042410384, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.282300591468811, + "step": 2580, + "unclipped_grad_norm": 0.6432276964187622 + }, + { + "bits_per_byte": 1.799047908990927, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.2470049858093262, + "step": 2590, + "unclipped_grad_norm": 0.6087315082550049 + }, + { + "bits_per_byte": 1.8000796329134532, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.2477201223373413, + "step": 2600, + "unclipped_grad_norm": 0.6460713148117065 + }, + { + "bits_per_byte": 1.731975707050337, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.2005140781402588, + "step": 2610, + "unclipped_grad_norm": 0.6017140746116638 + }, + { + "bits_per_byte": 1.8080539524860089, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.2532474994659424, + "step": 2620, + "unclipped_grad_norm": 0.6441754102706909 + }, + { + "bits_per_byte": 1.816733916875055, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.2592639923095703, + "step": 2630, + "unclipped_grad_norm": 0.6813428401947021 + }, + { + "bits_per_byte": 1.678848030424354, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.1636887788772583, + "step": 2640, + "unclipped_grad_norm": 0.595856785774231 + }, + { + "bits_per_byte": 1.7752633963072042, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2305188179016113, + "step": 2650, + "unclipped_grad_norm": 0.6481371521949768 + }, + { + "bits_per_byte": 1.8084792655815927, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.2535423040390015, + "step": 2660, + "unclipped_grad_norm": 0.6557333469390869 + }, + { + "bits_per_byte": 1.8080642714450608, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.2532546520233154, + "step": 2670, + "unclipped_grad_norm": 0.6596212387084961 + }, + { + "bits_per_byte": 1.742593399966654, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.2078737020492554, + "step": 2680, + "unclipped_grad_norm": 0.614648163318634 + }, + { + "bits_per_byte": 1.8988219240624769, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.3161630630493164, + "step": 2690, + "unclipped_grad_norm": 0.6085010170936584 + }, + { + "bits_per_byte": 1.8138036764696543, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.257232904434204, + "step": 2700, + "unclipped_grad_norm": 0.671608567237854 + }, + { + "bits_per_byte": 1.733393188058748, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.2014966011047363, + "step": 2710, + "unclipped_grad_norm": 0.6095597743988037 + }, + { + "bits_per_byte": 1.7407918816988661, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.206624984741211, + "step": 2720, + "unclipped_grad_norm": 0.618058443069458 + }, + { + "bits_per_byte": 1.7766681505994515, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.231492519378662, + "step": 2730, + "unclipped_grad_norm": 0.6472728848457336 + }, + { + "bits_per_byte": 1.7733017621914628, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.2291591167449951, + "step": 2740, + "unclipped_grad_norm": 0.6257908344268799 + }, + { + "bits_per_byte": 1.795363696644144, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.2444512844085693, + "step": 2750, + "unclipped_grad_norm": 0.6372259259223938 + }, + { + "bits_per_byte": 1.8274453403360924, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.266688585281372, + "step": 2760, + "unclipped_grad_norm": 0.6607920527458191 + }, + { + "bits_per_byte": 1.7009056653107637, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.1789779663085938, + "step": 2770, + "unclipped_grad_norm": 0.6046813726425171 + }, + { + "bits_per_byte": 1.7231344229347816, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.1943857669830322, + "step": 2780, + "unclipped_grad_norm": 0.5984002351760864 + }, + { + "bits_per_byte": 1.8003735512637777, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2479238510131836, + "step": 2790, + "unclipped_grad_norm": 0.6106930375099182 + }, + { + "bits_per_byte": 1.8636562874752567, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.291788101196289, + "step": 2800, + "unclipped_grad_norm": 0.7129590511322021 + }, + { + "bits_per_byte": 1.7946411975278693, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.2439504861831665, + "step": 2810, + "unclipped_grad_norm": 0.6775277256965637 + }, + { + "bits_per_byte": 1.791927483279905, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2420694828033447, + "step": 2820, + "unclipped_grad_norm": 0.6178675889968872 + }, + { + "bits_per_byte": 1.865970142059965, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.2933919429779053, + "step": 2830, + "unclipped_grad_norm": 0.670330822467804 + }, + { + "bits_per_byte": 1.7515937680342575, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.2141122817993164, + "step": 2840, + "unclipped_grad_norm": 0.6283618211746216 + }, + { + "bits_per_byte": 1.772156013771415, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.2283649444580078, + "step": 2850, + "unclipped_grad_norm": 0.6352250576019287 + }, + { + "bits_per_byte": 1.748516998410322, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.211979627609253, + "step": 2860, + "unclipped_grad_norm": 0.6485711932182312 + }, + { + "bits_per_byte": 1.8177360597816326, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.2599586248397827, + "step": 2870, + "unclipped_grad_norm": 0.6308175325393677 + }, + { + "bits_per_byte": 1.8428422591372362, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2773609161376953, + "step": 2880, + "unclipped_grad_norm": 0.6145550608634949 + }, + { + "bits_per_byte": 1.736003540733544, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.203305959701538, + "step": 2890, + "unclipped_grad_norm": 0.6001958847045898 + }, + { + "bits_per_byte": 1.6967804894471683, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.1761186122894287, + "step": 2900, + "unclipped_grad_norm": 0.5927048921585083 + }, + { + "bits_per_byte": 1.7640587266034922, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.222752332687378, + "step": 2910, + "unclipped_grad_norm": 0.6438782811164856 + }, + { + "bits_per_byte": 1.7472266125808982, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.2110852003097534, + "step": 2920, + "unclipped_grad_norm": 0.6283231377601624 + }, + { + "bits_per_byte": 1.755904169212832, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.2171000242233276, + "step": 2930, + "unclipped_grad_norm": 0.6333559155464172 + }, + { + "bits_per_byte": 1.7778348809029048, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.2323012351989746, + "step": 2940, + "unclipped_grad_norm": 0.6731265187263489 + }, + { + "bits_per_byte": 1.7617189026385036, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2211304903030396, + "step": 2950, + "unclipped_grad_norm": 0.6760640144348145 + }, + { + "bits_per_byte": 1.8026423463939598, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.2494964599609375, + "step": 2960, + "unclipped_grad_norm": 0.6630797386169434 + }, + { + "bits_per_byte": 1.7652381836231092, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.2235698699951172, + "step": 2970, + "unclipped_grad_norm": 0.6644895672798157 + }, + { + "bits_per_byte": 1.7522786029499933, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.2145869731903076, + "step": 2980, + "unclipped_grad_norm": 0.6685414910316467 + }, + { + "bits_per_byte": 1.7757676494395338, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.2308683395385742, + "step": 2990, + "unclipped_grad_norm": 0.6160974502563477 + }, + { + "bits_per_byte": 1.7724707420224939, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.2285830974578857, + "step": 3000, + "unclipped_grad_norm": 0.607780396938324 + }, + { + "bits_per_byte": 1.7762887568716481, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.231229543685913, + "step": 3010, + "unclipped_grad_norm": 0.6294593811035156 + }, + { + "bits_per_byte": 1.7199477563969447, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.1921769380569458, + "step": 3020, + "unclipped_grad_norm": 0.6106687188148499 + }, + { + "bits_per_byte": 1.8015117324471877, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.2487127780914307, + "step": 3030, + "unclipped_grad_norm": 0.666127622127533 + }, + { + "bits_per_byte": 1.7212353905039546, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.1930694580078125, + "step": 3040, + "unclipped_grad_norm": 0.6306331753730774 + }, + { + "bits_per_byte": 1.757228091659174, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.2180176973342896, + "step": 3050, + "unclipped_grad_norm": 0.6245158314704895 + }, + { + "bits_per_byte": 1.7509691270463237, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.213679313659668, + "step": 3060, + "unclipped_grad_norm": 0.639885425567627 + }, + { + "bits_per_byte": 1.691025605983997, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.1721296310424805, + "step": 3070, + "unclipped_grad_norm": 0.6122934818267822 + }, + { + "bits_per_byte": 1.7294964051554986, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.1987955570220947, + "step": 3080, + "unclipped_grad_norm": 0.610792875289917 + }, + { + "bits_per_byte": 1.7942898369721567, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.2437069416046143, + "step": 3090, + "unclipped_grad_norm": 0.6861922144889832 + }, + { + "bits_per_byte": 1.7002087916094675, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.178494930267334, + "step": 3100, + "unclipped_grad_norm": 0.6396893858909607 + }, + { + "bits_per_byte": 1.7804619158948343, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.2341221570968628, + "step": 3110, + "unclipped_grad_norm": 0.6085984110832214 + }, + { + "bits_per_byte": 1.7344268037904336, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.2022130489349365, + "step": 3120, + "unclipped_grad_norm": 0.6325599551200867 + }, + { + "bits_per_byte": 1.804337063435562, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.250671148300171, + "step": 3130, + "unclipped_grad_norm": 0.6261355876922607 + }, + { + "bits_per_byte": 1.7675470507109425, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.2251702547073364, + "step": 3140, + "unclipped_grad_norm": 0.6559895873069763 + }, + { + "bits_per_byte": 1.7346134049666198, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.2023423910140991, + "step": 3150, + "unclipped_grad_norm": 0.6373011469841003 + }, + { + "bits_per_byte": 1.7581727923603634, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.218672513961792, + "step": 3160, + "unclipped_grad_norm": 0.6461610198020935 + }, + { + "bits_per_byte": 1.6645058812209519, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.15374755859375, + "step": 3170, + "unclipped_grad_norm": 0.5917196869850159 + }, + { + "bits_per_byte": 1.7640267378304317, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.2227301597595215, + "step": 3180, + "unclipped_grad_norm": 0.6284464001655579 + }, + { + "bits_per_byte": 1.73202816175885, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.2005504369735718, + "step": 3190, + "unclipped_grad_norm": 0.5928532481193542 + }, + { + "bits_per_byte": 1.69034249089477, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.1716561317443848, + "step": 3200, + "unclipped_grad_norm": 0.5921319723129272 + }, + { + "bits_per_byte": 1.7046081078185382, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.181544303894043, + "step": 3210, + "unclipped_grad_norm": 0.6365057826042175 + }, + { + "bits_per_byte": 1.707051465339346, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.183237910270691, + "step": 3220, + "unclipped_grad_norm": 0.594910740852356 + }, + { + "bits_per_byte": 1.7874896429570413, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.2389934062957764, + "step": 3230, + "unclipped_grad_norm": 0.689224123954773 + }, + { + "bits_per_byte": 1.679454957199249, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.164109468460083, + "step": 3240, + "unclipped_grad_norm": 0.5985220074653625 + }, + { + "bits_per_byte": 1.8380165979366767, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.27401602268219, + "step": 3250, + "unclipped_grad_norm": 0.6826193928718567 + }, + { + "bits_per_byte": 1.68937973201524, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.1709887981414795, + "step": 3260, + "unclipped_grad_norm": 0.6134940385818481 + }, + { + "bits_per_byte": 1.7100723406017513, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.1853318214416504, + "step": 3270, + "unclipped_grad_norm": 0.6108191609382629 + }, + { + "bits_per_byte": 1.7702869063218378, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.22706937789917, + "step": 3280, + "unclipped_grad_norm": 0.6297270655632019 + }, + { + "bits_per_byte": 1.7746875983921155, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.2301197052001953, + "step": 3290, + "unclipped_grad_norm": 0.664413571357727 + }, + { + "bits_per_byte": 1.6921553600175148, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.1729127168655396, + "step": 3300, + "unclipped_grad_norm": 0.615717887878418 + }, + { + "bits_per_byte": 1.725326513802679, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.1959052085876465, + "step": 3310, + "unclipped_grad_norm": 0.6334171891212463 + }, + { + "bits_per_byte": 1.7268232788131381, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.196942687034607, + "step": 3320, + "unclipped_grad_norm": 0.6215294599533081 + }, + { + "bits_per_byte": 1.740506390498434, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.2064270973205566, + "step": 3330, + "unclipped_grad_norm": 0.6459984183311462 + }, + { + "bits_per_byte": 1.7147918885067286, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.188603162765503, + "step": 3340, + "unclipped_grad_norm": 0.6547986268997192 + }, + { + "bits_per_byte": 1.7022486778313513, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.1799088716506958, + "step": 3350, + "unclipped_grad_norm": 0.6313554048538208 + }, + { + "bits_per_byte": 1.7441553464017903, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.2089563608169556, + "step": 3360, + "unclipped_grad_norm": 0.6534625291824341 + }, + { + "bits_per_byte": 1.6944064409346582, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.1744730472564697, + "step": 3370, + "unclipped_grad_norm": 0.6572955846786499 + }, + { + "bits_per_byte": 1.7221503382065446, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.1937036514282227, + "step": 3380, + "unclipped_grad_norm": 0.5944768786430359 + }, + { + "bits_per_byte": 1.7498646544624827, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.2129137516021729, + "step": 3390, + "unclipped_grad_norm": 0.641020655632019 + }, + { + "bits_per_byte": 1.8062166618268416, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.2519739866256714, + "step": 3400, + "unclipped_grad_norm": 0.6903617978096008 + }, + { + "bits_per_byte": 1.8293177154560358, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.2679864168167114, + "step": 3410, + "unclipped_grad_norm": 0.6414300203323364 + }, + { + "bits_per_byte": 1.6874834513068266, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.1696743965148926, + "step": 3420, + "unclipped_grad_norm": 0.6029585599899292 + }, + { + "bits_per_byte": 1.6310922318980712, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1305869817733765, + "step": 3430, + "unclipped_grad_norm": 0.6737542152404785 + }, + { + "bits_per_byte": 1.715020969397678, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.1887619495391846, + "step": 3440, + "unclipped_grad_norm": 0.6997514963150024 + }, + { + "bits_per_byte": 1.7837383573764216, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.2363932132720947, + "step": 3450, + "unclipped_grad_norm": 0.6508408784866333 + }, + { + "bits_per_byte": 1.7008217377771426, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.178919792175293, + "step": 3460, + "unclipped_grad_norm": 0.6544042229652405 + }, + { + "bits_per_byte": 1.698592498656659, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.1773746013641357, + "step": 3470, + "unclipped_grad_norm": 0.6608152985572815 + }, + { + "bits_per_byte": 1.637566002841849, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.135074257850647, + "step": 3480, + "unclipped_grad_norm": 0.604396641254425 + }, + { + "bits_per_byte": 1.8491990818784272, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.2817671298980713, + "step": 3490, + "unclipped_grad_norm": 0.6825912594795227 + }, + { + "bits_per_byte": 1.7082491525199548, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1840680837631226, + "step": 3500, + "unclipped_grad_norm": 0.6790757775306702 + }, + { + "bits_per_byte": 1.7441649774302386, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.2089630365371704, + "step": 3510, + "unclipped_grad_norm": 0.6400102972984314 + }, + { + "bits_per_byte": 1.7249619105828498, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.1956524848937988, + "step": 3520, + "unclipped_grad_norm": 0.6423133611679077 + }, + { + "bits_per_byte": 1.7157216267172928, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.1892476081848145, + "step": 3530, + "unclipped_grad_norm": 0.6613569855690002 + }, + { + "bits_per_byte": 1.7255580024507404, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.1960656642913818, + "step": 3540, + "unclipped_grad_norm": 0.6887786388397217 + }, + { + "bits_per_byte": 1.7043931295049597, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.1813952922821045, + "step": 3550, + "unclipped_grad_norm": 0.639394223690033 + }, + { + "bits_per_byte": 1.6599995918030446, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.1506240367889404, + "step": 3560, + "unclipped_grad_norm": 0.6266466379165649 + }, + { + "bits_per_byte": 1.6771077379802735, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.162482500076294, + "step": 3570, + "unclipped_grad_norm": 0.6500812768936157 + }, + { + "bits_per_byte": 1.70040863544977, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.178633451461792, + "step": 3580, + "unclipped_grad_norm": 0.644574761390686 + }, + { + "bits_per_byte": 1.7343306654886013, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.2021464109420776, + "step": 3590, + "unclipped_grad_norm": 0.671259880065918 + }, + { + "bits_per_byte": 1.747299705207515, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.2111358642578125, + "step": 3600, + "unclipped_grad_norm": 0.6472942233085632 + }, + { + "bits_per_byte": 1.6662810861431578, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.1549780368804932, + "step": 3610, + "unclipped_grad_norm": 0.608910322189331 + }, + { + "bits_per_byte": 1.64372349769069, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.1393423080444336, + "step": 3620, + "unclipped_grad_norm": 0.621802568435669 + }, + { + "bits_per_byte": 1.703449976647628, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.180741548538208, + "step": 3630, + "unclipped_grad_norm": 0.6392863392829895 + }, + { + "bits_per_byte": 1.6776752807281206, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.1628758907318115, + "step": 3640, + "unclipped_grad_norm": 0.5891351103782654 + }, + { + "bits_per_byte": 1.7084309381819167, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.1841940879821777, + "step": 3650, + "unclipped_grad_norm": 0.6256741285324097 + }, + { + "bits_per_byte": 1.630717481701841, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.1303272247314453, + "step": 3660, + "unclipped_grad_norm": 0.6102724075317383 + }, + { + "bits_per_byte": 1.7067716495663923, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.1830439567565918, + "step": 3670, + "unclipped_grad_norm": 0.6156262159347534 + }, + { + "bits_per_byte": 1.6514180014902922, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.1446757316589355, + "step": 3680, + "unclipped_grad_norm": 0.6071178913116455 + }, + { + "bits_per_byte": 1.6306356179600303, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.1302704811096191, + "step": 3690, + "unclipped_grad_norm": 0.6209743022918701 + }, + { + "bits_per_byte": 1.6608648365195353, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.1512237787246704, + "step": 3700, + "unclipped_grad_norm": 0.6676256656646729 + }, + { + "bits_per_byte": 1.7052348125982824, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.181978702545166, + "step": 3710, + "unclipped_grad_norm": 0.636160671710968 + }, + { + "bits_per_byte": 1.6914129109137401, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.1723980903625488, + "step": 3720, + "unclipped_grad_norm": 0.6435323357582092 + }, + { + "bits_per_byte": 1.6087709476078889, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.1151150465011597, + "step": 3730, + "unclipped_grad_norm": 0.6332689523696899 + }, + { + "bits_per_byte": 1.6706470377179612, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.1580042839050293, + "step": 3740, + "unclipped_grad_norm": 0.6307963728904724 + }, + { + "bits_per_byte": 1.6905688200633053, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1718130111694336, + "step": 3750, + "unclipped_grad_norm": 0.6598763465881348 + }, + { + "bits_per_byte": 1.7722440688886567, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.2284259796142578, + "step": 3760, + "unclipped_grad_norm": 0.640985906124115 + }, + { + "bits_per_byte": 1.7387026364561848, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.205176830291748, + "step": 3770, + "unclipped_grad_norm": 0.6711131930351257 + }, + { + "bits_per_byte": 1.5617717007627514, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.0825376510620117, + "step": 3780, + "unclipped_grad_norm": 0.6163194179534912 + }, + { + "bits_per_byte": 1.6614848339758959, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.151653528213501, + "step": 3790, + "unclipped_grad_norm": 0.6834599375724792 + }, + { + "bits_per_byte": 1.656327418241822, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.1480786800384521, + "step": 3800, + "unclipped_grad_norm": 0.6011042594909668 + }, + { + "bits_per_byte": 1.6923225271541535, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.173028588294983, + "step": 3810, + "unclipped_grad_norm": 0.6273730397224426 + }, + { + "bits_per_byte": 1.6737349862142028, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1601446866989136, + "step": 3820, + "unclipped_grad_norm": 0.6733741760253906 + }, + { + "bits_per_byte": 1.7115077078058523, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.1863267421722412, + "step": 3830, + "unclipped_grad_norm": 0.6350049376487732 + }, + { + "bits_per_byte": 1.6544032763439689, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.146744966506958, + "step": 3840, + "unclipped_grad_norm": 0.6300914287567139 + }, + { + "bits_per_byte": 1.5964234531891937, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1065564155578613, + "step": 3850, + "unclipped_grad_norm": 0.6059905290603638 + }, + { + "bits_per_byte": 1.6516942056275778, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.144867181777954, + "step": 3860, + "unclipped_grad_norm": 0.6342678666114807 + }, + { + "bits_per_byte": 1.7301702331815791, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.1992626190185547, + "step": 3870, + "unclipped_grad_norm": 0.6215102076530457 + }, + { + "bits_per_byte": 1.6283571918014, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1286911964416504, + "step": 3880, + "unclipped_grad_norm": 0.5960464477539062 + }, + { + "bits_per_byte": 1.6026741626348024, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.1108890771865845, + "step": 3890, + "unclipped_grad_norm": 0.6551031470298767 + }, + { + "bits_per_byte": 1.6802876971947267, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.164686679840088, + "step": 3900, + "unclipped_grad_norm": 0.6671215295791626 + }, + { + "bits_per_byte": 1.6830413114176912, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1665953397750854, + "step": 3910, + "unclipped_grad_norm": 0.6533258557319641 + }, + { + "bits_per_byte": 1.6358492720209363, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.133884310722351, + "step": 3920, + "unclipped_grad_norm": 0.6260588765144348 + }, + { + "bits_per_byte": 1.6750401625516007, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.1610493659973145, + "step": 3930, + "unclipped_grad_norm": 0.6286660432815552 + }, + { + "bits_per_byte": 1.690076777699187, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.1714719533920288, + "step": 3940, + "unclipped_grad_norm": 0.638534665107727 + }, + { + "bits_per_byte": 1.7253781085979378, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1959409713745117, + "step": 3950, + "unclipped_grad_norm": 0.6417268514633179 + }, + { + "bits_per_byte": 1.7046425043487108, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.1815681457519531, + "step": 3960, + "unclipped_grad_norm": 0.6444203853607178 + }, + { + "bits_per_byte": 1.6336202048831037, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1323392391204834, + "step": 3970, + "unclipped_grad_norm": 0.6553277373313904 + }, + { + "bits_per_byte": 1.6738094547020264, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.160196304321289, + "step": 3980, + "unclipped_grad_norm": 0.6473615765571594 + }, + { + "bits_per_byte": 1.6985402159307965, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1773383617401123, + "step": 3990, + "unclipped_grad_norm": 0.6596603989601135 + }, + { + "bits_per_byte": 1.5694868424804571, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.0878853797912598, + "step": 4000, + "unclipped_grad_norm": 0.6214215755462646 + }, + { + "bits_per_byte": 1.6703202706813218, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.1577777862548828, + "step": 4010, + "unclipped_grad_norm": 0.6871203780174255 + }, + { + "bits_per_byte": 1.6408489796641692, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.1373498439788818, + "step": 4020, + "unclipped_grad_norm": 0.645101010799408 + }, + { + "bits_per_byte": 1.6883333895673907, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.1702635288238525, + "step": 4030, + "unclipped_grad_norm": 0.6595595479011536 + }, + { + "bits_per_byte": 1.60955054496425, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1156554222106934, + "step": 4040, + "unclipped_grad_norm": 0.6268433332443237 + }, + { + "bits_per_byte": 1.647514339281008, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.141969919204712, + "step": 4050, + "unclipped_grad_norm": 0.6421452760696411 + }, + { + "bits_per_byte": 1.6559180995327685, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1477949619293213, + "step": 4060, + "unclipped_grad_norm": 0.6443775296211243 + }, + { + "bits_per_byte": 1.6818680457735051, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.16578209400177, + "step": 4070, + "unclipped_grad_norm": 0.6187630891799927 + }, + { + "bits_per_byte": 1.7579626295610091, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.218526840209961, + "step": 4080, + "unclipped_grad_norm": 0.7221863865852356 + }, + { + "bits_per_byte": 1.645009927919144, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.1402339935302734, + "step": 4090, + "unclipped_grad_norm": 0.6955327391624451 + }, + { + "bits_per_byte": 1.6512618412433087, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.1445674896240234, + "step": 4100, + "unclipped_grad_norm": 0.650931179523468 + }, + { + "bits_per_byte": 1.63509650395811, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.1333625316619873, + "step": 4110, + "unclipped_grad_norm": 0.6691992878913879 + }, + { + "bits_per_byte": 1.6736204457687283, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.1600652933120728, + "step": 4120, + "unclipped_grad_norm": 0.6472700834274292 + }, + { + "bits_per_byte": 1.6208185042834795, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.1234657764434814, + "step": 4130, + "unclipped_grad_norm": 0.6534613370895386 + }, + { + "bits_per_byte": 1.6688040716313153, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.1567268371582031, + "step": 4140, + "unclipped_grad_norm": 0.6454835534095764 + }, + { + "bits_per_byte": 1.6179322914366998, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1214652061462402, + "step": 4150, + "unclipped_grad_norm": 0.6616674065589905 + }, + { + "bits_per_byte": 1.6541207088486012, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.146549105644226, + "step": 4160, + "unclipped_grad_norm": 0.6816359758377075 + }, + { + "bits_per_byte": 1.6906610027641678, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.1718769073486328, + "step": 4170, + "unclipped_grad_norm": 0.6716027855873108 + }, + { + "bits_per_byte": 1.7341571349938807, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.202026128768921, + "step": 4180, + "unclipped_grad_norm": 0.7155779004096985 + }, + { + "bits_per_byte": 1.6401925218858258, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.1368948221206665, + "step": 4190, + "unclipped_grad_norm": 0.6572866439819336 + }, + { + "bits_per_byte": 1.68856247045834, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1704223155975342, + "step": 4200, + "unclipped_grad_norm": 0.6852372884750366 + }, + { + "bits_per_byte": 1.666490217046607, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.155122995376587, + "step": 4210, + "unclipped_grad_norm": 0.645636796951294 + }, + { + "bits_per_byte": 1.6858561514643626, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.168546438217163, + "step": 4220, + "unclipped_grad_norm": 0.6557843089103699 + }, + { + "bits_per_byte": 1.6662501292660026, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.154956579208374, + "step": 4230, + "unclipped_grad_norm": 0.6790511012077332 + }, + { + "bits_per_byte": 1.609906549051536, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.1159021854400635, + "step": 4240, + "unclipped_grad_norm": 0.6366406083106995 + }, + { + "bits_per_byte": 1.649572971611836, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1433968544006348, + "step": 4250, + "unclipped_grad_norm": 0.6794030070304871 + }, + { + "bits_per_byte": 1.628443527092133, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1287510395050049, + "step": 4260, + "unclipped_grad_norm": 0.6431719064712524 + }, + { + "bits_per_byte": 1.6184941587570687, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.1218546628952026, + "step": 4270, + "unclipped_grad_norm": 0.6492335200309753 + }, + { + "bits_per_byte": 1.621275462186822, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1237825155258179, + "step": 4280, + "unclipped_grad_norm": 0.6348761916160583 + }, + { + "bits_per_byte": 1.6786368357290946, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.16354238986969, + "step": 4290, + "unclipped_grad_norm": 0.6547877788543701 + }, + { + "bits_per_byte": 1.593915774156963, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.1048182249069214, + "step": 4300, + "unclipped_grad_norm": 0.6597434282302856 + }, + { + "bits_per_byte": 1.686149037918782, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.168749451637268, + "step": 4310, + "unclipped_grad_norm": 0.6848970055580139 + }, + { + "bits_per_byte": 1.6599087849633891, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.1505610942840576, + "step": 4320, + "unclipped_grad_norm": 0.6610612869262695 + }, + { + "bits_per_byte": 1.6178521475213978, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.1214096546173096, + "step": 4330, + "unclipped_grad_norm": 0.6605933904647827 + }, + { + "bits_per_byte": 1.6522273518452526, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.1452367305755615, + "step": 4340, + "unclipped_grad_norm": 0.6742057800292969 + }, + { + "bits_per_byte": 1.6295751729348102, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.129535436630249, + "step": 4350, + "unclipped_grad_norm": 0.6645446419715881 + }, + { + "bits_per_byte": 1.6214135642554648, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.1238782405853271, + "step": 4360, + "unclipped_grad_norm": 0.639927864074707 + }, + { + "bits_per_byte": 1.6188106068346562, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.122074007987976, + "step": 4370, + "unclipped_grad_norm": 0.6986302137374878 + }, + { + "bits_per_byte": 1.6898489006867936, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.171314001083374, + "step": 4380, + "unclipped_grad_norm": 0.6770191788673401 + }, + { + "bits_per_byte": 1.64916915634761, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1431169509887695, + "step": 4390, + "unclipped_grad_norm": 0.6751537322998047 + }, + { + "bits_per_byte": 1.5996969709657163, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.108825445175171, + "step": 4400, + "unclipped_grad_norm": 0.6509587168693542 + }, + { + "bits_per_byte": 1.6680851841507087, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.1562285423278809, + "step": 4410, + "unclipped_grad_norm": 0.6488450169563293 + }, + { + "bits_per_byte": 1.6133293477690076, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.1182746887207031, + "step": 4420, + "unclipped_grad_norm": 0.6429536938667297 + }, + { + "bits_per_byte": 1.6717196935113925, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.1587477922439575, + "step": 4430, + "unclipped_grad_norm": 0.6755688190460205 + }, + { + "bits_per_byte": 1.6113522352146887, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.1169042587280273, + "step": 4440, + "unclipped_grad_norm": 0.6762297749519348 + }, + { + "bits_per_byte": 1.594701562888755, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.105362892150879, + "step": 4450, + "unclipped_grad_norm": 0.6539916396141052 + }, + { + "bits_per_byte": 1.574604014274229, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.0914323329925537, + "step": 4460, + "unclipped_grad_norm": 0.6519320011138916 + }, + { + "bits_per_byte": 1.6233662552733612, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1252317428588867, + "step": 4470, + "unclipped_grad_norm": 0.6663927435874939 + }, + { + "bits_per_byte": 1.651233980053869, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.1445481777191162, + "step": 4480, + "unclipped_grad_norm": 0.6907956004142761 + }, + { + "bits_per_byte": 1.6254278113092537, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.126660704612732, + "step": 4490, + "unclipped_grad_norm": 0.6732251048088074 + }, + { + "bits_per_byte": 1.6504411400333914, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.143998622894287, + "step": 4500, + "unclipped_grad_norm": 0.6376641988754272 + }, + { + "bits_per_byte": 1.5969390571764803, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.1069138050079346, + "step": 4510, + "unclipped_grad_norm": 0.6637102961540222 + }, + { + "bits_per_byte": 1.6246974009910393, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.1261544227600098, + "step": 4520, + "unclipped_grad_norm": 0.6717531681060791 + }, + { + "bits_per_byte": 1.6581115662618726, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.149315357208252, + "step": 4530, + "unclipped_grad_norm": 0.6531894207000732 + }, + { + "bits_per_byte": 1.7167389040971464, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.1899527311325073, + "step": 4540, + "unclipped_grad_norm": 0.703811764717102 + }, + { + "bits_per_byte": 1.5547266034528073, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.0776543617248535, + "step": 4550, + "unclipped_grad_norm": 0.6300640106201172 + }, + { + "bits_per_byte": 1.6345066034656506, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.1329536437988281, + "step": 4560, + "unclipped_grad_norm": 0.723333477973938 + }, + { + "bits_per_byte": 1.6147346180092077, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.1192487478256226, + "step": 4570, + "unclipped_grad_norm": 0.6539926528930664 + }, + { + "bits_per_byte": 1.5679087296761398, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.0867915153503418, + "step": 4580, + "unclipped_grad_norm": 0.6434015035629272 + }, + { + "bits_per_byte": 1.6062840784764127, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.1133912801742554, + "step": 4590, + "unclipped_grad_norm": 0.6853119134902954 + }, + { + "bits_per_byte": 1.6093730588685595, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.115532398223877, + "step": 4600, + "unclipped_grad_norm": 0.670010507106781 + }, + { + "bits_per_byte": 1.6695516802146158, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.1572450399398804, + "step": 4610, + "unclipped_grad_norm": 0.7086074948310852 + }, + { + "bits_per_byte": 1.6683496934677358, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.15641188621521, + "step": 4620, + "unclipped_grad_norm": 0.6766663789749146 + }, + { + "bits_per_byte": 1.639615692074832, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.1364949941635132, + "step": 4630, + "unclipped_grad_norm": 0.6698880195617676 + }, + { + "bits_per_byte": 1.6970934978717387, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.1763355731964111, + "step": 4640, + "unclipped_grad_norm": 0.703077495098114 + }, + { + "bits_per_byte": 1.5958786121512603, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.1061787605285645, + "step": 4650, + "unclipped_grad_norm": 0.6709136962890625 + }, + { + "bits_per_byte": 1.5661840876532878, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.0855960845947266, + "step": 4660, + "unclipped_grad_norm": 0.6643497347831726 + }, + { + "bits_per_byte": 1.7242953058281056, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.1951904296875, + "step": 4670, + "unclipped_grad_norm": 0.7115020751953125 + }, + { + "bits_per_byte": 1.70608750258126, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.1825697422027588, + "step": 4680, + "unclipped_grad_norm": 0.7020435333251953 + }, + { + "bits_per_byte": 1.6853506944534768, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.1681960821151733, + "step": 4690, + "unclipped_grad_norm": 0.7577327489852905 + }, + { + "bits_per_byte": 1.5994532715494436, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.1086565256118774, + "step": 4700, + "unclipped_grad_norm": 0.6607687473297119 + }, + { + "bits_per_byte": 1.6667487069708538, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.1553021669387817, + "step": 4710, + "unclipped_grad_norm": 0.7177369594573975 + }, + { + "bits_per_byte": 1.6056974456543196, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.1129846572875977, + "step": 4720, + "unclipped_grad_norm": 0.6873557567596436 + }, + { + "bits_per_byte": 1.5233628153455605, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.0559146404266357, + "step": 4730, + "unclipped_grad_norm": 0.6391798853874207 + }, + { + "bits_per_byte": 1.5809453585768425, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.0958278179168701, + "step": 4740, + "unclipped_grad_norm": 0.6574237942695618 + }, + { + "bits_per_byte": 1.630707678690742, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.130320429801941, + "step": 4750, + "unclipped_grad_norm": 0.6879438161849976 + }, + { + "bits_per_byte": 1.6100228093235194, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.1159827709197998, + "step": 4760, + "unclipped_grad_norm": 0.6870896816253662 + }, + { + "bits_per_byte": 1.5769849420927737, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.0930826663970947, + "step": 4770, + "unclipped_grad_norm": 0.6955029368400574 + }, + { + "bits_per_byte": 1.626970495687493, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.1277300119400024, + "step": 4780, + "unclipped_grad_norm": 0.703714907169342 + }, + { + "bits_per_byte": 1.568998067786705, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.0875465869903564, + "step": 4790, + "unclipped_grad_norm": 0.6794602274894714 + }, + { + "bits_per_byte": 1.5737296544772426, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.0908262729644775, + "step": 4800, + "unclipped_grad_norm": 0.7120228409767151 + }, + { + "bits_per_byte": 1.6262536719986969, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.1272331476211548, + "step": 4810, + "unclipped_grad_norm": 0.7055649757385254 + }, + { + "bits_per_byte": 1.5716624230138716, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.0893933773040771, + "step": 4820, + "unclipped_grad_norm": 0.6697573065757751 + }, + { + "bits_per_byte": 1.6060875023064765, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.1132550239562988, + "step": 4830, + "unclipped_grad_norm": 0.710551381111145 + }, + { + "bits_per_byte": 1.5948934955271181, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.1054959297180176, + "step": 4840, + "unclipped_grad_norm": 0.7147494554519653 + }, + { + "bits_per_byte": 1.6218992432615014, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.1242148876190186, + "step": 4850, + "unclipped_grad_norm": 0.6700032353401184 + }, + { + "bits_per_byte": 1.5424350033956424, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.0691344738006592, + "step": 4860, + "unclipped_grad_norm": 0.6617462635040283 + }, + { + "bits_per_byte": 1.626479141253978, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.1273894309997559, + "step": 4870, + "unclipped_grad_norm": 0.7252988815307617 + }, + { + "bits_per_byte": 1.483838622437524, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.028518557548523, + "step": 4880, + "unclipped_grad_norm": 0.6700296998023987 + }, + { + "bits_per_byte": 1.540047540236365, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.0674796104431152, + "step": 4890, + "unclipped_grad_norm": 0.6948246359825134 + }, + { + "bits_per_byte": 1.5882991647450855, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.100925087928772, + "step": 4900, + "unclipped_grad_norm": 0.7125248908996582 + }, + { + "bits_per_byte": 1.6945509063613828, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.1745731830596924, + "step": 4910, + "unclipped_grad_norm": 0.7282727956771851 + }, + { + "bits_per_byte": 1.620069347856321, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.1229465007781982, + "step": 4920, + "unclipped_grad_norm": 0.6988062858581543 + }, + { + "bits_per_byte": 1.577541305968315, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.0934683084487915, + "step": 4930, + "unclipped_grad_norm": 0.6639720797538757 + }, + { + "bits_per_byte": 1.5697768052298118, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.0880863666534424, + "step": 4940, + "unclipped_grad_norm": 0.6598480939865112 + }, + { + "bits_per_byte": 1.5230164422867227, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.0556745529174805, + "step": 4950, + "unclipped_grad_norm": 0.653609037399292 + }, + { + "bits_per_byte": 1.5246480416954582, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.0568054914474487, + "step": 4960, + "unclipped_grad_norm": 0.6958068013191223 + }, + { + "bits_per_byte": 1.5995143253905, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.108698844909668, + "step": 4970, + "unclipped_grad_norm": 0.7115789651870728 + }, + { + "bits_per_byte": 1.650496862412271, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.1440372467041016, + "step": 4980, + "unclipped_grad_norm": 0.7071532607078552 + }, + { + "bits_per_byte": 1.5967141038691517, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.1067578792572021, + "step": 4990, + "unclipped_grad_norm": 0.7384892106056213 + }, + { + "bits_per_byte": 1.5399667083904593, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.0674235820770264, + "step": 5000, + "unclipped_grad_norm": 0.6687040328979492 + }, + { + "bits_per_byte": 1.5343267093380644, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.063514232635498, + "step": 5010, + "unclipped_grad_norm": 0.6622800827026367 + }, + { + "bits_per_byte": 1.5808822409439758, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.095784068107605, + "step": 5020, + "unclipped_grad_norm": 0.6676583886146545 + }, + { + "bits_per_byte": 1.5838780067393552, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.09786057472229, + "step": 5030, + "unclipped_grad_norm": 0.7090701460838318 + }, + { + "bits_per_byte": 1.541603123313419, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.068557858467102, + "step": 5040, + "unclipped_grad_norm": 0.6795203685760498 + }, + { + "bits_per_byte": 1.507806296644416, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.0451316833496094, + "step": 5050, + "unclipped_grad_norm": 0.6525983214378357 + }, + { + "bits_per_byte": 1.5288581769885796, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.0597237348556519, + "step": 5060, + "unclipped_grad_norm": 0.651759684085846 + }, + { + "bits_per_byte": 1.6122224674280545, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.1175074577331543, + "step": 5070, + "unclipped_grad_norm": 0.6996030807495117 + }, + { + "bits_per_byte": 1.5728480714089197, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.0902152061462402, + "step": 5080, + "unclipped_grad_norm": 0.7345776557922363 + }, + { + "bits_per_byte": 1.5561607667783524, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.0786484479904175, + "step": 5090, + "unclipped_grad_norm": 0.6930875182151794 + }, + { + "bits_per_byte": 1.5598473868822476, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.081203818321228, + "step": 5100, + "unclipped_grad_norm": 0.6548681259155273 + }, + { + "bits_per_byte": 1.4730458511826285, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.0210375785827637, + "step": 5110, + "unclipped_grad_norm": 0.6644712686538696 + }, + { + "bits_per_byte": 1.5811238765684381, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0959515571594238, + "step": 5120, + "unclipped_grad_norm": 0.6931653022766113 + }, + { + "bits_per_byte": 1.5844427977647886, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.0982520580291748, + "step": 5130, + "unclipped_grad_norm": 0.686448872089386 + }, + { + "bits_per_byte": 1.5481269412085978, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.0730798244476318, + "step": 5140, + "unclipped_grad_norm": 0.679078996181488 + }, + { + "bits_per_byte": 1.6151162475114722, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.1195132732391357, + "step": 5150, + "unclipped_grad_norm": 0.7298036813735962 + }, + { + "bits_per_byte": 1.5006640291367344, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.040181040763855, + "step": 5160, + "unclipped_grad_norm": 0.7023690342903137 + }, + { + "bits_per_byte": 1.6822286933923642, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.166032075881958, + "step": 5170, + "unclipped_grad_norm": 0.7808181643486023 + }, + { + "bits_per_byte": 1.5830977214523907, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.0973197221755981, + "step": 5180, + "unclipped_grad_norm": 0.7051005363464355 + }, + { + "bits_per_byte": 1.5177551490315278, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.052027702331543, + "step": 5190, + "unclipped_grad_norm": 0.6638994216918945 + }, + { + "bits_per_byte": 1.5519999905060284, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.0757644176483154, + "step": 5200, + "unclipped_grad_norm": 0.7110773921012878 + }, + { + "bits_per_byte": 1.553785514387286, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.0770020484924316, + "step": 5210, + "unclipped_grad_norm": 0.6727936863899231 + }, + { + "bits_per_byte": 1.6304519404889088, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.130143165588379, + "step": 5220, + "unclipped_grad_norm": 0.737336277961731 + }, + { + "bits_per_byte": 1.5485336801778884, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.0733617544174194, + "step": 5230, + "unclipped_grad_norm": 0.7087617516517639 + }, + { + "bits_per_byte": 1.5433107390538359, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0697414875030518, + "step": 5240, + "unclipped_grad_norm": 0.6938673257827759 + }, + { + "bits_per_byte": 1.5172201110046935, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0516568422317505, + "step": 5250, + "unclipped_grad_norm": 0.677476167678833 + }, + { + "bits_per_byte": 1.5285298621080827, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.0594961643218994, + "step": 5260, + "unclipped_grad_norm": 0.7000418305397034 + }, + { + "bits_per_byte": 1.530004785321882, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.060518503189087, + "step": 5270, + "unclipped_grad_norm": 0.6852406859397888 + }, + { + "bits_per_byte": 1.5473502675573014, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.0725414752960205, + "step": 5280, + "unclipped_grad_norm": 0.7066041231155396 + }, + { + "bits_per_byte": 1.5522775705045209, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.0759568214416504, + "step": 5290, + "unclipped_grad_norm": 0.7406570315361023 + }, + { + "bits_per_byte": 1.5518569009405105, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.0756652355194092, + "step": 5300, + "unclipped_grad_norm": 0.7126148343086243 + }, + { + "bits_per_byte": 1.5478675913710969, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.0729000568389893, + "step": 5310, + "unclipped_grad_norm": 0.6942803859710693 + }, + { + "bits_per_byte": 1.550509932818953, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0747315883636475, + "step": 5320, + "unclipped_grad_norm": 0.6907455325126648 + }, + { + "bits_per_byte": 1.544284676805672, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.0704165697097778, + "step": 5330, + "unclipped_grad_norm": 0.7187883853912354 + }, + { + "bits_per_byte": 1.4951578325867105, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.0363644361495972, + "step": 5340, + "unclipped_grad_norm": 0.7044789791107178 + }, + { + "bits_per_byte": 1.580473266200224, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.0955005884170532, + "step": 5350, + "unclipped_grad_norm": 0.726965606212616 + }, + { + "bits_per_byte": 1.5009686104114124, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.0403921604156494, + "step": 5360, + "unclipped_grad_norm": 0.7176401615142822 + }, + { + "bits_per_byte": 1.4711079506727063, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.0196943283081055, + "step": 5370, + "unclipped_grad_norm": 0.7109264135360718 + }, + { + "bits_per_byte": 1.5906493076691257, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.1025540828704834, + "step": 5380, + "unclipped_grad_norm": 0.7163623571395874 + }, + { + "bits_per_byte": 1.5261117860369515, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.0578200817108154, + "step": 5390, + "unclipped_grad_norm": 0.7170036435127258 + }, + { + "bits_per_byte": 1.5557079364586306, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.0783345699310303, + "step": 5400, + "unclipped_grad_norm": 0.7109150290489197 + }, + { + "bits_per_byte": 1.5523785243205774, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.0760267972946167, + "step": 5410, + "unclipped_grad_norm": 0.6841652989387512 + }, + { + "bits_per_byte": 1.5366292130678156, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.065110206604004, + "step": 5420, + "unclipped_grad_norm": 0.7674379348754883 + }, + { + "bits_per_byte": 1.570237718734124, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.0884058475494385, + "step": 5430, + "unclipped_grad_norm": 0.7482572793960571 + }, + { + "bits_per_byte": 1.5099595194332183, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.0466241836547852, + "step": 5440, + "unclipped_grad_norm": 0.7148683667182922 + }, + { + "bits_per_byte": 1.5041766027979564, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.0426157712936401, + "step": 5450, + "unclipped_grad_norm": 0.7205177545547485 + }, + { + "bits_per_byte": 1.4857185647941054, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.0298216342926025, + "step": 5460, + "unclipped_grad_norm": 0.699296236038208 + }, + { + "bits_per_byte": 1.5412328446661114, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0683012008666992, + "step": 5470, + "unclipped_grad_norm": 0.7379934787750244 + }, + { + "bits_per_byte": 1.553612499840518, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.0768821239471436, + "step": 5480, + "unclipped_grad_norm": 0.7247312664985657 + }, + { + "bits_per_byte": 1.5568433659196268, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.0791215896606445, + "step": 5490, + "unclipped_grad_norm": 0.7230545282363892 + }, + { + "bits_per_byte": 1.5454637898599874, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.071233868598938, + "step": 5500, + "unclipped_grad_norm": 0.7159186005592346 + }, + { + "bits_per_byte": 1.6399625810816223, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.136735439300537, + "step": 5510, + "unclipped_grad_norm": 0.7487865686416626 + }, + { + "bits_per_byte": 1.5508425472657217, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.0749621391296387, + "step": 5520, + "unclipped_grad_norm": 0.7227869629859924 + }, + { + "bits_per_byte": 1.507233250451741, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.0447344779968262, + "step": 5530, + "unclipped_grad_norm": 0.7033931612968445 + }, + { + "bits_per_byte": 1.597289213853637, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.10715651512146, + "step": 5540, + "unclipped_grad_norm": 0.7876261472702026 + }, + { + "bits_per_byte": 1.512559897114264, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.048426628112793, + "step": 5550, + "unclipped_grad_norm": 0.7062801122665405 + }, + { + "bits_per_byte": 1.5152815225641678, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.050313115119934, + "step": 5560, + "unclipped_grad_norm": 0.754061222076416 + }, + { + "bits_per_byte": 1.4831498319208185, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.028041124343872, + "step": 5570, + "unclipped_grad_norm": 0.7236031293869019 + }, + { + "bits_per_byte": 1.5690197376007136, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.0875616073608398, + "step": 5580, + "unclipped_grad_norm": 0.7374637722969055 + }, + { + "bits_per_byte": 1.5019833080515028, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.041095495223999, + "step": 5590, + "unclipped_grad_norm": 0.7137759923934937 + }, + { + "bits_per_byte": 1.54801171283252, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.0729999542236328, + "step": 5600, + "unclipped_grad_norm": 0.7259743809700012 + }, + { + "bits_per_byte": 1.5450353810766881, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.070936918258667, + "step": 5610, + "unclipped_grad_norm": 0.6988793611526489 + }, + { + "bits_per_byte": 1.5537045105587297, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.0769459009170532, + "step": 5620, + "unclipped_grad_norm": 0.72163325548172 + }, + { + "bits_per_byte": 1.5318818039733988, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0618195533752441, + "step": 5630, + "unclipped_grad_norm": 0.7131519317626953 + }, + { + "bits_per_byte": 1.5027740842801702, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.0416436195373535, + "step": 5640, + "unclipped_grad_norm": 0.7095826864242554 + }, + { + "bits_per_byte": 1.5224679896131212, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.055294394493103, + "step": 5650, + "unclipped_grad_norm": 0.7380594611167908 + }, + { + "bits_per_byte": 1.590806843777316, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.102663278579712, + "step": 5660, + "unclipped_grad_norm": 0.7485193610191345 + }, + { + "bits_per_byte": 1.5054829830139103, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.0435212850570679, + "step": 5670, + "unclipped_grad_norm": 0.7387163639068604 + }, + { + "bits_per_byte": 1.5299944663628302, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0605113506317139, + "step": 5680, + "unclipped_grad_norm": 0.7047367691993713 + }, + { + "bits_per_byte": 1.5861086217210458, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.0994067192077637, + "step": 5690, + "unclipped_grad_norm": 0.7179912328720093 + }, + { + "bits_per_byte": 1.515916826476455, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.0507534742355347, + "step": 5700, + "unclipped_grad_norm": 0.7373331785202026 + }, + { + "bits_per_byte": 1.527769698791269, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.058969259262085, + "step": 5710, + "unclipped_grad_norm": 0.7480198740959167 + }, + { + "bits_per_byte": 1.5396189594704148, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0671825408935547, + "step": 5720, + "unclipped_grad_norm": 0.6888155341148376 + }, + { + "bits_per_byte": 1.5500142788191664, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.074388027191162, + "step": 5730, + "unclipped_grad_norm": 0.7189749479293823 + }, + { + "bits_per_byte": 1.5236374716389882, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.0561050176620483, + "step": 5740, + "unclipped_grad_norm": 0.7438164949417114 + }, + { + "bits_per_byte": 1.5411162404288263, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.0682203769683838, + "step": 5750, + "unclipped_grad_norm": 0.794143795967102 + }, + { + "bits_per_byte": 1.5646985015151347, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.084566354751587, + "step": 5760, + "unclipped_grad_norm": 0.7443080544471741 + }, + { + "bits_per_byte": 1.5111640859198614, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.0474591255187988, + "step": 5770, + "unclipped_grad_norm": 0.7111071944236755 + }, + { + "bits_per_byte": 1.4933853793869185, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.0351358652114868, + "step": 5780, + "unclipped_grad_norm": 0.7597989439964294 + }, + { + "bits_per_byte": 1.51288804001211, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 1.0486540794372559, + "step": 5790, + "unclipped_grad_norm": 0.7139179110527039 + }, + { + "bits_per_byte": 1.48582313024583, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.0298941135406494, + "step": 5800, + "unclipped_grad_norm": 0.7527741193771362 + }, + { + "bits_per_byte": 1.569064281107287, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.0875924825668335, + "step": 5810, + "unclipped_grad_norm": 0.7431255578994751 + }, + { + "bits_per_byte": 1.4993827583878063, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.0392929315567017, + "step": 5820, + "unclipped_grad_norm": 0.7169746160507202 + }, + { + "bits_per_byte": 1.504188125635564, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.04262375831604, + "step": 5830, + "unclipped_grad_norm": 0.7262318134307861 + }, + { + "bits_per_byte": 1.5003941883575305, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.0399940013885498, + "step": 5840, + "unclipped_grad_norm": 0.726586639881134 + }, + { + "bits_per_byte": 1.4721355470116115, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.0204066038131714, + "step": 5850, + "unclipped_grad_norm": 0.7553265690803528 + }, + { + "bits_per_byte": 1.5378941454649118, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.06598699092865, + "step": 5860, + "unclipped_grad_norm": 0.7790977358818054 + }, + { + "bits_per_byte": 1.5240519498275678, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0563923120498657, + "step": 5870, + "unclipped_grad_norm": 0.7502213716506958 + }, + { + "bits_per_byte": 1.5468687161348855, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.0722076892852783, + "step": 5880, + "unclipped_grad_norm": 0.767290472984314 + }, + { + "bits_per_byte": 1.5495691877187334, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.0740795135498047, + "step": 5890, + "unclipped_grad_norm": 0.7468048930168152 + }, + { + "bits_per_byte": 1.5445479822441428, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.07059907913208, + "step": 5900, + "unclipped_grad_norm": 0.7380504012107849 + }, + { + "bits_per_byte": 1.5492126676834947, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.073832392692566, + "step": 5910, + "unclipped_grad_norm": 0.7607431411743164 + }, + { + "bits_per_byte": 1.4937749200911228, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.0354058742523193, + "step": 5920, + "unclipped_grad_norm": 0.7359480261802673 + }, + { + "bits_per_byte": 1.5842945487197448, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.098149299621582, + "step": 5930, + "unclipped_grad_norm": 0.7712169289588928 + }, + { + "bits_per_byte": 1.449506241793734, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.0047211647033691, + "step": 5940, + "unclipped_grad_norm": 0.7011276483535767 + }, + { + "bits_per_byte": 1.504274804891599, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.0426838397979736, + "step": 5950, + "unclipped_grad_norm": 0.7238391041755676 + }, + { + "bits_per_byte": 1.5397549977472473, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.0672768354415894, + "step": 5960, + "unclipped_grad_norm": 0.7776085734367371 + }, + { + "bits_per_byte": 1.5223966168130132, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.0552449226379395, + "step": 5970, + "unclipped_grad_norm": 0.8004531264305115 + }, + { + "bits_per_byte": 1.5577928821350404, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.0797797441482544, + "step": 5980, + "unclipped_grad_norm": 0.7434797883033752 + }, + { + "bits_per_byte": 1.5245451960702423, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0567342042922974, + "step": 5990, + "unclipped_grad_norm": 0.7621127367019653 + }, + { + "bits_per_byte": 1.4749999180617317, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.0223920345306396, + "step": 6000, + "unclipped_grad_norm": 0.7376543283462524 + }, + { + "bits_per_byte": 1.5041539010880425, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0426000356674194, + "step": 6010, + "unclipped_grad_norm": 0.7570760846138 + }, + { + "bits_per_byte": 1.4634781123498286, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0144057273864746, + "step": 6020, + "unclipped_grad_norm": 0.7092445492744446 + }, + { + "bits_per_byte": 1.5053727421347072, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.0434448719024658, + "step": 6030, + "unclipped_grad_norm": 0.7244347333908081 + }, + { + "bits_per_byte": 1.5688415635744197, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.0874381065368652, + "step": 6040, + "unclipped_grad_norm": 0.7976362705230713 + }, + { + "bits_per_byte": 1.4774671811710096, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.0241022109985352, + "step": 6050, + "unclipped_grad_norm": 0.7659919857978821 + }, + { + "bits_per_byte": 1.4645516280565143, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.0151498317718506, + "step": 6060, + "unclipped_grad_norm": 0.7441014051437378 + }, + { + "bits_per_byte": 1.5433717928948922, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 1.0697838068008423, + "step": 6070, + "unclipped_grad_norm": 0.7805195450782776 + }, + { + "bits_per_byte": 1.4776137103895446, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.0242037773132324, + "step": 6080, + "unclipped_grad_norm": 0.7491018772125244 + }, + { + "bits_per_byte": 1.486596708209411, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.0304303169250488, + "step": 6090, + "unclipped_grad_norm": 0.7174160480499268 + }, + { + "bits_per_byte": 1.47886454020927, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.0250707864761353, + "step": 6100, + "unclipped_grad_norm": 0.7355735301971436 + }, + { + "bits_per_byte": 1.5482112127075207, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.0731382369995117, + "step": 6110, + "unclipped_grad_norm": 0.7645819187164307 + }, + { + "bits_per_byte": 1.5315517692663931, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0615907907485962, + "step": 6120, + "unclipped_grad_norm": 0.7616561651229858 + }, + { + "bits_per_byte": 1.4671989570012456, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.0169848203659058, + "step": 6130, + "unclipped_grad_norm": 0.7611680030822754 + }, + { + "bits_per_byte": 1.4875635946725616, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.0311005115509033, + "step": 6140, + "unclipped_grad_norm": 0.7648262977600098 + }, + { + "bits_per_byte": 1.5445696520581516, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.0706140995025635, + "step": 6150, + "unclipped_grad_norm": 0.7702162861824036 + }, + { + "bits_per_byte": 1.4100173053291247, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 0.9773495197296143, + "step": 6160, + "unclipped_grad_norm": 0.7114855051040649 + }, + { + "bits_per_byte": 1.5173619966916552, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 1.0517551898956299, + "step": 6170, + "unclipped_grad_norm": 0.7402294874191284 + }, + { + "bits_per_byte": 1.4772606300073232, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0239590406417847, + "step": 6180, + "unclipped_grad_norm": 0.7306315302848816 + }, + { + "bits_per_byte": 1.495765275309558, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.0367854833602905, + "step": 6190, + "unclipped_grad_norm": 0.7585544586181641 + }, + { + "bits_per_byte": 1.5043707712107803, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.042750358581543, + "step": 6200, + "unclipped_grad_norm": 0.747658908367157 + }, + { + "bits_per_byte": 1.5010081664211108, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.040419578552246, + "step": 6210, + "unclipped_grad_norm": 0.7751714587211609 + }, + { + "bits_per_byte": 1.5035775872250012, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0422005653381348, + "step": 6220, + "unclipped_grad_norm": 0.7300167083740234 + }, + { + "bits_per_byte": 1.4580931635686634, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.01067316532135, + "step": 6230, + "unclipped_grad_norm": 0.7508504390716553 + }, + { + "bits_per_byte": 1.4949628042606322, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.0362292528152466, + "step": 6240, + "unclipped_grad_norm": 0.8087684512138367 + }, + { + "bits_per_byte": 1.5367451293744974, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.0651905536651611, + "step": 6250, + "unclipped_grad_norm": 0.7689018845558167 + }, + { + "bits_per_byte": 1.4083784826490529, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 0.9762135744094849, + "step": 6260, + "unclipped_grad_norm": 0.7644893527030945 + }, + { + "bits_per_byte": 1.5135271275427165, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.0490970611572266, + "step": 6270, + "unclipped_grad_norm": 0.7905248999595642 + }, + { + "bits_per_byte": 1.5059574831476408, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.0438501834869385, + "step": 6280, + "unclipped_grad_norm": 0.8002632856369019 + }, + { + "bits_per_byte": 1.4386743444857673, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 0.9972130656242371, + "step": 6290, + "unclipped_grad_norm": 0.7424378395080566 + }, + { + "bits_per_byte": 1.5108663839512178, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.0472527742385864, + "step": 6300, + "unclipped_grad_norm": 0.7815948724746704 + }, + { + "bits_per_byte": 1.4647595550814074, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.0152939558029175, + "step": 6310, + "unclipped_grad_norm": 0.7460298538208008 + }, + { + "bits_per_byte": 1.420051375119739, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 0.9843046069145203, + "step": 6320, + "unclipped_grad_norm": 0.7480984330177307 + }, + { + "bits_per_byte": 1.511718901951545, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 1.0478436946868896, + "step": 6330, + "unclipped_grad_norm": 0.7749253511428833 + }, + { + "bits_per_byte": 1.4587121291291185, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 1.0111021995544434, + "step": 6340, + "unclipped_grad_norm": 0.750465989112854 + }, + { + "bits_per_byte": 1.5740031068921143, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.0910158157348633, + "step": 6350, + "unclipped_grad_norm": 0.8142428994178772 + }, + { + "bits_per_byte": 1.4492702815967504, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.0045576095581055, + "step": 6360, + "unclipped_grad_norm": 0.804401695728302 + }, + { + "bits_per_byte": 1.4448909153751799, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 1.0015220642089844, + "step": 6370, + "unclipped_grad_norm": 0.7500267028808594 + }, + { + "bits_per_byte": 1.4666350258890664, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.0165939331054688, + "step": 6380, + "unclipped_grad_norm": 0.7499048709869385 + }, + { + "bits_per_byte": 1.5287405408553896, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0596421957015991, + "step": 6390, + "unclipped_grad_norm": 0.7983035445213318 + }, + { + "bits_per_byte": 1.5132952949293532, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.048936367034912, + "step": 6400, + "unclipped_grad_norm": 0.7945475578308105 + }, + { + "bits_per_byte": 1.4842119967725471, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.0287773609161377, + "step": 6410, + "unclipped_grad_norm": 0.8107759952545166 + }, + { + "bits_per_byte": 1.4670899200005985, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0169092416763306, + "step": 6420, + "unclipped_grad_norm": 0.791615903377533 + }, + { + "bits_per_byte": 1.5136435597973505, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.0491777658462524, + "step": 6430, + "unclipped_grad_norm": 0.7751617431640625 + }, + { + "bits_per_byte": 1.4515793206672345, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.0061581134796143, + "step": 6440, + "unclipped_grad_norm": 0.7813049554824829 + }, + { + "bits_per_byte": 1.523892521910218, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0562818050384521, + "step": 6450, + "unclipped_grad_norm": 0.8193799257278442 + }, + { + "bits_per_byte": 1.5827926242297599, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.097108244895935, + "step": 6460, + "unclipped_grad_norm": 0.815384566783905 + }, + { + "bits_per_byte": 1.5468171213396267, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.072171926498413, + "step": 6470, + "unclipped_grad_norm": 0.8593401312828064 + }, + { + "bits_per_byte": 1.4332476039204423, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 0.9934515357017517, + "step": 6480, + "unclipped_grad_norm": 0.7557542324066162 + }, + { + "bits_per_byte": 1.538845381506834, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0666463375091553, + "step": 6490, + "unclipped_grad_norm": 0.802010178565979 + }, + { + "bits_per_byte": 1.512460835107367, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.0483579635620117, + "step": 6500, + "unclipped_grad_norm": 0.811919093132019 + }, + { + "bits_per_byte": 1.4542873594877201, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 1.0080351829528809, + "step": 6510, + "unclipped_grad_norm": 0.7658334374427795 + }, + { + "bits_per_byte": 1.413757584020089, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 0.9799420833587646, + "step": 6520, + "unclipped_grad_norm": 0.7362275719642639 + }, + { + "bits_per_byte": 1.458498526676747, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.0109541416168213, + "step": 6530, + "unclipped_grad_norm": 0.7702703475952148 + }, + { + "bits_per_byte": 1.4985250809079536, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.038698434829712, + "step": 6540, + "unclipped_grad_norm": 0.8070250749588013 + }, + { + "bits_per_byte": 1.4283477322060354, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 0.9900552034378052, + "step": 6550, + "unclipped_grad_norm": 0.7630698680877686 + }, + { + "bits_per_byte": 1.5059423486743648, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.043839693069458, + "step": 6560, + "unclipped_grad_norm": 0.7847185730934143 + }, + { + "bits_per_byte": 1.4655090554738677, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 1.0158134698867798, + "step": 6570, + "unclipped_grad_norm": 0.7837479710578918 + }, + { + "bits_per_byte": 1.5058788870761963, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.0437957048416138, + "step": 6580, + "unclipped_grad_norm": 0.8260042667388916 + }, + { + "bits_per_byte": 1.4850650307208266, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.0293686389923096, + "step": 6590, + "unclipped_grad_norm": 0.7737330794334412 + }, + { + "bits_per_byte": 1.4176852378091684, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9826645255088806, + "step": 6600, + "unclipped_grad_norm": 0.7685086131095886 + }, + { + "bits_per_byte": 1.4428018421151492, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.000074028968811, + "step": 6610, + "unclipped_grad_norm": 0.7548773884773254 + }, + { + "bits_per_byte": 1.4245123471091439, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 0.9873967170715332, + "step": 6620, + "unclipped_grad_norm": 0.7893918752670288 + }, + { + "bits_per_byte": 1.463975486176124, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.0147504806518555, + "step": 6630, + "unclipped_grad_norm": 0.7861996293067932 + }, + { + "bits_per_byte": 1.5099068927420543, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.0465877056121826, + "step": 6640, + "unclipped_grad_norm": 0.8437765836715698 + }, + { + "bits_per_byte": 1.5082221506942022, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.0454199314117432, + "step": 6650, + "unclipped_grad_norm": 0.7631747126579285 + }, + { + "bits_per_byte": 1.5093254193994872, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.0461846590042114, + "step": 6660, + "unclipped_grad_norm": 0.7733337879180908 + }, + { + "bits_per_byte": 1.4808910117843863, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.026475429534912, + "step": 6670, + "unclipped_grad_norm": 0.8145992159843445 + }, + { + "bits_per_byte": 1.4583311875574574, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.0108381509780884, + "step": 6680, + "unclipped_grad_norm": 0.7763416767120361 + }, + { + "bits_per_byte": 1.48509357984087, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.029388427734375, + "step": 6690, + "unclipped_grad_norm": 0.8123944401741028 + }, + { + "bits_per_byte": 1.540551449403393, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.067828893661499, + "step": 6700, + "unclipped_grad_norm": 0.8398496508598328 + }, + { + "bits_per_byte": 1.5131737031951933, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.0488520860671997, + "step": 6710, + "unclipped_grad_norm": 0.782952606678009 + }, + { + "bits_per_byte": 1.525510706672186, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.0574034452438354, + "step": 6720, + "unclipped_grad_norm": 0.8299216628074646 + }, + { + "bits_per_byte": 1.4785902278811438, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 1.0248806476593018, + "step": 6730, + "unclipped_grad_norm": 0.7975350618362427 + }, + { + "bits_per_byte": 1.4183937203393977, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 0.9831556081771851, + "step": 6740, + "unclipped_grad_norm": 0.7755722999572754 + }, + { + "bits_per_byte": 1.4976699831678637, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 1.0381057262420654, + "step": 6750, + "unclipped_grad_norm": 0.8062617182731628 + }, + { + "bits_per_byte": 1.4670971432719349, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 1.0169142484664917, + "step": 6760, + "unclipped_grad_norm": 0.785228431224823 + }, + { + "bits_per_byte": 1.3873008049072597, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 0.9616036415100098, + "step": 6770, + "unclipped_grad_norm": 0.7726664543151855 + }, + { + "bits_per_byte": 1.4705109988915614, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 1.0192805528640747, + "step": 6780, + "unclipped_grad_norm": 0.8086632490158081 + }, + { + "bits_per_byte": 1.4429045157577143, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.0001451969146729, + "step": 6790, + "unclipped_grad_norm": 0.818213701248169 + }, + { + "bits_per_byte": 1.5045286512842726, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0428597927093506, + "step": 6800, + "unclipped_grad_norm": 0.8267188668251038 + }, + { + "bits_per_byte": 1.501266140397405, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0405983924865723, + "step": 6810, + "unclipped_grad_norm": 0.8420217633247375 + }, + { + "bits_per_byte": 1.4878318876079075, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 1.0312864780426025, + "step": 6820, + "unclipped_grad_norm": 0.8419885039329529 + }, + { + "bits_per_byte": 1.4652020664420775, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.0156006813049316, + "step": 6830, + "unclipped_grad_norm": 0.782660722732544 + }, + { + "bits_per_byte": 1.4358041260255179, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 0.9952235817909241, + "step": 6840, + "unclipped_grad_norm": 0.8103845119476318 + }, + { + "bits_per_byte": 1.4906290134415403, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.0332252979278564, + "step": 6850, + "unclipped_grad_norm": 0.8234423995018005 + }, + { + "bits_per_byte": 1.4584503715345054, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.010920763015747, + "step": 6860, + "unclipped_grad_norm": 0.7921655178070068 + }, + { + "bits_per_byte": 1.4079278881037924, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 0.9759012460708618, + "step": 6870, + "unclipped_grad_norm": 0.7879679799079895 + }, + { + "bits_per_byte": 1.4277146640682095, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 0.9896163940429688, + "step": 6880, + "unclipped_grad_norm": 0.7540534734725952 + }, + { + "bits_per_byte": 1.4341213617781507, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 0.9940571784973145, + "step": 6890, + "unclipped_grad_norm": 0.7987163662910461 + }, + { + "bits_per_byte": 1.5343765843068147, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 1.0635488033294678, + "step": 6900, + "unclipped_grad_norm": 0.8171630501747131 + }, + { + "bits_per_byte": 1.492606469961161, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.0345959663391113, + "step": 6910, + "unclipped_grad_norm": 0.8321383595466614 + }, + { + "bits_per_byte": 1.4313504632987746, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 0.992136538028717, + "step": 6920, + "unclipped_grad_norm": 0.8009306192398071 + }, + { + "bits_per_byte": 1.4367294786784852, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.995864987373352, + "step": 6930, + "unclipped_grad_norm": 0.8468543291091919 + }, + { + "bits_per_byte": 1.4102267801978756, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 0.9774947166442871, + "step": 6940, + "unclipped_grad_norm": 0.7895923256874084 + }, + { + "bits_per_byte": 1.4339222058684515, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 0.9939191341400146, + "step": 6950, + "unclipped_grad_norm": 0.8116638660430908 + }, + { + "bits_per_byte": 1.4886653155339888, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 1.0318641662597656, + "step": 6960, + "unclipped_grad_norm": 0.8071072101593018 + }, + { + "bits_per_byte": 1.4701075275926374, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 1.0190008878707886, + "step": 6970, + "unclipped_grad_norm": 0.8008604645729065 + }, + { + "bits_per_byte": 1.4455920886427476, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.002008080482483, + "step": 6980, + "unclipped_grad_norm": 0.7700145840644836 + }, + { + "bits_per_byte": 1.484166077404767, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.0287455320358276, + "step": 6990, + "unclipped_grad_norm": 0.8664272427558899 + }, + { + "bits_per_byte": 1.5201646259701156, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 1.0536978244781494, + "step": 7000, + "unclipped_grad_norm": 0.823028564453125 + }, + { + "bits_per_byte": 1.512097435766094, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 1.048106074333191, + "step": 7010, + "unclipped_grad_norm": 0.8564846515655518 + }, + { + "bits_per_byte": 1.4570136284691975, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 1.0099248886108398, + "step": 7020, + "unclipped_grad_norm": 0.8378328084945679 + }, + { + "bits_per_byte": 1.5138229377022003, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 1.049302101135254, + "step": 7030, + "unclipped_grad_norm": 0.8506705164909363 + }, + { + "bits_per_byte": 1.4941986853428488, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.0356996059417725, + "step": 7040, + "unclipped_grad_norm": 0.8149515986442566 + }, + { + "bits_per_byte": 1.485162544883866, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.0294362306594849, + "step": 7050, + "unclipped_grad_norm": 0.7959237694740295 + }, + { + "bits_per_byte": 1.4833622304946341, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 1.0281883478164673, + "step": 7060, + "unclipped_grad_norm": 0.846172571182251 + }, + { + "bits_per_byte": 1.452753274242024, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 1.006971836090088, + "step": 7070, + "unclipped_grad_norm": 0.8459955453872681 + }, + { + "bits_per_byte": 1.4353539614368844, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 0.9949115514755249, + "step": 7080, + "unclipped_grad_norm": 0.8262940049171448 + }, + { + "bits_per_byte": 1.458162472576961, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 1.010721206665039, + "step": 7090, + "unclipped_grad_norm": 0.8544578552246094 + }, + { + "bits_per_byte": 1.4575949298291138, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 1.0103278160095215, + "step": 7100, + "unclipped_grad_norm": 0.7949163913726807 + }, + { + "bits_per_byte": 1.4806316619468853, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 1.0262956619262695, + "step": 7110, + "unclipped_grad_norm": 0.8059237599372864 + }, + { + "bits_per_byte": 1.5197814486239933, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.0534322261810303, + "step": 7120, + "unclipped_grad_norm": 0.8576845526695251 + }, + { + "bits_per_byte": 1.4784170413517248, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 1.0247606039047241, + "step": 7130, + "unclipped_grad_norm": 0.8865160942077637 + }, + { + "bits_per_byte": 1.4469725933812232, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.002964973449707, + "step": 7140, + "unclipped_grad_norm": 0.8292717933654785 + }, + { + "bits_per_byte": 1.431326987666932, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 0.9921202659606934, + "step": 7150, + "unclipped_grad_norm": 0.7803593873977661 + }, + { + "bits_per_byte": 1.508955484717481, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0459282398223877, + "step": 7160, + "unclipped_grad_norm": 0.8625690340995789 + }, + { + "bits_per_byte": 1.4504712364477252, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.0053900480270386, + "step": 7170, + "unclipped_grad_norm": 0.8165751695632935 + }, + { + "bits_per_byte": 1.471559577113872, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 1.0200073719024658, + "step": 7180, + "unclipped_grad_norm": 0.8274741768836975 + }, + { + "bits_per_byte": 1.4350309780185642, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 0.9946876764297485, + "step": 7190, + "unclipped_grad_norm": 0.8526366949081421 + }, + { + "bits_per_byte": 1.4706673311211957, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 1.0193889141082764, + "step": 7200, + "unclipped_grad_norm": 0.8346464037895203 + }, + { + "bits_per_byte": 1.5350235830393606, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 1.0639972686767578, + "step": 7210, + "unclipped_grad_norm": 0.8620064854621887 + }, + { + "bits_per_byte": 1.504267237654961, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 1.0426785945892334, + "step": 7220, + "unclipped_grad_norm": 0.8516167998313904 + }, + { + "bits_per_byte": 1.4407633317544724, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 0.9986610412597656, + "step": 7230, + "unclipped_grad_norm": 0.8479061126708984 + }, + { + "bits_per_byte": 1.4553653467433283, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 1.0087823867797852, + "step": 7240, + "unclipped_grad_norm": 0.816892147064209 + }, + { + "bits_per_byte": 1.4837560707651098, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 1.0284613370895386, + "step": 7250, + "unclipped_grad_norm": 0.8647069931030273 + }, + { + "bits_per_byte": 1.3830619485114442, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 0.9586654901504517, + "step": 7260, + "unclipped_grad_norm": 0.8161391615867615 + }, + { + "bits_per_byte": 1.4094007475257815, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 0.9769221544265747, + "step": 7270, + "unclipped_grad_norm": 0.8319637179374695 + }, + { + "bits_per_byte": 1.4853560253660867, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.0295703411102295, + "step": 7280, + "unclipped_grad_norm": 0.8720204830169678 + }, + { + "bits_per_byte": 1.4114874130286998, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 0.9783685207366943, + "step": 7290, + "unclipped_grad_norm": 0.80472332239151 + }, + { + "bits_per_byte": 1.4451105372203317, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 1.0016742944717407, + "step": 7300, + "unclipped_grad_norm": 0.8547036647796631 + }, + { + "bits_per_byte": 1.4482086326929742, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 1.0038217306137085, + "step": 7310, + "unclipped_grad_norm": 0.8266134262084961 + }, + { + "bits_per_byte": 1.3891315602256944, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 0.9628726243972778, + "step": 7320, + "unclipped_grad_norm": 0.8016625046730042 + }, + { + "bits_per_byte": 1.4291524390294224, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 0.9906129837036133, + "step": 7330, + "unclipped_grad_norm": 0.867045521736145 + }, + { + "bits_per_byte": 1.3836929528574597, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.9591028690338135, + "step": 7340, + "unclipped_grad_norm": 0.8164173364639282 + }, + { + "bits_per_byte": 1.467159572974198, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 1.0169575214385986, + "step": 7350, + "unclipped_grad_norm": 0.8525064587593079 + }, + { + "bits_per_byte": 1.433820048173839, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 0.9938483238220215, + "step": 7360, + "unclipped_grad_norm": 0.8385956883430481 + }, + { + "bits_per_byte": 1.4710990075748616, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 1.0196881294250488, + "step": 7370, + "unclipped_grad_norm": 0.8249295353889465 + }, + { + "bits_per_byte": 1.3941142415864916, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 0.9663263559341431, + "step": 7380, + "unclipped_grad_norm": 0.8101018667221069 + }, + { + "bits_per_byte": 1.4706592479366052, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 1.0193833112716675, + "step": 7390, + "unclipped_grad_norm": 0.8176234364509583 + }, + { + "bits_per_byte": 1.410369611789417, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.977593719959259, + "step": 7400, + "unclipped_grad_norm": 0.8377200961112976 + }, + { + "bits_per_byte": 1.4092478549491645, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 0.9768161773681641, + "step": 7410, + "unclipped_grad_norm": 0.81974858045578 + }, + { + "bits_per_byte": 1.474973260750848, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0223735570907593, + "step": 7420, + "unclipped_grad_norm": 0.8284385204315186 + }, + { + "bits_per_byte": 1.4820893868955984, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 1.027306079864502, + "step": 7430, + "unclipped_grad_norm": 0.845361590385437 + }, + { + "bits_per_byte": 1.502685341232325, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 1.0415821075439453, + "step": 7440, + "unclipped_grad_norm": 0.8964833617210388 + }, + { + "bits_per_byte": 1.4439314241660162, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 1.0008569955825806, + "step": 7450, + "unclipped_grad_norm": 0.8369156718254089 + }, + { + "bits_per_byte": 1.4378882977799987, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 0.9966682195663452, + "step": 7460, + "unclipped_grad_norm": 0.8354071378707886 + }, + { + "bits_per_byte": 1.4212995392083758, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 0.9851697683334351, + "step": 7470, + "unclipped_grad_norm": 0.8366437554359436 + }, + { + "bits_per_byte": 1.4686721603885364, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 1.0180059671401978, + "step": 7480, + "unclipped_grad_norm": 0.8683294653892517 + }, + { + "bits_per_byte": 1.5098209014166228, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 1.0465281009674072, + "step": 7490, + "unclipped_grad_norm": 0.8586563467979431 + }, + { + "bits_per_byte": 1.4743733852646386, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.0219577550888062, + "step": 7500, + "unclipped_grad_norm": 0.8332951068878174 + }, + { + "bits_per_byte": 1.4255870666943855, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 0.988141655921936, + "step": 7510, + "unclipped_grad_norm": 0.8199127912521362 + }, + { + "bits_per_byte": 1.4619254529778392, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 1.0133295059204102, + "step": 7520, + "unclipped_grad_norm": 0.8662075400352478 + }, + { + "bits_per_byte": 1.4951213722647276, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.0363391637802124, + "step": 7530, + "unclipped_grad_norm": 0.868276834487915 + }, + { + "bits_per_byte": 1.4492965949423324, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.0045758485794067, + "step": 7540, + "unclipped_grad_norm": 0.8661652207374573 + }, + { + "bits_per_byte": 1.4698851540250717, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 1.0188467502593994, + "step": 7550, + "unclipped_grad_norm": 0.8560738563537598 + }, + { + "bits_per_byte": 1.4346130601769675, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 0.9943979978561401, + "step": 7560, + "unclipped_grad_norm": 0.8252226710319519 + }, + { + "bits_per_byte": 1.3999351663875943, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 0.9703611135482788, + "step": 7570, + "unclipped_grad_norm": 0.8430092930793762 + }, + { + "bits_per_byte": 1.563467105734957, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 1.0837128162384033, + "step": 7580, + "unclipped_grad_norm": 0.8905554413795471 + }, + { + "bits_per_byte": 1.4270704170580772, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 0.9891698360443115, + "step": 7590, + "unclipped_grad_norm": 0.8126006722450256 + }, + { + "bits_per_byte": 1.4057284019819078, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 0.9743766784667969, + "step": 7600, + "unclipped_grad_norm": 0.882316529750824 + }, + { + "bits_per_byte": 1.4318451713939815, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 0.9924794435501099, + "step": 7610, + "unclipped_grad_norm": 0.8412167429924011 + }, + { + "bits_per_byte": 1.3879486635530598, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 0.9620527029037476, + "step": 7620, + "unclipped_grad_norm": 0.8003552556037903 + }, + { + "bits_per_byte": 1.3963014309488393, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 0.9678424000740051, + "step": 7630, + "unclipped_grad_norm": 0.852692186832428 + }, + { + "bits_per_byte": 1.4094292106544992, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.9769418835639954, + "step": 7640, + "unclipped_grad_norm": 0.8325322270393372 + }, + { + "bits_per_byte": 1.5484170759406035, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 1.073280930519104, + "step": 7650, + "unclipped_grad_norm": 0.8584242463111877 + }, + { + "bits_per_byte": 1.4568894569952746, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 1.0098388195037842, + "step": 7660, + "unclipped_grad_norm": 0.8722478151321411 + }, + { + "bits_per_byte": 1.4809008147954856, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 1.0264822244644165, + "step": 7670, + "unclipped_grad_norm": 0.8899861574172974 + }, + { + "bits_per_byte": 1.4283591690523179, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 0.9900631308555603, + "step": 7680, + "unclipped_grad_norm": 0.8703398108482361 + }, + { + "bits_per_byte": 1.448618811315282, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 1.004106044769287, + "step": 7690, + "unclipped_grad_norm": 0.8683562874794006 + }, + { + "bits_per_byte": 1.3795722485427868, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 0.9562466144561768, + "step": 7700, + "unclipped_grad_norm": 0.8153799176216125 + }, + { + "bits_per_byte": 1.4603757173109144, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 1.012255311012268, + "step": 7710, + "unclipped_grad_norm": 0.8314683437347412 + }, + { + "bits_per_byte": 1.475302951492552, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 1.0226020812988281, + "step": 7720, + "unclipped_grad_norm": 0.8867014646530151 + }, + { + "bits_per_byte": 1.4762574552048406, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 1.023263692855835, + "step": 7730, + "unclipped_grad_norm": 0.8719123005867004 + }, + { + "bits_per_byte": 1.4191212929438728, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 0.9836599230766296, + "step": 7740, + "unclipped_grad_norm": 0.8960307240486145 + }, + { + "bits_per_byte": 1.432920836883803, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9932250380516052, + "step": 7750, + "unclipped_grad_norm": 0.8596091270446777 + }, + { + "bits_per_byte": 1.4627151973106012, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 1.0138769149780273, + "step": 7760, + "unclipped_grad_norm": 0.8388862609863281 + }, + { + "bits_per_byte": 1.3769384202361483, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 0.9544209837913513, + "step": 7770, + "unclipped_grad_norm": 0.8569913506507874 + }, + { + "bits_per_byte": 1.4625328957006867, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 1.0137505531311035, + "step": 7780, + "unclipped_grad_norm": 0.862941563129425 + }, + { + "bits_per_byte": 1.410140186933166, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 0.9774346947669983, + "step": 7790, + "unclipped_grad_norm": 0.8385294079780579 + }, + { + "bits_per_byte": 1.4256966196429852, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 0.9882175922393799, + "step": 7800, + "unclipped_grad_norm": 0.8127948641777039 + }, + { + "bits_per_byte": 1.4127408225881881, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9792373180389404, + "step": 7810, + "unclipped_grad_norm": 0.8709434866905212 + }, + { + "bits_per_byte": 1.3889279327670727, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 0.9627314805984497, + "step": 7820, + "unclipped_grad_norm": 0.861073911190033 + }, + { + "bits_per_byte": 1.430519529121131, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 0.9915605783462524, + "step": 7830, + "unclipped_grad_norm": 0.8894413709640503 + }, + { + "bits_per_byte": 1.4341919606563298, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.994106113910675, + "step": 7840, + "unclipped_grad_norm": 0.8452141880989075 + }, + { + "bits_per_byte": 1.3809828502451635, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.9572243690490723, + "step": 7850, + "unclipped_grad_norm": 0.8403438925743103 + }, + { + "bits_per_byte": 1.4340615118156506, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 0.9940156936645508, + "step": 7860, + "unclipped_grad_norm": 0.8349435329437256 + }, + { + "bits_per_byte": 1.4495812262295102, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0047731399536133, + "step": 7870, + "unclipped_grad_norm": 0.8659698367118835 + }, + { + "bits_per_byte": 1.3787726152076, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9556923508644104, + "step": 7880, + "unclipped_grad_norm": 0.8691674470901489 + }, + { + "bits_per_byte": 1.4674514275327122, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 1.0171598196029663, + "step": 7890, + "unclipped_grad_norm": 0.8606283664703369 + }, + { + "bits_per_byte": 1.4335653418679113, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 0.9936717748641968, + "step": 7900, + "unclipped_grad_norm": 0.864492654800415 + }, + { + "bits_per_byte": 1.4437606453937095, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.0007386207580566, + "step": 7910, + "unclipped_grad_norm": 0.8537375330924988 + }, + { + "bits_per_byte": 1.459385957155199, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 1.0115692615509033, + "step": 7920, + "unclipped_grad_norm": 0.8682068586349487 + }, + { + "bits_per_byte": 1.4180243016053444, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.98289954662323, + "step": 7930, + "unclipped_grad_norm": 0.841934323310852 + }, + { + "bits_per_byte": 1.4232665047862938, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 0.9865331649780273, + "step": 7940, + "unclipped_grad_norm": 0.886671245098114 + }, + { + "bits_per_byte": 1.480160601466172, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 1.02596914768219, + "step": 7950, + "unclipped_grad_norm": 0.906080961227417 + }, + { + "bits_per_byte": 1.4180263653971548, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 0.9829009771347046, + "step": 7960, + "unclipped_grad_norm": 0.846083402633667 + }, + { + "bits_per_byte": 1.4403725011803867, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 0.9983901381492615, + "step": 7970, + "unclipped_grad_norm": 0.8702674508094788 + }, + { + "bits_per_byte": 1.4775808617032298, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 1.0241810083389282, + "step": 7980, + "unclipped_grad_norm": 0.9070017337799072 + }, + { + "bits_per_byte": 1.5088130830825668, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 1.0458295345306396, + "step": 7990, + "unclipped_grad_norm": 0.943093478679657 + }, + { + "bits_per_byte": 1.4082921473583199, + "learning_rate": 3e-05, + "loss_nats": 0.9761537313461304, + "step": 8000, + "unclipped_grad_norm": 0.8546885848045349 + } + ], + "variant": "uniform_groups_6_7_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073002.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073002.json new file mode 100644 index 0000000..8393bca --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073002.json @@ -0,0 +1,14077 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "3a84665628247285ad2538590e9020753e559e267ffc0e2004854b1e04cb1b08", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.00040311424527317286, + 0.00035970969474874437, + 0.00032544726855121553, + 0.00029308689408935606, + 0.00026978139067068696, + 0.00025443214690312743, + 0.00023969457834027708, + 0.0002233884297311306, + 0.0002095247182296589, + 0.0001984660339076072, + 0.0001901928335428238, + 0.00018121300672646612, + 0.0001734284160193056, + 0.0001677417167229578, + 0.00016109299031086266, + 0.00015624657680746168, + 0.0001518603676231578, + 0.00014820833166595548, + 0.00014440594532061368, + 0.0001414585713064298, + 0.00013923499500378966, + 0.00013687640603166074, + 0.00013471563579514623, + 0.00013265138841234148, + 0.00013098257477395236, + 0.0001293915556743741, + 0.00012844328011851758, + 0.0001269879430765286, + 0.0001256384130101651, + 0.00012456141121219844, + 0.0001231025526067242, + 0.00012163266364950687 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00029608183103846386, + "first_to_last_ratio": 2.343484699021421, + "imbalance_abs_log_ratio": 0.8516390092453431, + "last_quartile_mean": 0.0001263425492652459, + "mean": 0.0001858347804954974, + "normalized": [ + 2.16920774571012, + 1.9356424765570717, + 1.751272112160408, + 1.5771369240348272, + 1.451727119925344, + 1.369130935687747, + 1.289826251583108, + 1.2020808437231323, + 1.127478492836466, + 1.067970341065492, + 1.0234512239081746, + 0.9751296621832141, + 0.9332398141881068, + 0.9026389800429313, + 0.8668613586828855, + 0.8407822065969367, + 0.8171794710239251, + 0.7975274126338607, + 0.777066300159633, + 0.7612061150730458, + 0.7492407752334779, + 0.7365489154759011, + 0.7249215428669998, + 0.7138135717041164, + 0.7048334785593375, + 0.6962720074755282, + 0.6911692191098191, + 0.683337870004401, + 0.6760758813563923, + 0.6702804011180051, + 0.662430102042749, + 0.6545204472768429 + ], + "population_cv": 0.39438035124793563 + }, + "activation_output_rms_by_block": [ + 0.0036361459642648697, + 0.005152894649654627, + 0.006386642809957266, + 0.007284422870725393, + 0.0037505170330405235, + 0.0054016136564314365, + 0.006585344672203064, + 0.007922882214188576, + 0.0044892593286931515, + 0.006135277915745974, + 0.007316684816032648, + 0.0082859443500638, + 0.0044554853811860085, + 0.006464940495789051, + 0.007893112488090992, + 0.009196000173687935, + 0.004961193539202213, + 0.006907128728926182, + 0.008217990398406982, + 0.009676996618509293, + 0.005381465889513493, + 0.007502692751586437, + 0.0091153085231781, + 0.010543142445385456, + 0.006136673036962748, + 0.008469858206808567, + 0.010564134456217289, + 0.012301022186875343, + 0.005773921962827444, + 0.008156762458384037, + 0.010457510128617287, + 0.012218860909342766 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.005765057983808219, + "first_to_last_ratio": 0.622587017371887, + "imbalance_abs_log_ratio": 0.4738718734697682, + "last_quartile_mean": 0.009259842918254435, + "mean": 0.007398182220640592, + "normalized": [ + 0.49149180917985336, + 0.6965082091758006, + 0.8632718983507628, + 0.9846233376628902, + 0.5069511565390686, + 0.730127144119427, + 0.8901300989627224, + 1.0709228264321597, + 0.6068057253534959, + 0.8292953231982888, + 0.9889841312125875, + 1.119997332175246, + 0.6022405569783624, + 0.8738552664669654, + 1.0668989020126547, + 1.2430080659586231, + 0.6705962885532487, + 0.9336251153230055, + 1.1108121094231989, + 1.3080235563150786, + 0.7274038039370601, + 1.01412651484229, + 1.232101109614035, + 1.4250990487866828, + 0.8294838994154143, + 1.1448566626512724, + 1.427936504016329, + 1.6627087330393202, + 0.7804514393709395, + 1.1025360304896303, + 1.4135242707919937, + 1.6516031296515918 + ], + "population_cv": 0.30590027610070303 + }, + "bits_per_byte": 8.038491115106986, + "branch_output_rms_by_sublayer": [ + 0.0023513585329055786, + 0.0027412420604377985, + 0.0023208872880786657, + 0.002683233004063368, + 0.002778682392090559, + 0.002711885841563344, + 0.002497476991266012, + 0.0027249811682850122, + 0.002628475893288851, + 0.0027012599166482687, + 0.002920303260907531, + 0.002700963756069541, + 0.002932395786046982, + 0.0026683746837079525, + 0.0032228652853518724, + 0.002738022478297353, + 0.003561305580660701, + 0.0027370047755539417, + 0.003354643704369664, + 0.002654933836311102, + 0.0034162327647209167, + 0.002663377905264497, + 0.0036349669098854065, + 0.002684312639757991, + 0.0035210708156228065, + 0.0027356327045708895, + 0.0038089773152023554, + 0.002652956871315837, + 0.003915763925760984, + 0.002679797587916255, + 0.0037244565319269896, + 0.002698024269193411, + 0.004162859171628952, + 0.0026484597474336624, + 0.004167158622294664, + 0.0026680538430809975, + 0.004440344404429197, + 0.002654345240443945, + 0.0041700867004692554, + 0.002591830212622881, + 0.004737259820103645, + 0.002620960585772991, + 0.004552462603896856, + 0.002651234855875373, + 0.004706811159849167, + 0.0026270023081451654, + 0.005186605267226696, + 0.002615687670186162, + 0.005580419674515724, + 0.00258000404573977, + 0.004905648995190859, + 0.002626793459057808, + 0.005227713380008936, + 0.002573782578110695, + 0.005349088925868273, + 0.002592637902125716, + 0.005235649179667234, + 0.002581534907221794, + 0.004925078712403774, + 0.002551236655563116, + 0.005810525268316269, + 0.002563327783718705, + 0.005564783234149218, + 0.002589041367173195 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.010012756197717748, + 0.008904953530422711, + 0.008980942528705073, + 0.007787912846174446, + 0.007907551845183929, + 0.008289745083748344, + 0.007663833460373458, + 0.007350080301903749, + 0.00709564946584838, + 0.006310668475383808, + 0.006285078227083486, + 0.006362421438950662, + 0.005604026016423943, + 0.005959901601525332, + 0.006167718108848569, + 0.005523196426059091, + 0.0054641843609823, + 0.00557989785603791, + 0.005260407000687673, + 0.004763357766419076, + 0.004970214176820314, + 0.004845025235106639, + 0.004921304265254701, + 0.004820106373346748, + 0.0054804096177630225, + 0.0042302744374162005, + 0.004588429583035324, + 0.004469681656036737, + 0.004349111730454971, + 0.003928502929307311, + 0.004468266803351117, + 0.003904642007649288 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.008362221974278682, + "first_to_last_ratio": 1.8887369414995314, + "imbalance_abs_log_ratio": 0.6359083207487641, + "last_quartile_mean": 0.004427414845626747, + "mean": 0.00600782035481319, + "normalized": [ + 1.6666204390908572, + 1.4822269982304765, + 1.4948753455169401, + 1.2962958920592769, + 1.3162097696294732, + 1.3798257261648947, + 1.2756429133626719, + 1.2234187888150154, + 1.1810688480662828, + 1.0504089840715678, + 1.0461494944748422, + 1.0590232502297414, + 0.9327885465041001, + 0.9920239370590588, + 1.0266149359654666, + 0.9193344840336578, + 0.9095119424808776, + 0.9287724210274617, + 0.875593258455752, + 0.7928595538984271, + 0.8272907449435313, + 0.8064530809788657, + 0.8191497039874, + 0.8023053434820334, + 0.9122126318860999, + 0.7041279844573081, + 0.7637428072161453, + 0.7439772483303089, + 0.7239084182952745, + 0.6538982022256998, + 0.7437417465006834, + 0.6499265585598059 + ], + "population_cv": 0.2648157248292805 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.027978340163826942, + 0.014031016267836094, + 0.0140878576785326, + 0.014151837676763535, + 0.014187770895659924, + 0.014265215955674648, + 0.014337530359625816, + 0.014401593245565891, + 0.014477464370429516, + 0.009722241200506687, + 0.009757626801729202, + 0.009786579757928848, + 0.009825476445257664, + 0.009839320555329323, + 0.00987118761986494, + 0.00996283907443285, + 0.009999874979257584, + 0.007562503684312105, + 0.007595349568873644, + 0.007635751739144325, + 0.007658981718122959, + 0.007660485804080963, + 0.007689184043556452, + 0.007726665120571852, + 0.007755559403449297, + 0.006238248664885759, + 0.00625597033649683, + 0.006290202960371971, + 0.006315375678241253, + 0.006327399518340826, + 0.00635211868211627, + 0.006382675375789404, + 0.006408242508769035, + 0.005390914157032967, + 0.005408597644418478, + 0.0054517327807843685, + 0.005475745070725679, + 0.005509276874363422, + 0.005517482291907072, + 0.0055727302096784115, + 0.005585856735706329, + 0.0048178439028561115, + 0.004830927588045597, + 0.004884926602244377, + 0.004899685736745596, + 0.004938251338899136, + 0.004948681220412254, + 0.004975184798240662, + 0.004989070352166891, + 0.004412420094013214, + 0.004419817589223385, + 0.004455919377505779, + 0.004463871940970421, + 0.004530393984168768, + 0.004542885348200798, + 0.0045819394290447235, + 0.004592607729136944, + 0.004127359017729759, + 0.004137600306421518, + 0.004185030702501535, + 0.004193917382508516, + 0.004221425391733646, + 0.004230021964758635, + 0.004281661007553339 + ], + "loss_nats": 5.571857452392578, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0023513585329055786, + 0.0036361459642648697, + 0.004408807028084993, + 0.005152894649654627, + 0.005783648230135441, + 0.006386642809957266, + 0.006751363631337881, + 0.007284422870725393, + 0.002628475893288851, + 0.0037505170330405235, + 0.004723479505628347, + 0.0054016136564314365, + 0.006024772301316261, + 0.006585344672203064, + 0.007445912808179855, + 0.007922882214188576, + 0.003561305580660701, + 0.0044892593286931515, + 0.005543973762542009, + 0.006135277915745974, + 0.006855612155050039, + 0.007316684816032648, + 0.007876631803810596, + 0.0082859443500638, + 0.0035210708156228065, + 0.0044554853811860085, + 0.0058924350887537, + 0.006464940495789051, + 0.007390674669295549, + 0.007893112488090992, + 0.008790239691734314, + 0.009196000173687935, + 0.004162859171628952, + 0.004961193539202213, + 0.006355749908834696, + 0.006907128728926182, + 0.007784162648022175, + 0.008217990398406982, + 0.00937180407345295, + 0.009676996618509293, + 0.004737259820103645, + 0.005381465889513493, + 0.006999304983764887, + 0.007502692751586437, + 0.008728929795324802, + 0.0091153085231781, + 0.010217384435236454, + 0.010543142445385456, + 0.005580419674515724, + 0.006136673036962748, + 0.008008211851119995, + 0.008469858206808567, + 0.010209310799837112, + 0.010564134456217289, + 0.012040933594107628, + 0.012301022186875343, + 0.005235649179667234, + 0.005773921962827444, + 0.007745372597128153, + 0.008156762458384037, + 0.010180563665926456, + 0.010457510128617287, + 0.011906592175364494, + 0.012218860909342766 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.889882206451148e-05, + 6.542459595948458e-05, + 6.18507110630162e-05, + 5.8447600167710334e-05, + 5.3635932999895886e-05, + 5.006335413781926e-05, + 4.666198947234079e-05, + 4.3237694626441225e-05, + 3.917520007234998e-05, + 3.6482542782323435e-05, + 3.386033858987503e-05, + 3.135838051093742e-05, + 2.8993785235797986e-05, + 2.7288886485621333e-05, + 2.567829142208211e-05, + 2.4245826352853328e-05, + 2.3225769837154076e-05, + 2.2156153136165813e-05, + 2.1295820261002518e-05, + 2.0489669623202644e-05, + 1.989009979297407e-05, + 1.9237015294493176e-05, + 1.8684570022742264e-05, + 1.8197497411165386e-05, + 1.7772616047295742e-05, + 1.7398180716554634e-05, + 1.7072889022529125e-05, + 1.6799443983472884e-05, + 1.6413854609709233e-05, + 1.622365743969567e-05, + 1.605328725418076e-05, + 1.591364707564935e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.602758756140247e-05, + "first_to_last_ratio": 3.353751062355353, + "imbalance_abs_log_ratio": 1.2100794395240582, + "last_quartile_mean": 1.6705947018635925e-05, + "mean": 3.100400385847024e-05, + "normalized": [ + 2.222255627983624, + 2.1101982911026727, + 1.9949265696571863, + 1.8851629755471904, + 1.7299679500988914, + 1.6147383533543858, + 1.5050310819643642, + 1.3945842228576797, + 1.2635529350073726, + 1.1767042395189378, + 1.0921279311034675, + 1.0114300286532305, + 0.935162612162975, + 0.8801729805670262, + 0.8282250105276917, + 0.7820224272817399, + 0.7491216277477283, + 0.7146223190174449, + 0.6868732296065864, + 0.6608717285914316, + 0.6415332640187416, + 0.6204687427568377, + 0.6026502289199551, + 0.586940238242612, + 0.5732361577693551, + 0.5611591585388569, + 0.5506672331891367, + 0.5418475645971546, + 0.5294108039928203, + 0.523276203736615, + 0.5177811010301183, + 0.5132771608561701 + ], + "population_cv": 0.5210982252317796 + }, + "activation_output_rms_by_block": [ + 0.0058697923086583614, + 0.009906654246151447, + 0.014539025723934174, + 0.01922357641160488, + 0.006823165342211723, + 0.012289714068174362, + 0.017840292304754257, + 0.023710304871201515, + 0.008299196138978004, + 0.014792175963521004, + 0.021555693820118904, + 0.028360262513160706, + 0.009239793755114079, + 0.01666518859565258, + 0.02497156709432602, + 0.03287752345204353, + 0.010306233540177345, + 0.019625531509518623, + 0.02919023297727108, + 0.03968764841556549, + 0.01330806314945221, + 0.024521460756659508, + 0.03633224591612816, + 0.05011952668428421, + 0.015319778583943844, + 0.0297367125749588, + 0.04422325640916824, + 0.06067219376564026, + 0.019544215872883797, + 0.03773074597120285, + 0.056277770549058914, + 0.08031155914068222 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.01377531565958634, + "first_to_last_ratio": 0.3205274060435306, + "imbalance_abs_log_ratio": 1.137787496000981, + "last_quartile_mean": 0.042977029108442366, + "mean": 0.026058471950818785, + "normalized": [ + 0.22525466265775904, + 0.3801701905180273, + 0.5579385372777909, + 0.737709273509449, + 0.26184057741717787, + 0.4716206725923623, + 0.6846254200332609, + 0.9098885351355575, + 0.3184836069682603, + 0.5676532373593847, + 0.827204828464303, + 1.0883317550885634, + 0.3545792620746419, + 0.6395305383640864, + 0.9582897700776881, + 1.261682707800159, + 0.39550414005965967, + 0.7531343950849723, + 1.1201820671742762, + 1.5230228576130482, + 0.5107000584903466, + 0.9410168333331232, + 1.3942584962272342, + 1.9233486437300251, + 0.5879001122113947, + 1.141153350475888, + 1.697077882871743, + 2.3283097289875387, + 0.7500138883726717, + 1.447926265301113, + 2.159672702807526, + 3.081975001920968 + ], + "population_cv": 0.65750425588141 + }, + "bits_per_byte": 5.118777423800464, + "branch_output_rms_by_sublayer": [ + 0.0030784201808273792, + 0.00497082807123661, + 0.0026541082188487053, + 0.004923256114125252, + 0.003272744594141841, + 0.005568843334913254, + 0.0027562633622437716, + 0.005842715967446566, + 0.0031189140863716602, + 0.006161381956189871, + 0.003062804462388158, + 0.006163209211081266, + 0.003518019337207079, + 0.0063879219815135, + 0.0038524146657437086, + 0.0064848605543375015, + 0.004727440420538187, + 0.006554283667355776, + 0.004043832886964083, + 0.007049264386296272, + 0.004852323792874813, + 0.006710963789373636, + 0.005033119581639767, + 0.00674351816996932, + 0.005156268365681171, + 0.0069910138845443726, + 0.005080085713416338, + 0.006684042979031801, + 0.006521036848425865, + 0.006940248887985945, + 0.005779625847935677, + 0.007169182877987623, + 0.006071291398257017, + 0.007245705928653479, + 0.007010761648416519, + 0.00755178602412343, + 0.007437867112457752, + 0.00728053692728281, + 0.007951895706355572, + 0.007710824254900217, + 0.008829177357256413, + 0.007786651607602835, + 0.008747570216655731, + 0.007822283543646336, + 0.009046150371432304, + 0.008046568371355534, + 0.011369734071195126, + 0.008091960102319717, + 0.010478147305548191, + 0.007819924503564835, + 0.01127540785819292, + 0.008290551602840424, + 0.011129395104944706, + 0.008259772323071957, + 0.012989840470254421, + 0.008409800939261913, + 0.01352131087332964, + 0.009147482924163342, + 0.013932913541793823, + 0.009211433120071888, + 0.01540595106780529, + 0.00820086244493723, + 0.02047763392329216, + 0.008961410261690617 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0002864165859932561, + 0.00022453663048096247, + 0.0002293037593064239, + 0.00019940245562751334, + 0.00020718243170007479, + 0.00019011424481742655, + 0.000208948772157069, + 0.00020324074053499426, + 0.00021001675634827979, + 0.000180301591055868, + 0.00018975257899460683, + 0.0001892256815985771, + 0.0001870888624365184, + 0.00017262266362295088, + 0.00020722116406264032, + 0.0001794750563662315, + 0.00017759466304144414, + 0.0001844276499031075, + 0.0001747213727116689, + 0.00017297250488464418, + 0.00017892531813943932, + 0.00016759173350147172, + 0.00016177406307711834, + 0.00016683866885964005, + 0.00016128330533670126, + 0.0001540861079199751, + 0.0001445186136385793, + 0.00015759855892937027, + 0.00015013910107872255, + 0.0001465190477044789, + 0.0001467184631203816, + 0.00017172502607979377 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00021864320257721504, + "first_to_last_ratio": 1.4190835080460582, + "imbalance_abs_log_ratio": 0.3500112463730666, + "last_quartile_mean": 0.00015407352797600034, + "mean": 0.00018382138040718533, + "normalized": [ + 1.5581244431894195, + 1.2214935497904988, + 1.2474270337786058, + 1.0847620401164118, + 1.127085604738372, + 1.0342335826023166, + 1.1366946091593026, + 1.1056425541185297, + 1.1425045110806409, + 0.980852122078941, + 1.0322660975251259, + 1.0293997421813537, + 1.0177753100433433, + 0.9390782684830893, + 1.1272963112540109, + 0.9763557208017575, + 0.966126261526552, + 1.0032981446150562, + 0.95049537939842, + 0.9409814271957394, + 0.9733651098860172, + 0.9117096886675364, + 0.8800611915696115, + 0.9076129691229246, + 0.8773914382507646, + 0.8382382265798287, + 0.7861904492200749, + 0.8573461834541307, + 0.816766258343547, + 0.797072937761225, + 0.7981577703060626, + 0.9341950631607883 + ], + "population_cv": 0.1566357587601453 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6923249959945679, + "mean_weights": [ + 0.48043298721313477, + 0.5195670127868652 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927929520606995, + "mean_weights": [ + 0.512978732585907, + 0.487021267414093 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928666830062866, + "mean_weights": [ + 0.5081413984298706, + 0.4918586015701294 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6926385164260864, + "mean_weights": [ + 0.5151234865188599, + 0.48487651348114014 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6925259232521057, + "mean_weights": [ + 0.5162523984909058, + 0.483747661113739 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927998065948486, + "mean_weights": [ + 0.5108168125152588, + 0.4891831874847412 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6924365758895874, + "mean_weights": [ + 0.5170767307281494, + 0.4829232692718506 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929617524147034, + "mean_weights": [ + 0.5073461532592773, + 0.49265387654304504 + ], + "sources": 2 + }, + { + "entropy_mean": 1.097869873046875, + "mean_weights": [ + 0.34936052560806274, + 0.32311511039733887, + 0.3275243639945984 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0983173847198486, + "mean_weights": [ + 0.34335026144981384, + 0.32689353823661804, + 0.3297562003135681 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0981323719024658, + "mean_weights": [ + 0.3444468677043915, + 0.33024442195892334, + 0.3253086805343628 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984491109848022, + "mean_weights": [ + 0.3401055932044983, + 0.32976871728897095, + 0.33012568950653076 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982551574707031, + "mean_weights": [ + 0.34354785084724426, + 0.3309495449066162, + 0.3255026340484619 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098495602607727, + "mean_weights": [ + 0.33823567628860474, + 0.33379966020584106, + 0.3279646039009094 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0980799198150635, + "mean_weights": [ + 0.34470295906066895, + 0.33187657594680786, + 0.3234204649925232 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985522270202637, + "mean_weights": [ + 0.3359280824661255, + 0.3344811797142029, + 0.32959073781967163 + ], + "sources": 3 + }, + { + "entropy_mean": 1.386026382446289, + "mean_weights": [ + 0.2575710117816925, + 0.25011467933654785, + 0.24575293064117432, + 0.24656136333942413 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862006664276123, + "mean_weights": [ + 0.2531803846359253, + 0.2523835301399231, + 0.2467462718486786, + 0.24768981337547302 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3858786821365356, + "mean_weights": [ + 0.2582573890686035, + 0.25250816345214844, + 0.2448825091123581, + 0.24435192346572876 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861615657806396, + "mean_weights": [ + 0.2538246512413025, + 0.25333255529403687, + 0.2476385533809662, + 0.24520421028137207 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3855650424957275, + "mean_weights": [ + 0.2593255341053009, + 0.25452280044555664, + 0.24627012014389038, + 0.23988154530525208 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861019611358643, + "mean_weights": [ + 0.25143834948539734, + 0.25587013363838196, + 0.24893389642238617, + 0.24375763535499573 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3856532573699951, + "mean_weights": [ + 0.2575244903564453, + 0.2522701025009155, + 0.24599556624889374, + 0.244209885597229 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862508535385132, + "mean_weights": [ + 0.2500683665275574, + 0.25261130928993225, + 0.25026029348373413, + 0.24706009030342102 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6089305877685547, + "mean_weights": [ + 0.20503470301628113, + 0.20676736533641815, + 0.19997166097164154, + 0.1957210749387741, + 0.19250519573688507 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093056201934814, + "mean_weights": [ + 0.2015950083732605, + 0.20415446162223816, + 0.20093141496181488, + 0.19674775004386902, + 0.19657135009765625 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085529327392578, + "mean_weights": [ + 0.20560836791992188, + 0.20931032299995422, + 0.2015652358531952, + 0.1941160261631012, + 0.18940003216266632 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093418598175049, + "mean_weights": [ + 0.2010945975780487, + 0.20402711629867554, + 0.20015233755111694, + 0.19849012792110443, + 0.19623583555221558 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6080262660980225, + "mean_weights": [ + 0.20550988614559174, + 0.2128736823797226, + 0.2033504843711853, + 0.19303101301193237, + 0.1852349489927292 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093814373016357, + "mean_weights": [ + 0.201116681098938, + 0.20250454545021057, + 0.19945028424263, + 0.19840437173843384, + 0.19852414727210999 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6076775789260864, + "mean_weights": [ + 0.20595376193523407, + 0.21550177037715912, + 0.20281332731246948, + 0.192641019821167, + 0.18309010565280914 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609355092048645, + "mean_weights": [ + 0.2006453275680542, + 0.20332372188568115, + 0.20115655660629272, + 0.19840209186077118, + 0.19647231698036194 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7902021408081055, + "mean_weights": [ + 0.1712493896484375, + 0.18097557127475739, + 0.17131495475769043, + 0.16204427182674408, + 0.15440428256988525, + 0.16001154482364655 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917003631591797, + "mean_weights": [ + 0.166920006275177, + 0.169246643781662, + 0.1683211624622345, + 0.16566120088100433, + 0.16516101360321045, + 0.16468997299671173 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7897300720214844, + "mean_weights": [ + 0.17206445336341858, + 0.1824418306350708, + 0.1732848882675171, + 0.16243736445903778, + 0.15677839517593384, + 0.1529930830001831 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917163372039795, + "mean_weights": [ + 0.1664613038301468, + 0.16857843101024628, + 0.16665872931480408, + 0.16678547859191895, + 0.16457100212574005, + 0.16694504022598267 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7894790172576904, + "mean_weights": [ + 0.17305465042591095, + 0.1832677721977234, + 0.17241500318050385, + 0.16389404237270355, + 0.1557493507862091, + 0.15161918103694916 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917404174804688, + "mean_weights": [ + 0.1662193089723587, + 0.16747929155826569, + 0.16679547727108002, + 0.16684876382350922, + 0.1652650386095047, + 0.16739211976528168 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7884407043457031, + "mean_weights": [ + 0.1728946715593338, + 0.18641221523284912, + 0.17566218972206116, + 0.162957563996315, + 0.15429502725601196, + 0.14777831733226776 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0792863368988037, + "mean_weights": [ + 0.1242552250623703, + 0.12249711155891418, + 0.1226290911436081, + 0.12312449514865875, + 0.12551335990428925, + 0.12617439031600952, + 0.12737232446670532, + 0.1284339725971222 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1906251907348633, + "mean_weights": [ + 0.1189710721373558, + 0.1320197880268097, + 0.12536749243736267, + 0.11699433624744415, + 0.1089882180094719, + 0.10414649546146393, + 0.09839121252298355, + 0.0947696715593338, + 0.10035170614719391 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197190284729004, + "mean_weights": [ + 0.11040810495615005, + 0.10999961197376251, + 0.11076272279024124, + 0.11037062108516693, + 0.1112704798579216, + 0.11132501065731049, + 0.11100593954324722, + 0.11242730170488358, + 0.11243019998073578 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1903748512268066, + "mean_weights": [ + 0.11926820874214172, + 0.1319776475429535, + 0.12610547244548798, + 0.1162252277135849, + 0.11094896495342255, + 0.10520607978105545, + 0.10005321353673935, + 0.09510394930839539, + 0.09511125087738037 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1971068382263184, + "mean_weights": [ + 0.11035673320293427, + 0.10916527360677719, + 0.10961130261421204, + 0.10942646861076355, + 0.11103048175573349, + 0.11168020963668823, + 0.11179907619953156, + 0.11320506036281586, + 0.11372540891170502 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1895041465759277, + "mean_weights": [ + 0.11986196041107178, + 0.13382798433303833, + 0.12647810578346252, + 0.11638365685939789, + 0.1101519837975502, + 0.10433225333690643, + 0.10013146698474884, + 0.09432508051395416, + 0.09450753033161163 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1969637870788574, + "mean_weights": [ + 0.10975326597690582, + 0.10749907791614532, + 0.10838121175765991, + 0.10997828841209412, + 0.11044105142354965, + 0.11142522096633911, + 0.11395899951457977, + 0.114388607442379, + 0.11417427659034729 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1906301975250244, + "mean_weights": [ + 0.11944454163312912, + 0.13162696361541748, + 0.12473027408123016, + 0.11637496948242188, + 0.11088147759437561, + 0.1052699089050293, + 0.10123603045940399, + 0.09599653631448746, + 0.09443928301334381 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028569476678967476, + 0.013716368936002254, + 0.015035169199109077, + 0.014954926446080208, + 0.015682779252529144, + 0.015792855992913246, + 0.016550997272133827, + 0.0167688000947237, + 0.017768075689673424, + 0.01213071122765541, + 0.012958874925971031, + 0.013117672875523567, + 0.014059046283364296, + 0.01421078946441412, + 0.015295291319489479, + 0.015549414791166782, + 0.01675753854215145, + 0.012827446684241295, + 0.013785677962005138, + 0.013961246237158775, + 0.015057040378451347, + 0.015238803811371326, + 0.01630363240838051, + 0.01647798717021942, + 0.01765827089548111, + 0.014391484670341015, + 0.015293174423277378, + 0.01552277896553278, + 0.016430003568530083, + 0.01674848049879074, + 0.01777404174208641, + 0.017890140414237976, + 0.01905418373644352, + 0.01608569547533989, + 0.01705252006649971, + 0.017328448593616486, + 0.018349383026361465, + 0.01856718212366104, + 0.019673768430948257, + 0.019752446562051773, + 0.021168433129787445, + 0.01855558343231678, + 0.019336773082613945, + 0.019871903583407402, + 0.020676400512456894, + 0.021252725273370743, + 0.022059855982661247, + 0.022855134680867195, + 0.023706713691353798, + 0.02137739397585392, + 0.02210981771349907, + 0.022864198312163353, + 0.023635847494006157, + 0.02437782660126686, + 0.02513287402689457, + 0.026087002828717232, + 0.02712021954357624, + 0.023500699549913406, + 0.02555587701499462, + 0.025034543126821518, + 0.02737937495112419, + 0.026489073410630226, + 0.029305877164006233, + 0.02842867560684681 + ], + "loss_nats": 3.5480661392211914, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1936891078948975, + "mean_weights": [ + 0.10076597332954407, + 0.09523037075996399, + 0.10244341194629669, + 0.11051754653453827, + 0.1152997761964798, + 0.11859878152608871, + 0.11965619027614594, + 0.1194571927189827, + 0.11803075671195984 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.0030784201808273792, + 0.0058697923086583614, + 0.0066830781288445, + 0.009906654246151447, + 0.010638412088155746, + 0.014539025723934174, + 0.014878508634865284, + 0.01922357641160488, + 0.0031189140863716602, + 0.006823165342211723, + 0.0077349781058728695, + 0.012289714068174362, + 0.012974022887647152, + 0.017840292304754257, + 0.018578486517071724, + 0.023710304871201515, + 0.004727440420538187, + 0.008299196138978004, + 0.009581772610545158, + 0.014792175963521004, + 0.016329027712345123, + 0.021555693820118904, + 0.0230940543115139, + 0.028360262513160706, + 0.005156268365681171, + 0.009239793755114079, + 0.011861526407301426, + 0.01666518859565258, + 0.01973063312470913, + 0.02497156709432602, + 0.02728474885225296, + 0.03287752345204353, + 0.006071291398257017, + 0.010306233540177345, + 0.01407517958432436, + 0.019625531509518623, + 0.0237212385982275, + 0.02919023297727108, + 0.033749211579561234, + 0.03968764841556549, + 0.008829177357256413, + 0.01330806314945221, + 0.0188510213047266, + 0.024521460756659508, + 0.030244281515479088, + 0.03633224591612816, + 0.043665602803230286, + 0.05011952668428421, + 0.010478147305548191, + 0.015319778583943844, + 0.02365351840853691, + 0.0297367125749588, + 0.037896472960710526, + 0.04422325640916824, + 0.05400286614894867, + 0.06067219376564026, + 0.01352131087332964, + 0.019544215872883797, + 0.03060680814087391, + 0.03773074597120285, + 0.050156645476818085, + 0.056277770549058914, + 0.07311917841434479, + 0.08031155914068222 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00014324201038107276, + 0.00015136586443986744, + 0.00014967737661208957, + 0.00014159611600916833, + 0.0001353175175609067, + 0.0001222255959874019, + 0.00011082481069024652, + 0.00010351398668717593, + 9.393482469022274e-05, + 8.556910324841738e-05, + 7.825411012163386e-05, + 7.310608634725213e-05, + 7.381839532172307e-05, + 6.744825805071741e-05, + 6.219510396476835e-05, + 5.7578872656449676e-05, + 5.545985550270416e-05, + 5.114106897963211e-05, + 4.689516936196014e-05, + 4.330167575972155e-05, + 3.944907439290546e-05, + 3.6895027733407915e-05, + 3.409233977436088e-05, + 3.217554694856517e-05, + 2.9740363970631734e-05, + 2.7942895030719228e-05, + 2.6785535737872124e-05, + 2.60814449575264e-05, + 2.657388904481195e-05, + 2.62554003711557e-05, + 2.6092206098837778e-05, + 2.6060875825351104e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00013222040979599115, + "first_to_last_ratio": 4.907671619988896, + "imbalance_abs_log_ratio": 1.5907996175175587, + "last_quartile_mean": 2.6941576379613252e-05, + "mean": 6.889407507060241e-05, + "normalized": [ + 2.079162979316855, + 2.1970810158166465, + 2.172572553716712, + 2.055272762774754, + 1.9641386784311101, + 1.7741089616508463, + 1.6086261492976521, + 1.5025092735637302, + 1.3634673895245513, + 1.2420386391823455, + 1.1358612484664221, + 1.0611374965457228, + 1.0714766871617658, + 0.9790139134838035, + 0.9027641912752442, + 0.8357594262996207, + 0.8050018154082058, + 0.7423144722855037, + 0.6806850852399445, + 0.6285253951859596, + 0.5726047465254187, + 0.5355326665696291, + 0.49485154912702095, + 0.4670292316950591, + 0.4316824623910534, + 0.4055921354932691, + 0.3887930233539299, + 0.3785731201238746, + 0.38572096392293703, + 0.381098089266011, + 0.37872931848055985, + 0.37827455842384133 + ], + "population_cv": 0.6057974711252765 + }, + "activation_output_rms_by_block": [ + 0.021282995119690895, + 0.0385221391916275, + 0.0492069274187088, + 0.058616481721401215, + 0.020450301468372345, + 0.026761407032608986, + 0.03491077944636345, + 0.04084091633558273, + 0.03162103518843651, + 0.03245885297656059, + 0.037762295454740524, + 0.04262661561369896, + 0.021038135513663292, + 0.032374389469623566, + 0.03912488743662834, + 0.04699741303920746, + 0.021943897008895874, + 0.03520995378494263, + 0.04641484469175339, + 0.05725608393549919, + 0.021381167694926262, + 0.03468061611056328, + 0.05172840878367424, + 0.06649065017700195, + 0.022232238203287125, + 0.04204196855425835, + 0.06422603875398636, + 0.08832460641860962, + 0.030814431607723236, + 0.05930396541953087, + 0.09623674303293228, + 0.13132493197917938 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.03632399346679449, + "first_to_last_ratio": 0.5436656141093544, + "imbalance_abs_log_ratio": 0.6094209010654029, + "last_quartile_mean": 0.0668131154961884, + "mean": 0.045131441205739975, + "normalized": [ + 0.4715780075063069, + 0.8535543772248984, + 1.0903025940250828, + 1.2987948125606537, + 0.4531275962393021, + 0.59296593057182, + 0.7735356663487931, + 0.9049326864923702, + 0.7006431512852914, + 0.7192071006239518, + 0.8367181380845817, + 0.9444993218669373, + 0.46615253028940695, + 0.7173356002977825, + 0.866909772685307, + 1.0413452746824794, + 0.48622194245604927, + 0.7801646223623034, + 1.0284370153428688, + 1.2686517958619314, + 0.47375326654108557, + 0.7684358217692475, + 1.1461723224804805, + 1.4732667160769828, + 0.49261086305525625, + 0.9315450034622714, + 1.4230885838810277, + 1.9570526457589879, + 0.682770830810609, + 1.3140277339955273, + 2.1323658288291614, + 2.9098324465312446 + ], + "population_cv": 0.5289411938542868 + }, + "bits_per_byte": 3.0655088878877534, + "branch_output_rms_by_sublayer": [ + 0.011178011074662209, + 0.01691485568881035, + 0.015253882855176926, + 0.01378655806183815, + 0.013331761583685875, + 0.01176875177770853, + 0.011751476675271988, + 0.010544294491410255, + 0.018059242516756058, + 0.00856309849768877, + 0.012096631340682507, + 0.008312363177537918, + 0.013841930776834488, + 0.00866015162318945, + 0.018816329538822174, + 0.008847259916365147, + 0.02903188206255436, + 0.008799311704933643, + 0.016501834616065025, + 0.008168498985469341, + 0.015075752511620522, + 0.009078224189579487, + 0.015267335809767246, + 0.008866427466273308, + 0.018013082444667816, + 0.00902662705630064, + 0.01619129814207554, + 0.009386355988681316, + 0.014995395205914974, + 0.009684286080300808, + 0.017178796231746674, + 0.009723695926368237, + 0.017652353271842003, + 0.011091473512351513, + 0.021533234044909477, + 0.011643776670098305, + 0.024808693677186966, + 0.012764958664774895, + 0.017812008038163185, + 0.013894377276301384, + 0.015229975804686546, + 0.014356160536408424, + 0.014934040606021881, + 0.015296010300517082, + 0.020059101283550262, + 0.01718618907034397, + 0.0139989722520113, + 0.018162544816732407, + 0.013308282941579819, + 0.016624029725790024, + 0.013448691926896572, + 0.01914389245212078, + 0.010757665149867535, + 0.021523259580135345, + 0.013923868536949158, + 0.023335089907050133, + 0.008742057718336582, + 0.028821779415011406, + 0.005247499793767929, + 0.02934672124683857, + 0.015282699838280678, + 0.03330143541097641, + 0.007872065529227257, + 0.032740842550992966 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0005178265426944053, + 0.000461280552838665, + 0.00040118542797703627, + 0.00041285388531506835, + 0.0004265148935797383, + 0.00030251852161156746, + 0.000235491658286173, + 0.00023104668417863852, + 0.0002569051818366097, + 0.00019042642143150365, + 0.0001863479501845335, + 0.00017507626807504475, + 0.00019959213219311256, + 0.00016960233718525322, + 0.00015977517897883075, + 0.00015048516913732213, + 0.00015520847558480954, + 0.0001398656284931499, + 0.0001415997381320876, + 0.00013055702859682934, + 0.00011587537524521379, + 0.00010522434164938231, + 0.00011182251779680717, + 0.00010135927201532077, + 9.97362365737907e-05, + 9.13636814501056e-05, + 9.27924414531388e-05, + 8.988624378643033e-05, + 9.522083756420456e-05, + 9.421272333322292e-05, + 0.00010810603577593033, + 0.00010344073371366734 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00037358977081016154, + "first_to_last_ratio": 3.8576104600680394, + "imbalance_abs_log_ratio": 1.3500479399892162, + "last_quartile_mean": 9.684486670631132e-05, + "mean": 0.0001954125036458623, + "normalized": [ + 2.6499150926024684, + 2.360547786003622, + 2.0530182075968253, + 2.112730135545808, + 2.1826387033692223, + 1.548102173440287, + 1.2051002565984439, + 1.1823536358622913, + 1.314681389591057, + 0.9744843235651148, + 0.9536132371664604, + 0.8959317587595519, + 1.0213887467243388, + 0.8679195753646157, + 0.8176302744085632, + 0.7700897656479544, + 0.7942607186799429, + 0.715745542806321, + 0.7246196404540354, + 0.6681099016746249, + 0.5929783052941677, + 0.5384729210576815, + 0.5722382944310377, + 0.5186938917634966, + 0.5103882029705653, + 0.46754265845587895, + 0.47485416604304176, + 0.45998204879114246, + 0.48728119126281294, + 0.48212228785503203, + 0.5532196443879879, + 0.5293455218256071 + ], + "population_cv": 0.613815704950741 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6528184413909912, + "mean_weights": [ + 0.38624268770217896, + 0.613757312297821 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6734062433242798, + "mean_weights": [ + 0.4389711618423462, + 0.5610288381576538 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6905777454376221, + "mean_weights": [ + 0.4995812177658081, + 0.5004187822341919 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6690053343772888, + "mean_weights": [ + 0.4342353343963623, + 0.5657646656036377 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6898853182792664, + "mean_weights": [ + 0.49623239040374756, + 0.5037676095962524 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6792309284210205, + "mean_weights": [ + 0.4671386480331421, + 0.5328614115715027 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6784142255783081, + "mean_weights": [ + 0.4441643953323364, + 0.5558356046676636 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6703981161117554, + "mean_weights": [ + 0.43978798389434814, + 0.5602120161056519 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0913604497909546, + "mean_weights": [ + 0.3023092448711395, + 0.34480464458465576, + 0.3528861105442047 + ], + "sources": 3 + }, + { + "entropy_mean": 1.089369535446167, + "mean_weights": [ + 0.3087027072906494, + 0.33108434081077576, + 0.3602129817008972 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0881578922271729, + "mean_weights": [ + 0.28794729709625244, + 0.34241706132888794, + 0.3696356415748596 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0682929754257202, + "mean_weights": [ + 0.2929821014404297, + 0.29851263761520386, + 0.40850529074668884 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0881413221359253, + "mean_weights": [ + 0.2947655916213989, + 0.3256474435329437, + 0.37958699464797974 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0634163618087769, + "mean_weights": [ + 0.30100953578948975, + 0.2872318625450134, + 0.4117586016654968 + ], + "sources": 3 + }, + { + "entropy_mean": 1.089205265045166, + "mean_weights": [ + 0.30032017827033997, + 0.3300884962081909, + 0.3695913553237915 + ], + "sources": 3 + }, + { + "entropy_mean": 1.064763069152832, + "mean_weights": [ + 0.30071765184402466, + 0.2947157025337219, + 0.4045666456222534 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3814082145690918, + "mean_weights": [ + 0.23981568217277527, + 0.23160725831985474, + 0.25908163189888, + 0.26949542760849 + ], + "sources": 4 + }, + { + "entropy_mean": 1.353334903717041, + "mean_weights": [ + 0.24079793691635132, + 0.22324138879776, + 0.2532752752304077, + 0.28268539905548096 + ], + "sources": 4 + }, + { + "entropy_mean": 1.382047414779663, + "mean_weights": [ + 0.24418292939662933, + 0.2350480556488037, + 0.24408933520317078, + 0.2766796946525574 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3482730388641357, + "mean_weights": [ + 0.24944069981575012, + 0.21831944584846497, + 0.22942021489143372, + 0.3028196692466736 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3784008026123047, + "mean_weights": [ + 0.24724817276000977, + 0.2337314486503601, + 0.23788070678710938, + 0.28113967180252075 + ], + "sources": 4 + }, + { + "entropy_mean": 1.359812617301941, + "mean_weights": [ + 0.24445325136184692, + 0.22314491868019104, + 0.23145301640033722, + 0.300948828458786 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3807117938995361, + "mean_weights": [ + 0.26491183042526245, + 0.22713643312454224, + 0.23517248034477234, + 0.272779256105423 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3521664142608643, + "mean_weights": [ + 0.2515465021133423, + 0.2181379199028015, + 0.22067484259605408, + 0.3096407353878021 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6067180633544922, + "mean_weights": [ + 0.20082926750183105, + 0.19309574365615845, + 0.18871736526489258, + 0.20190724730491638, + 0.21545037627220154 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5801301002502441, + "mean_weights": [ + 0.19789861142635345, + 0.17831012606620789, + 0.1730201542377472, + 0.2085207849740982, + 0.24225032329559326 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6038086414337158, + "mean_weights": [ + 0.18802708387374878, + 0.1982581466436386, + 0.19408312439918518, + 0.19562922418117523, + 0.2240024358034134 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5824180841445923, + "mean_weights": [ + 0.19356803596019745, + 0.17884904146194458, + 0.17995178699493408, + 0.1943705528974533, + 0.2532605826854706 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6023390293121338, + "mean_weights": [ + 0.1880589872598648, + 0.19926156103610992, + 0.19290593266487122, + 0.19386804103851318, + 0.22590547800064087 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5778322219848633, + "mean_weights": [ + 0.1966763734817505, + 0.17573349177837372, + 0.17385661602020264, + 0.19225390255451202, + 0.26147961616516113 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5985815525054932, + "mean_weights": [ + 0.18121682107448578, + 0.19927914440631866, + 0.1950448453426361, + 0.18987047672271729, + 0.23458871245384216 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5775493383407593, + "mean_weights": [ + 0.19788184762001038, + 0.1786750853061676, + 0.17467278242111206, + 0.1828126311302185, + 0.26595765352249146 + ], + "sources": 5 + }, + { + "entropy_mean": 1.770434021949768, + "mean_weights": [ + 0.1411547064781189, + 0.1659526228904724, + 0.15802936255931854, + 0.14489328861236572, + 0.1748214066028595, + 0.21514862775802612 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7714686393737793, + "mean_weights": [ + 0.15612822771072388, + 0.15411731600761414, + 0.14935821294784546, + 0.1460190862417221, + 0.1866903305053711, + 0.20768682658672333 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7776567935943604, + "mean_weights": [ + 0.13932284712791443, + 0.1750364899635315, + 0.1672687530517578, + 0.14928272366523743, + 0.17054149508476257, + 0.19854767620563507 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7821569442749023, + "mean_weights": [ + 0.1655939370393753, + 0.15893717110157013, + 0.15130957961082458, + 0.15201205015182495, + 0.17560634016990662, + 0.1965409219264984 + ], + "sources": 6 + }, + { + "entropy_mean": 1.777613639831543, + "mean_weights": [ + 0.13705489039421082, + 0.18230172991752625, + 0.17300409078598022, + 0.14922931790351868, + 0.16424784064292908, + 0.19416214525699615 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7825405597686768, + "mean_weights": [ + 0.17173904180526733, + 0.16271236538887024, + 0.15274004638195038, + 0.149128720164299, + 0.16517823934555054, + 0.1985015720129013 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7724939584732056, + "mean_weights": [ + 0.1346086859703064, + 0.19440440833568573, + 0.18090233206748962, + 0.14513063430786133, + 0.16155409812927246, + 0.18339985609054565 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0578832626342773, + "mean_weights": [ + 0.12476368248462677, + 0.1402517557144165, + 0.13457702100276947, + 0.11336540430784225, + 0.11046391725540161, + 0.11703505367040634, + 0.13093619048595428, + 0.12860697507858276 + ], + "sources": 8 + }, + { + "entropy_mean": 2.128608226776123, + "mean_weights": [ + 0.08006880432367325, + 0.16819390654563904, + 0.16026879847049713, + 0.11163920164108276, + 0.10532286763191223, + 0.10054989159107208, + 0.09025676548480988, + 0.07004459947347641, + 0.11365516483783722 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1897337436676025, + "mean_weights": [ + 0.10394354164600372, + 0.12485955655574799, + 0.12057505548000336, + 0.11058592051267624, + 0.1131386086344719, + 0.11536270380020142, + 0.10704357922077179, + 0.10088072717189789, + 0.10361029207706451 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1132915019989014, + "mean_weights": [ + 0.0825263261795044, + 0.1721593588590622, + 0.17114144563674927, + 0.119026780128479, + 0.11452290415763855, + 0.10674363374710083, + 0.09225906431674957, + 0.0698423907160759, + 0.0717780739068985 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1669161319732666, + "mean_weights": [ + 0.10877299308776855, + 0.12829391658306122, + 0.12420225143432617, + 0.10428173094987869, + 0.11510628461837769, + 0.11560319364070892, + 0.10505110025405884, + 0.09527423232793808, + 0.10341431200504303 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1076245307922363, + "mean_weights": [ + 0.08400769531726837, + 0.18135181069374084, + 0.1725059598684311, + 0.11670413613319397, + 0.11179809272289276, + 0.10496973246335983, + 0.09320565313100815, + 0.07170289009809494, + 0.06375404447317123 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1796748638153076, + "mean_weights": [ + 0.10037121176719666, + 0.13004210591316223, + 0.1296113133430481, + 0.1155841201543808, + 0.12148430943489075, + 0.11535383760929108, + 0.10310998558998108, + 0.09307970851659775, + 0.09136340022087097 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1013083457946777, + "mean_weights": [ + 0.08414585143327713, + 0.1813945174217224, + 0.1714932918548584, + 0.11759895086288452, + 0.11762338876724243, + 0.1078210398554802, + 0.09181235730648041, + 0.06972138583660126, + 0.05838923528790474 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.043951187282800674, + 0.01811642199754715, + 0.02411353960633278, + 0.027508266270160675, + 0.030689669772982597, + 0.031747885048389435, + 0.03488561883568764, + 0.03633201867341995, + 0.039259083569049835, + 0.025927696377038956, + 0.027304083108901978, + 0.02746882103383541, + 0.02823389135301113, + 0.028995849192142487, + 0.030411744490265846, + 0.030749645084142685, + 0.032306067645549774, + 0.02369791455566883, + 0.026135947555303574, + 0.024249941110610962, + 0.026129933074116707, + 0.024974599480628967, + 0.026715755462646484, + 0.025422748178243637, + 0.02755962684750557, + 0.02084737829864025, + 0.022076468914747238, + 0.021185552701354027, + 0.022380074486136436, + 0.021657979115843773, + 0.02300109900534153, + 0.022412722930312157, + 0.02373199351131916, + 0.018759438768029213, + 0.01972167193889618, + 0.019840829074382782, + 0.02048543281853199, + 0.020679406821727753, + 0.02126745507121086, + 0.021829258650541306, + 0.022193212062120438, + 0.018861277028918266, + 0.019752370193600655, + 0.01962844282388687, + 0.020671406760811806, + 0.020655309781432152, + 0.0219560656696558, + 0.021845169365406036, + 0.02333778329193592, + 0.020504038780927658, + 0.021631082519888878, + 0.02164324000477791, + 0.023027418181300163, + 0.023166116327047348, + 0.024748992174863815, + 0.024934817105531693, + 0.02760966867208481, + 0.021578378975391388, + 0.025387262925505638, + 0.023904941976070404, + 0.028059357777237892, + 0.02557688020169735, + 0.02944902516901493, + 0.026909923180937767 + ], + "loss_nats": 2.1248488426208496, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1772379875183105, + "mean_weights": [ + 0.12192893028259277, + 0.09367966651916504, + 0.09476842731237411, + 0.10061563551425934, + 0.09909208863973618, + 0.10972880572080612, + 0.11962491273880005, + 0.12675750255584717, + 0.13380400836467743 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.011178011074662209, + 0.021282995119690895, + 0.02975720539689064, + 0.0385221391916275, + 0.041666410863399506, + 0.0492069274187088, + 0.052145831286907196, + 0.058616481721401215, + 0.018059242516756058, + 0.020450301468372345, + 0.023179661482572556, + 0.026761407032608986, + 0.03066849894821644, + 0.03491077944636345, + 0.03610856086015701, + 0.04084091633558273, + 0.02903188206255436, + 0.03162103518843651, + 0.029164357110857964, + 0.03245885297656059, + 0.033432114869356155, + 0.037762295454740524, + 0.038270141929388046, + 0.04262661561369896, + 0.018013082444667816, + 0.021038135513663292, + 0.028284955769777298, + 0.032374389469623566, + 0.03442969173192978, + 0.03912488743662834, + 0.042014069855213165, + 0.04699741303920746, + 0.017652353271842003, + 0.021943897008895874, + 0.02949862740933895, + 0.03520995378494263, + 0.03998821601271629, + 0.04641484469175339, + 0.04944854974746704, + 0.05725608393549919, + 0.015229975804686546, + 0.021381167694926262, + 0.025042233988642693, + 0.03468061611056328, + 0.03923683241009712, + 0.05172840878367424, + 0.05233567953109741, + 0.06649065017700195, + 0.013308282941579819, + 0.022232238203287125, + 0.027949577197432518, + 0.04204196855425835, + 0.04692548140883446, + 0.06422603875398636, + 0.0690988078713417, + 0.08832460641860962, + 0.008742057718336582, + 0.030814431607723236, + 0.03217193856835365, + 0.05930396541953087, + 0.06522670388221741, + 0.09623674303293228, + 0.1006287932395935, + 0.13132493197917938 + ] + }, + { + "activation_grad_rms_by_block": [ + 8.528726175427437e-05, + 9.161033085547388e-05, + 8.541857096133754e-05, + 8.255070861196145e-05, + 9.058073919732124e-05, + 8.587513002566993e-05, + 7.91602287790738e-05, + 7.62160707381554e-05, + 8.354712917935103e-05, + 7.9968391219154e-05, + 7.557502976851538e-05, + 6.984498759265989e-05, + 6.638017657678574e-05, + 6.359681719914079e-05, + 6.12839066889137e-05, + 6.072717587812804e-05, + 6.762889825040475e-05, + 6.852047954453155e-05, + 6.663307431153953e-05, + 5.96626014157664e-05, + 5.906488513574004e-05, + 5.6397810112684965e-05, + 5.159576176083647e-05, + 4.820970934815705e-05, + 5.2661067456938326e-05, + 5.0039932830259204e-05, + 4.779542359756306e-05, + 4.596988219418563e-05, + 4.683280712924898e-05, + 4.135129347559996e-05, + 3.7199395592324436e-05, + 3.6130601074546576e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 8.458738011540845e-05, + "first_to_last_ratio": 1.8903242596226555, + "imbalance_abs_log_ratio": 0.6367483802937116, + "last_quartile_mean": 4.474755041883327e-05, + "mean": 6.47911336955076e-05, + "normalized": [ + 1.3163415561624585, + 1.4139331360677492, + 1.318368210112986, + 1.2741050187502005, + 1.3980422016230565, + 1.3254148388458316, + 1.2217756391035726, + 1.1763348839725576, + 1.2894839835954888, + 1.2342489883719807, + 1.1664409226683357, + 1.0780022451976745, + 1.0245256224215185, + 0.9815666677175365, + 0.9458687198918844, + 0.9372760193319268, + 1.0437986556653494, + 1.0575595090919445, + 1.0284288993103334, + 0.9208451529210271, + 0.9116198739988309, + 0.8704556765086424, + 0.7963398511178381, + 0.7440788052069499, + 0.8127820035442546, + 0.7723268598050292, + 0.7376846316994918, + 0.7095088412903173, + 0.7228274064371899, + 0.6382245705088986, + 0.5741433043469607, + 0.5576473047121839 + ], + "population_cv": 0.24631462776874896 + }, + "activation_output_rms_by_block": [ + 0.07764441519975662, + 0.16625984013080597, + 0.18882200121879578, + 0.19636768102645874, + 0.07038271427154541, + 0.07961133122444153, + 0.0880427286028862, + 0.09483752399682999, + 0.027199499309062958, + 0.04108385741710663, + 0.0596889927983284, + 0.09065155684947968, + 0.04480036348104477, + 0.07587439566850662, + 0.09959447383880615, + 0.12446895241737366, + 0.04226217418909073, + 0.06674416363239288, + 0.09182427823543549, + 0.1260598599910736, + 0.06347424536943436, + 0.08848312497138977, + 0.13219042122364044, + 0.17595286667346954, + 0.05587945505976677, + 0.09246493875980377, + 0.1313299685716629, + 0.17409837245941162, + 0.0647493451833725, + 0.11551958322525024, + 0.18951404094696045, + 0.2736441493034363 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.12024602945894003, + "first_to_last_ratio": 0.8767484178879785, + "imbalance_abs_log_ratio": 0.13153519448246428, + "last_quartile_mean": 0.13714998168870807, + "mean": 0.10654754110146314, + "normalized": [ + 0.7287302400138673, + 1.5604286913808736, + 1.7721854419801606, + 1.8430052819282017, + 0.6605756786496023, + 0.747190694420899, + 0.8263234204442581, + 0.8900958484487039, + 0.25528040373227767, + 0.3855917930380324, + 0.5602099511755766, + 0.8508085302799618, + 0.4204729927871565, + 0.7121177540479599, + 0.9347421142639443, + 1.1682010784166694, + 0.39665086356939283, + 0.626426128115277, + 0.8618150854180016, + 1.1831325123780123, + 0.5957363700114775, + 0.8304567525132187, + 1.240670782798808, + 1.6514024148704949, + 0.5244556043442979, + 0.8678279930622821, + 1.2325950201572455, + 1.6339970933127508, + 0.6077037960145224, + 1.0842069374012415, + 1.7786805682027889, + 2.568282162822043 + ], + "population_cv": 0.520581456286688 + }, + "bits_per_byte": 2.058527421410793, + "branch_output_rms_by_sublayer": [ + 0.044723425060510635, + 0.06015840545296669, + 0.1036699116230011, + 0.022979404777288437, + 0.036396536976099014, + 0.01715695858001709, + 0.02552078478038311, + 0.012017938308417797, + 0.0697498694062233, + 0.00469345785677433, + 0.029063157737255096, + 0.00785134918987751, + 0.029626501724123955, + 0.011395902372896671, + 0.019046317785978317, + 0.013235785067081451, + 0.024095606058835983, + 0.01318417675793171, + 0.02553870901465416, + 0.014292423613369465, + 0.026315176859498024, + 0.01666281931102276, + 0.037741728127002716, + 0.02632926218211651, + 0.03504020348191261, + 0.023363495245575905, + 0.027524499222636223, + 0.025399191305041313, + 0.019405795261263847, + 0.027666591107845306, + 0.02637774497270584, + 0.02816031686961651, + 0.032773036509752274, + 0.022962244227528572, + 0.03643559664487839, + 0.021667590364813805, + 0.04535706713795662, + 0.022618049755692482, + 0.05992609262466431, + 0.033063288778066635, + 0.05625652149319649, + 0.028837205842137337, + 0.04259878396987915, + 0.029408607631921768, + 0.059102658182382584, + 0.04440159723162651, + 0.04854818060994148, + 0.054808661341667175, + 0.027723144739866257, + 0.04646182432770729, + 0.018703022971749306, + 0.049030233174562454, + 0.02063162811100483, + 0.053471341729164124, + 0.020702911540865898, + 0.06089336425065994, + 0.011052361689507961, + 0.06354375183582306, + 0.010437924414873123, + 0.060927074402570724, + 0.023198481649160385, + 0.08526723086833954, + 0.010163932107388973, + 0.09487737715244293 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0002643458994360495, + 0.00022443487727371583, + 0.0001515374763011363, + 0.00012498595878459616, + 0.00019600585113533253, + 0.0001082702478107536, + 0.00012971980880691262, + 0.00014422245328434027, + 0.00013717849585498257, + 0.00014433774557803567, + 0.00016407455024232943, + 0.00022483638324941415, + 0.00015243197900686103, + 0.0001558985946002539, + 0.00015581702553954527, + 0.00015474147432576287, + 0.00016410087715559653, + 0.0001908331289246804, + 0.00023754013567813933, + 0.00028679949004032263, + 0.0002762290487168957, + 0.0002415826624401833, + 0.00028361747160882036, + 0.0002768149282418043, + 0.0002282667694762165, + 0.0002115062191737221, + 0.00020630958846329318, + 0.0002041892501248128, + 0.00022384820025012043, + 0.00020188214420974895, + 0.00020055172152892362, + 0.00018729506672313418 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0001679403216041046, + "first_to_last_ratio": 0.8074786865709453, + "imbalance_abs_log_ratio": 0.21383861857304873, + "last_quartile_mean": 0.00020798111999374648, + "mean": 0.00019544392262457614, + "normalized": [ + 1.3525409021994796, + 1.1483338763356064, + 0.7753501580717925, + 0.6394978012423487, + 1.0028751393402793, + 0.553970910718608, + 0.6637188154276278, + 0.7379224247426407, + 0.7018816139833914, + 0.7385123243524477, + 0.8394968134030781, + 1.1503882046068905, + 0.7799269322877039, + 0.7976640690931895, + 0.7972467163322888, + 0.791743597077727, + 0.8396315165594289, + 0.9764086105212261, + 1.2153876799455405, + 1.4674259817801005, + 1.4133417146333986, + 1.236071499159552, + 1.4511450026185506, + 1.416339400706421, + 1.167939971787657, + 1.0821836582762394, + 1.0555948002516746, + 1.0447459679625617, + 1.1453321098149747, + 1.0329415286938333, + 1.0261343450118883, + 0.958305913061851 + ], + "population_cv": 0.25287108051184815 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.4446738660335541, + "mean_weights": [ + 0.8258414268493652, + 0.17415863275527954 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6425603628158569, + "mean_weights": [ + 0.5444087982177734, + 0.4555911421775818 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5523572564125061, + "mean_weights": [ + 0.7502831816673279, + 0.24971677362918854 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6872317790985107, + "mean_weights": [ + 0.46741122007369995, + 0.5325888395309448 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5679361820220947, + "mean_weights": [ + 0.7347431182861328, + 0.2652568817138672 + ], + "sources": 2 + }, + { + "entropy_mean": 0.689821720123291, + "mean_weights": [ + 0.5243097543716431, + 0.47569021582603455 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6549065709114075, + "mean_weights": [ + 0.5973234176635742, + 0.4026765823364258 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6765522956848145, + "mean_weights": [ + 0.5231114625930786, + 0.4768885374069214 + ], + "sources": 2 + }, + { + "entropy_mean": 0.9529682993888855, + "mean_weights": [ + 0.227272167801857, + 0.20359492301940918, + 0.569132924079895 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0920777320861816, + "mean_weights": [ + 0.3702181875705719, + 0.3004237413406372, + 0.3293580412864685 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9901816844940186, + "mean_weights": [ + 0.23029528558254242, + 0.24198928475379944, + 0.5277154445648193 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0847725868225098, + "mean_weights": [ + 0.3884490132331848, + 0.26475080847740173, + 0.34680017828941345 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0387849807739258, + "mean_weights": [ + 0.3465476930141449, + 0.2144937813282013, + 0.4389585256576538 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0858886241912842, + "mean_weights": [ + 0.35480111837387085, + 0.2698972821235657, + 0.3753015995025635 + ], + "sources": 3 + }, + { + "entropy_mean": 1.037653923034668, + "mean_weights": [ + 0.34405460953712463, + 0.2239522486925125, + 0.43199312686920166 + ], + "sources": 3 + }, + { + "entropy_mean": 1.085527777671814, + "mean_weights": [ + 0.39743858575820923, + 0.28739622235298157, + 0.3151651918888092 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3676875829696655, + "mean_weights": [ + 0.2783840298652649, + 0.1904977262020111, + 0.27583539485931396, + 0.25528281927108765 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3762872219085693, + "mean_weights": [ + 0.27658799290657043, + 0.22197720408439636, + 0.22042176127433777, + 0.2810130715370178 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3612674474716187, + "mean_weights": [ + 0.28275132179260254, + 0.1913897842168808, + 0.23638509213924408, + 0.28947383165359497 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3701072931289673, + "mean_weights": [ + 0.2928614616394043, + 0.22246310114860535, + 0.1957024782896042, + 0.28897297382354736 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3577061891555786, + "mean_weights": [ + 0.2904869318008423, + 0.19605615735054016, + 0.22146062552928925, + 0.2919963002204895 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3764147758483887, + "mean_weights": [ + 0.28333598375320435, + 0.22358763217926025, + 0.21278449892997742, + 0.280291885137558 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3309056758880615, + "mean_weights": [ + 0.35595324635505676, + 0.17674033343791962, + 0.1921970546245575, + 0.27510935068130493 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3756229877471924, + "mean_weights": [ + 0.2812517285346985, + 0.25040507316589355, + 0.20001757144927979, + 0.2683256268501282 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5963032245635986, + "mean_weights": [ + 0.231016606092453, + 0.16559332609176636, + 0.17250967025756836, + 0.22563201189041138, + 0.2052484154701233 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5986617803573608, + "mean_weights": [ + 0.2279680371284485, + 0.19001957774162292, + 0.1610751748085022, + 0.19961179792881012, + 0.22132541239261627 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5874857902526855, + "mean_weights": [ + 0.19270992279052734, + 0.15967589616775513, + 0.17465290427207947, + 0.2361069619655609, + 0.23685429990291595 + ], + "sources": 5 + }, + { + "entropy_mean": 1.592451572418213, + "mean_weights": [ + 0.21386082470417023, + 0.17769242823123932, + 0.1546935737133026, + 0.19596397876739502, + 0.25778916478157043 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5809762477874756, + "mean_weights": [ + 0.1939769834280014, + 0.14925703406333923, + 0.1691964864730835, + 0.22610151767730713, + 0.26146799325942993 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5673577785491943, + "mean_weights": [ + 0.18586690723896027, + 0.1508616805076599, + 0.1523689329624176, + 0.21025262773036957, + 0.30064988136291504 + ], + "sources": 5 + }, + { + "entropy_mean": 1.573469638824463, + "mean_weights": [ + 0.1656060516834259, + 0.15645712614059448, + 0.18155483901500702, + 0.2362135648727417, + 0.2601684033870697 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5833090543746948, + "mean_weights": [ + 0.19776277244091034, + 0.18736881017684937, + 0.14759504795074463, + 0.19266074895858765, + 0.2746126055717468 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7685401439666748, + "mean_weights": [ + 0.1509920060634613, + 0.14269404113292694, + 0.15828225016593933, + 0.1978396773338318, + 0.20908138155937195, + 0.1411106288433075 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7701091766357422, + "mean_weights": [ + 0.1610272228717804, + 0.1619984209537506, + 0.12478937208652496, + 0.13537414371967316, + 0.19122713804244995, + 0.22558367252349854 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7620766162872314, + "mean_weights": [ + 0.13113059103488922, + 0.13635733723640442, + 0.14406415820121765, + 0.17101475596427917, + 0.1874743402004242, + 0.22995883226394653 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7765696048736572, + "mean_weights": [ + 0.16152691841125488, + 0.1631445586681366, + 0.1322881281375885, + 0.14121896028518677, + 0.18837812542915344, + 0.2134433090686798 + ], + "sources": 6 + }, + { + "entropy_mean": 1.731997013092041, + "mean_weights": [ + 0.12811064720153809, + 0.1246149018406868, + 0.13021837174892426, + 0.1700078845024109, + 0.16490857303142548, + 0.2821396291255951 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7829519510269165, + "mean_weights": [ + 0.16977211833000183, + 0.16247490048408508, + 0.13806304335594177, + 0.1484968662261963, + 0.1846892535686493, + 0.19650378823280334 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7176504135131836, + "mean_weights": [ + 0.12628555297851562, + 0.13085421919822693, + 0.13267022371292114, + 0.17179277539253235, + 0.14069028198719025, + 0.2977069318294525 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0449423789978027, + "mean_weights": [ + 0.11371783912181854, + 0.12031818926334381, + 0.12710094451904297, + 0.13793660700321198, + 0.11715660989284515, + 0.09558754414319992, + 0.12475547939538956, + 0.16342678666114807 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1322290897369385, + "mean_weights": [ + 0.08624304831027985, + 0.12605661153793335, + 0.10185308754444122, + 0.11819663643836975, + 0.0929785817861557, + 0.09093897044658661, + 0.14344632625579834, + 0.14773906767368317, + 0.09254765510559082 + ], + "sources": 9 + }, + { + "entropy_mean": 2.168891429901123, + "mean_weights": [ + 0.0928073525428772, + 0.1098485141992569, + 0.11138319969177246, + 0.12975133955478668, + 0.09740372002124786, + 0.09348709136247635, + 0.11735343188047409, + 0.12117107957601547, + 0.12679427862167358 + ], + "sources": 9 + }, + { + "entropy_mean": 2.136042833328247, + "mean_weights": [ + 0.07904765009880066, + 0.11678352952003479, + 0.10424578934907913, + 0.11133725196123123, + 0.0942583978176117, + 0.09768063575029373, + 0.12924277782440186, + 0.13889019191265106, + 0.12851378321647644 + ], + "sources": 9 + }, + { + "entropy_mean": 2.149570941925049, + "mean_weights": [ + 0.11269639432430267, + 0.13354159891605377, + 0.11851184070110321, + 0.10149240493774414, + 0.11150965839624405, + 0.08938908576965332, + 0.0815473422408104, + 0.12378380447626114, + 0.1275278627872467 + ], + "sources": 9 + }, + { + "entropy_mean": 2.132641315460205, + "mean_weights": [ + 0.08616970479488373, + 0.12974806129932404, + 0.09755691140890121, + 0.11541416496038437, + 0.09301938116550446, + 0.08902814239263535, + 0.1216580718755722, + 0.14431646466255188, + 0.12308907508850098 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1749625205993652, + "mean_weights": [ + 0.09120984375476837, + 0.10304136574268341, + 0.12183589488267899, + 0.13506704568862915, + 0.1011541485786438, + 0.10207287967205048, + 0.11853612959384918, + 0.11691644042730331, + 0.1101662665605545 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1387367248535156, + "mean_weights": [ + 0.08380413055419922, + 0.12909701466560364, + 0.11478430777788162, + 0.13366541266441345, + 0.10079227387905121, + 0.09633473306894302, + 0.11540117859840393, + 0.13785114884376526, + 0.08826980739831924 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.0559883676469326, + 0.047728151082992554, + 0.053217265754938126, + 0.06359421461820602, + 0.0976603701710701, + 0.07032905519008636, + 0.10072958469390869, + 0.09164400398731232, + 0.10331781953573227, + 0.07293618470430374, + 0.08130300790071487, + 0.08005860447883606, + 0.07772581279277802, + 0.0740523710846901, + 0.08088268339633942, + 0.07599979639053345, + 0.08118601888418198, + 0.05806852504611015, + 0.06110713630914688, + 0.05658586695790291, + 0.06011219695210457, + 0.0569867268204689, + 0.061063650995492935, + 0.05275420844554901, + 0.06389462202787399, + 0.04438146576285362, + 0.048022426664829254, + 0.043103594332933426, + 0.04510673135519028, + 0.04336969181895256, + 0.04772734269499779, + 0.04871966317296028, + 0.05430826172232628, + 0.04435953497886658, + 0.04472019523382187, + 0.04327674210071564, + 0.04554502293467522, + 0.044258344918489456, + 0.04563351348042488, + 0.047928739339113235, + 0.04629233479499817, + 0.039803389459848404, + 0.040277499705553055, + 0.03980528563261032, + 0.04047258943319321, + 0.039808690547943115, + 0.040992848575115204, + 0.0400518961250782, + 0.04189141094684601, + 0.03651462122797966, + 0.03770526498556137, + 0.03765308856964111, + 0.03934870660305023, + 0.03935003653168678, + 0.04155908152461052, + 0.0415923036634922, + 0.0509808212518692, + 0.046349626034498215, + 0.04445044323801994, + 0.04736790806055069, + 0.051252033561468124, + 0.05253349244594574, + 0.05288374796509743, + 0.0545339360833168 + ], + "loss_nats": 1.4268624782562256, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.9434709548950195, + "mean_weights": [ + 0.11412596702575684, + 0.08500669151544571, + 0.07906557619571686, + 0.07300996035337448, + 0.07137146592140198, + 0.05805925279855728, + 0.05492931604385376, + 0.1692352145910263, + 0.2951965630054474 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.044723425060510635, + 0.07764441519975662, + 0.15646056830883026, + 0.16625984013080597, + 0.18201445043087006, + 0.18882200121879578, + 0.1925463080406189, + 0.19636768102645874, + 0.0697498694062233, + 0.07038271427154541, + 0.07821270078420639, + 0.07961133122444153, + 0.08641614764928818, + 0.0880427286028862, + 0.09255155920982361, + 0.09483752399682999, + 0.024095606058835983, + 0.027199499309062958, + 0.035023972392082214, + 0.04108385741710663, + 0.05166352912783623, + 0.0596889927983284, + 0.07823063433170319, + 0.09065155684947968, + 0.03504020348191261, + 0.04480036348104477, + 0.06263082474470139, + 0.07587439566850662, + 0.08406144380569458, + 0.09959447383880615, + 0.108298160135746, + 0.12446895241737366, + 0.032773036509752274, + 0.04226217418909073, + 0.05910598486661911, + 0.06674416363239288, + 0.08497931808233261, + 0.09182427823543549, + 0.11635421961545944, + 0.1260598599910736, + 0.05625652149319649, + 0.06347424536943436, + 0.07844577729701996, + 0.08848312497138977, + 0.11559294164180756, + 0.13219042122364044, + 0.15366315841674805, + 0.17595286667346954, + 0.027723144739866257, + 0.05587945505976677, + 0.06122737005352974, + 0.09246493875980377, + 0.09674285352230072, + 0.1313299685716629, + 0.13407547771930695, + 0.17409837245941162, + 0.011052361689507961, + 0.0647493451833725, + 0.06607621908187866, + 0.11551958322525024, + 0.1194254606962204, + 0.18951404094696045, + 0.19126173853874207, + 0.2736441493034363 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.78356843511574e-05, + 8.105584856821224e-05, + 7.713741069892421e-05, + 7.573752372991294e-05, + 8.431130845565349e-05, + 8.090827759588137e-05, + 7.636182272108272e-05, + 7.42332631489262e-05, + 8.353397424798459e-05, + 8.029176387935877e-05, + 7.681030547246337e-05, + 6.982892227824777e-05, + 6.758490053471178e-05, + 6.427145126508549e-05, + 6.119196768850088e-05, + 6.042229142622091e-05, + 6.92713656462729e-05, + 6.933583790669218e-05, + 6.665492401225492e-05, + 5.8421293942956254e-05, + 5.637412323267199e-05, + 5.375188629841432e-05, + 4.86714779981412e-05, + 4.51200467068702e-05, + 5.1981431170133874e-05, + 4.941512452205643e-05, + 4.7119952796492726e-05, + 4.542959504760802e-05, + 5.0686921895248815e-05, + 4.230862396070734e-05, + 3.510902388370596e-05, + 3.336306690471247e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 7.844764240871882e-05, + "first_to_last_ratio": 1.7657762441900402, + "imbalance_abs_log_ratio": 0.5685903921102413, + "last_quartile_mean": 4.4426717522583203e-05, + "mean": 6.295410662460199e-05, + "normalized": [ + 1.2363877199512159, + 1.2875386994457996, + 1.225295931191556, + 1.2030593044793567, + 1.339250336096506, + 1.285194595458893, + 1.2129760362803892, + 1.179164746020181, + 1.3269027030452711, + 1.27540152953233, + 1.2200999996789166, + 1.1092036091408715, + 1.0735582499442555, + 1.020925475892127, + 0.9720091503067653, + 0.9597831605572865, + 1.1003470521683516, + 1.101371167414776, + 1.0587859567243336, + 0.9279981414290399, + 0.8954796796471639, + 0.8538265282507954, + 0.7731263392930232, + 0.7167133190519714, + 0.8257035792772242, + 0.7849388573921144, + 0.7484810018426757, + 0.7216303666813448, + 0.8051408337425401, + 0.6720550291182028, + 0.5576923534641284, + 0.5299585474805933 + ], + "population_cv": 0.2323951264097566 + }, + "activation_output_rms_by_block": [ + 0.1121254414319992, + 0.24011382460594177, + 0.2569790184497833, + 0.2627657949924469, + 0.08153681457042694, + 0.09013202041387558, + 0.10180940479040146, + 0.1096077710390091, + 0.02774897962808609, + 0.04423725977540016, + 0.0665174275636673, + 0.11069628596305847, + 0.051317207515239716, + 0.08783987164497375, + 0.11682964116334915, + 0.1450834721326828, + 0.04204847663640976, + 0.07195216417312622, + 0.10633229464292526, + 0.155330628156662, + 0.08552751690149307, + 0.1200723722577095, + 0.1710408478975296, + 0.22501137852668762, + 0.07476522773504257, + 0.11732935905456543, + 0.16285960376262665, + 0.21229317784309387, + 0.07707780599594116, + 0.13612744212150574, + 0.22218629717826843, + 0.3350115120410919 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.15688376128673553, + "first_to_last_ratio": 0.9382646363730996, + "imbalance_abs_log_ratio": 0.06372324143709511, + "last_quartile_mean": 0.16720630321651697, + "mean": 0.1318845731439069, + "normalized": [ + 0.8501785975350783, + 1.8206361736026524, + 1.9485146164091431, + 1.9923922012146784, + 0.6182437613947266, + 0.6834159467273501, + 0.7719584054710579, + 0.831088643850784, + 0.21040352913609972, + 0.33542406606669856, + 0.5043609421329839, + 0.8393421863091701, + 0.3891069765926644, + 0.6660359854910899, + 0.8858476649567616, + 1.100079172826179, + 0.3188278631385358, + 0.5455692235862571, + 0.8062527110498355, + 1.1777770853242384, + 0.6485028147163888, + 0.9104353101760552, + 1.2968980663939902, + 1.7061235682293536, + 0.5668989628791679, + 0.88963671988035, + 1.2348646993376633, + 1.609689236446544, + 0.5844338284496482, + 1.0321710786671707, + 1.6847027054167139, + 2.5401872565909698 + ], + "population_cv": 0.5548257521400705 + }, + "bits_per_byte": 1.9056506671976357, + "branch_output_rms_by_sublayer": [ + 0.06863304227590561, + 0.08467824012041092, + 0.15616057813167572, + 0.01947619952261448, + 0.03639376908540726, + 0.0199362151324749, + 0.020877473056316376, + 0.012658778578042984, + 0.08112318068742752, + 0.003328317776322365, + 0.027429405599832535, + 0.007666433695703745, + 0.03061605431139469, + 0.013459446839988232, + 0.0273919478058815, + 0.015174255706369877, + 0.02391901984810829, + 0.014367136172950268, + 0.023378528654575348, + 0.015914268791675568, + 0.03017618879675865, + 0.0191176887601614, + 0.055119846016168594, + 0.03345009684562683, + 0.037739042192697525, + 0.02850916050374508, + 0.02975611574947834, + 0.032781392335891724, + 0.02270466461777687, + 0.03685897961258888, + 0.026798442006111145, + 0.03802680969238281, + 0.025793075561523438, + 0.0312507338821888, + 0.037740740925073624, + 0.02772175520658493, + 0.05554965138435364, + 0.033233169466257095, + 0.07241867482662201, + 0.05041496083140373, + 0.07373028248548508, + 0.04118127375841141, + 0.05537226423621178, + 0.043196968734264374, + 0.06967988610267639, + 0.06284650415182114, + 0.0638149231672287, + 0.07953153550624847, + 0.03796331584453583, + 0.062112342566251755, + 0.02357284352183342, + 0.06389806419610977, + 0.03144083544611931, + 0.06886715441942215, + 0.026627687737345695, + 0.0773809626698494, + 0.013732252642512321, + 0.0758017897605896, + 0.015502845868468285, + 0.07571707665920258, + 0.026023875921964645, + 0.10603764653205872, + 0.017491539940238, + 0.13137222826480865 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00028490599448325585, + 0.00017565412372546463, + 0.0001239473972547926, + 9.41345354933899e-05, + 0.00016685711667843435, + 9.138449828423743e-05, + 0.00012152938159332018, + 0.0001298326157156257, + 0.00013620946017082233, + 0.00012989560729533152, + 0.00016796296532218283, + 0.0002543360079708181, + 0.00017134049457053318, + 0.00018192461765448767, + 0.0001869018528115125, + 0.00018188908885206068, + 0.000197176345155188, + 0.00022273150740070432, + 0.00029280453626098155, + 0.00033655184359946544, + 0.00032749828134108407, + 0.00029884980050387376, + 0.0003304685613775357, + 0.00033331291329288327, + 0.0002821923420206284, + 0.0002584030770865149, + 0.000254467414782124, + 0.0002511615981803968, + 0.00028004845500698177, + 0.00024469662443981266, + 0.00022619282348909932, + 0.00020880664215528489 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00014853070790356507, + "first_to_last_ratio": 0.5923549550154606, + "imbalance_abs_log_ratio": 0.5236492376033384, + "last_quartile_mean": 0.0002507461221451053, + "mean": 0.0002170021413740259, + "normalized": [ + 1.3129178941704107, + 0.8094580201524665, + 0.5711805260076042, + 0.4337954220052565, + 0.7689192172110351, + 0.4211225645314102, + 0.5600377066503301, + 0.5983010807798693, + 0.6276871707733569, + 0.5985913616927994, + 0.7740152436223249, + 1.1720437704457642, + 0.7895797409446292, + 0.8383540203915387, + 0.8612903615977116, + 0.838190294807072, + 0.9086377853540807, + 1.0264023478773108, + 1.3493163450230765, + 1.5509148502796712, + 1.5091937769250237, + 1.3771744306834965, + 1.5228815682880223, + 1.535989051455412, + 1.3004127066849556, + 1.1907858394868562, + 1.172649325812517, + 1.1574152982550228, + 1.2905331405199774, + 1.1276230865300652, + 1.0423529558596951, + 0.962233095181235 + ], + "population_cv": 0.3335565811978094 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.27759552001953125, + "mean_weights": [ + 0.9133362174034119, + 0.08666379749774933 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5897272229194641, + "mean_weights": [ + 0.6399283409118652, + 0.3600716292858124 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6202129125595093, + "mean_weights": [ + 0.6677408218383789, + 0.3322591781616211 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6833909153938293, + "mean_weights": [ + 0.4370756149291992, + 0.5629243850708008 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5648316144943237, + "mean_weights": [ + 0.7345826625823975, + 0.26541733741760254 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6891762018203735, + "mean_weights": [ + 0.5292186141014099, + 0.4707813858985901 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6616126298904419, + "mean_weights": [ + 0.5340027809143066, + 0.46599721908569336 + ], + "sources": 2 + }, + { + "entropy_mean": 0.677177906036377, + "mean_weights": [ + 0.5088601112365723, + 0.49113988876342773 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6097527742385864, + "mean_weights": [ + 0.0915260910987854, + 0.10751913487911224, + 0.8009548187255859 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0935723781585693, + "mean_weights": [ + 0.35386547446250916, + 0.31079742312431335, + 0.3353371024131775 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9486287832260132, + "mean_weights": [ + 0.17675328254699707, + 0.25682568550109863, + 0.5664210915565491 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0780861377716064, + "mean_weights": [ + 0.41310545802116394, + 0.254355251789093, + 0.33253931999206543 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0347599983215332, + "mean_weights": [ + 0.33605778217315674, + 0.2185833752155304, + 0.44535884261131287 + ], + "sources": 3 + }, + { + "entropy_mean": 1.082558274269104, + "mean_weights": [ + 0.35693180561065674, + 0.2650938034057617, + 0.37797439098358154 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0250523090362549, + "mean_weights": [ + 0.3271831274032593, + 0.22001737356185913, + 0.4527994990348816 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0847759246826172, + "mean_weights": [ + 0.3981170058250427, + 0.2824544310569763, + 0.31942856311798096 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3649413585662842, + "mean_weights": [ + 0.2614266574382782, + 0.18852941691875458, + 0.29208099842071533, + 0.2579629421234131 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3699798583984375, + "mean_weights": [ + 0.28992554545402527, + 0.21131852269172668, + 0.21504375338554382, + 0.28371214866638184 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3513643741607666, + "mean_weights": [ + 0.2640669643878937, + 0.18520507216453552, + 0.25140926241874695, + 0.29931870102882385 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3624136447906494, + "mean_weights": [ + 0.2960764467716217, + 0.20179283618927002, + 0.1957336813211441, + 0.306397020816803 + ], + "sources": 4 + }, + { + "entropy_mean": 1.344443917274475, + "mean_weights": [ + 0.2756163477897644, + 0.18691295385360718, + 0.22674894332885742, + 0.310721755027771 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3650298118591309, + "mean_weights": [ + 0.29695892333984375, + 0.20487771928310394, + 0.20126259326934814, + 0.29690074920654297 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3157862424850464, + "mean_weights": [ + 0.3690307140350342, + 0.1595282256603241, + 0.1986914426088333, + 0.2727496325969696 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3748846054077148, + "mean_weights": [ + 0.2870766520500183, + 0.24565547704696655, + 0.19725458323955536, + 0.27001330256462097 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5905678272247314, + "mean_weights": [ + 0.24015748500823975, + 0.15661165118217468, + 0.16623832285404205, + 0.22488132119178772, + 0.2121112197637558 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5939161777496338, + "mean_weights": [ + 0.22482120990753174, + 0.1766967624425888, + 0.16066738963127136, + 0.1912076622247696, + 0.2466069757938385 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5702942609786987, + "mean_weights": [ + 0.19429656863212585, + 0.14914008975028992, + 0.16449880599975586, + 0.24546070396900177, + 0.2466038316488266 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5767078399658203, + "mean_weights": [ + 0.2156025767326355, + 0.163354754447937, + 0.14497725665569305, + 0.188253253698349, + 0.28781211376190186 + ], + "sources": 5 + }, + { + "entropy_mean": 1.563880443572998, + "mean_weights": [ + 0.19932349026203156, + 0.13482177257537842, + 0.16151736676692963, + 0.22656697034835815, + 0.27777040004730225 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5532903671264648, + "mean_weights": [ + 0.18264183402061462, + 0.1530366837978363, + 0.13780325651168823, + 0.20116378366947174, + 0.3253544569015503 + ], + "sources": 5 + }, + { + "entropy_mean": 1.555888056755066, + "mean_weights": [ + 0.16727644205093384, + 0.14606758952140808, + 0.17037177085876465, + 0.238726407289505, + 0.27755776047706604 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5832005739212036, + "mean_weights": [ + 0.19321653246879578, + 0.1922680139541626, + 0.14841610193252563, + 0.18377900123596191, + 0.2823203504085541 + ], + "sources": 5 + }, + { + "entropy_mean": 1.754276990890503, + "mean_weights": [ + 0.14657320082187653, + 0.11666879802942276, + 0.14185255765914917, + 0.20404310524463654, + 0.22556880116462708, + 0.16529354453086853 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7707866430282593, + "mean_weights": [ + 0.16043853759765625, + 0.16230282187461853, + 0.13029643893241882, + 0.13386015594005585, + 0.1856813132762909, + 0.22742073237895966 + ], + "sources": 6 + }, + { + "entropy_mean": 1.736708641052246, + "mean_weights": [ + 0.13142991065979004, + 0.11066372692584991, + 0.133383646607399, + 0.1694999635219574, + 0.19246995449066162, + 0.26255279779434204 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7766164541244507, + "mean_weights": [ + 0.15833690762519836, + 0.1716131567955017, + 0.14024052023887634, + 0.1328560709953308, + 0.18226483464241028, + 0.2146884948015213 + ], + "sources": 6 + }, + { + "entropy_mean": 1.701073408126831, + "mean_weights": [ + 0.13303303718566895, + 0.10862451791763306, + 0.11869382858276367, + 0.1635148823261261, + 0.15635144710540771, + 0.3197822570800781 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7839610576629639, + "mean_weights": [ + 0.16950416564941406, + 0.17183998227119446, + 0.1458210051059723, + 0.14086657762527466, + 0.17608115077018738, + 0.19588710367679596 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7059028148651123, + "mean_weights": [ + 0.131662979722023, + 0.12348289042711258, + 0.13605868816375732, + 0.1631917655467987, + 0.1328202337026596, + 0.31278344988822937 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0386276245117188, + "mean_weights": [ + 0.10574116557836533, + 0.11599776148796082, + 0.13880112767219543, + 0.14508166909217834, + 0.1208304762840271, + 0.0973203182220459, + 0.11794178932905197, + 0.1582857072353363 + ], + "sources": 8 + }, + { + "entropy_mean": 2.073401689529419, + "mean_weights": [ + 0.08234729617834091, + 0.11217708140611649, + 0.09830836206674576, + 0.10492381453514099, + 0.0801539495587349, + 0.08196873217821121, + 0.12732365727424622, + 0.15057885646820068, + 0.16221825778484344 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1613893508911133, + "mean_weights": [ + 0.08800435066223145, + 0.10946163535118103, + 0.12182775884866714, + 0.13633206486701965, + 0.09780478477478027, + 0.09447456896305084, + 0.10239346325397491, + 0.1139272153377533, + 0.1357741802930832 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1004738807678223, + "mean_weights": [ + 0.0764283686876297, + 0.1106799989938736, + 0.10473106801509857, + 0.09947474300861359, + 0.08496987819671631, + 0.09045234322547913, + 0.11614634096622467, + 0.15784716606140137, + 0.15927010774612427 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1486666202545166, + "mean_weights": [ + 0.11571940034627914, + 0.13334739208221436, + 0.11260700225830078, + 0.09253433346748352, + 0.12580914795398712, + 0.09090389311313629, + 0.07658449560403824, + 0.12050127983093262, + 0.13199307024478912 + ], + "sources": 9 + }, + { + "entropy_mean": 2.09881854057312, + "mean_weights": [ + 0.08689843118190765, + 0.12281537055969238, + 0.09745042026042938, + 0.10618248581886292, + 0.08201787620782852, + 0.08432166278362274, + 0.10715842247009277, + 0.1642390340566635, + 0.14891628921031952 + ], + "sources": 9 + }, + { + "entropy_mean": 2.162623405456543, + "mean_weights": [ + 0.08653080463409424, + 0.10681838542222977, + 0.12933851778507233, + 0.15187114477157593, + 0.10101784765720367, + 0.1048596054315567, + 0.10826808214187622, + 0.10903221368789673, + 0.10226339101791382 + ], + "sources": 9 + }, + { + "entropy_mean": 2.128880739212036, + "mean_weights": [ + 0.08515369892120361, + 0.1271955668926239, + 0.10760273039340973, + 0.13336311280727386, + 0.09612702578306198, + 0.09186217933893204, + 0.10966718196868896, + 0.14823369681835175, + 0.10079480707645416 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.059164728969335556, + 0.05497055873274803, + 0.06322737783193588, + 0.09420984983444214, + 0.1427508294582367, + 0.08687379956245422, + 0.13196426630020142, + 0.13156811892986298, + 0.1375482678413391, + 0.08686517924070358, + 0.10773976147174835, + 0.10728900134563446, + 0.09626162052154541, + 0.09486141800880432, + 0.10284213721752167, + 0.09727711230516434, + 0.10356217622756958, + 0.07466398179531097, + 0.07561852782964706, + 0.07148022204637527, + 0.07180605828762054, + 0.07027329504489899, + 0.0725661888718605, + 0.0614251047372818, + 0.07843957841396332, + 0.051713962107896805, + 0.05497821792960167, + 0.049748897552490234, + 0.051289189606904984, + 0.048187900334596634, + 0.05525459349155426, + 0.05541291832923889, + 0.06343960762023926, + 0.04771421477198601, + 0.0510752871632576, + 0.045783884823322296, + 0.05331894010305405, + 0.04879521578550339, + 0.05376656726002693, + 0.05469390004873276, + 0.053772926330566406, + 0.04578454792499542, + 0.04647686332464218, + 0.04619058966636658, + 0.04720637574791908, + 0.04727534204721451, + 0.04898388683795929, + 0.04840882867574692, + 0.050989627838134766, + 0.04462234675884247, + 0.045993346720933914, + 0.04598758742213249, + 0.04785829409956932, + 0.048046499490737915, + 0.05040410906076431, + 0.050525642931461334, + 0.060722220689058304, + 0.05428502336144447, + 0.0517817847430706, + 0.05911613628268242, + 0.05938456952571869, + 0.0664587989449501, + 0.06048142537474632, + 0.06681882590055466 + ], + "loss_nats": 1.3208963871002197, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.8679020404815674, + "mean_weights": [ + 0.10851846635341644, + 0.08631414920091629, + 0.07985623925924301, + 0.06279467046260834, + 0.06403353810310364, + 0.04922313988208771, + 0.04555980861186981, + 0.17178596556186676, + 0.3319139778614044 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.06863304227590561, + 0.1121254414319992, + 0.2329072803258896, + 0.24011382460594177, + 0.2503340542316437, + 0.2569790184497833, + 0.25972625613212585, + 0.2627657949924469, + 0.08112318068742752, + 0.08153681457042694, + 0.08877323567867279, + 0.09013202041387558, + 0.1004815474152565, + 0.10180940479040146, + 0.1074439063668251, + 0.1096077710390091, + 0.02391901984810829, + 0.02774897962808609, + 0.03736373037099838, + 0.04423725977540016, + 0.05812005326151848, + 0.0665174275636673, + 0.09495555609464645, + 0.11069628596305847, + 0.037739042192697525, + 0.051317207515239716, + 0.07054752856492996, + 0.08783987164497375, + 0.09602206200361252, + 0.11682964116334915, + 0.12318167835474014, + 0.1450834721326828, + 0.025793075561523438, + 0.04204847663640976, + 0.05959298834204674, + 0.07195216417312622, + 0.0953904464840889, + 0.10633229464292526, + 0.1374969780445099, + 0.155330628156662, + 0.07373028248548508, + 0.08552751690149307, + 0.1056193858385086, + 0.1200723722577095, + 0.14871154725551605, + 0.1710408478975296, + 0.19494028389453888, + 0.22501137852668762, + 0.03796331584453583, + 0.07476522773504257, + 0.08087591826915741, + 0.11732935905456543, + 0.12419891357421875, + 0.16285960376262665, + 0.16685618460178375, + 0.21229317784309387, + 0.013732252642512321, + 0.07707780599594116, + 0.07903609424829483, + 0.13612744212150574, + 0.13956527411937714, + 0.22218629717826843, + 0.22416621446609497, + 0.3350115120410919 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00012951558164786547, + 0.00013356709678191692, + 0.00012791644257958978, + 0.00012608003453351557, + 0.00014208961511030793, + 0.0001364551135338843, + 0.00012968444207217544, + 0.0001266632170882076, + 0.00014814182941336185, + 0.00014347422984428704, + 0.00013805474736727774, + 0.00012389813491608948, + 0.00012311733735259622, + 0.00011545565212145448, + 0.00010836163710337132, + 0.00010558807844063267, + 0.0001276766270166263, + 0.00012500194134190679, + 0.00011527249444043264, + 9.89057298284024e-05, + 9.474242688156664e-05, + 8.906467701308429e-05, + 8.076374069787562e-05, + 7.427590753650293e-05, + 8.562662696931511e-05, + 8.121215068968013e-05, + 7.700689457124099e-05, + 7.388105586869642e-05, + 9.238972415914759e-05, + 7.262064173119143e-05, + 5.337457696441561e-05, + 4.774491026182659e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00013149644291843288, + "first_to_last_ratio": 1.8017636131760206, + "imbalance_abs_log_ratio": 0.5887659703237834, + "last_quartile_mean": 7.298207265193923e-05, + "mean": 0.00010773822862120142, + "normalized": [ + 1.2021320872392602, + 1.239737264026566, + 1.1872892678543405, + 1.1702441756008668, + 1.3188412036166208, + 1.2665431321842968, + 1.2036994085742312, + 1.1756571340479791, + 1.3750163828497322, + 1.3316928603748481, + 1.281390543858547, + 1.149992314721499, + 1.1427451418889247, + 1.07163124546429, + 1.0057863257095283, + 0.9800428296614334, + 1.185063358202469, + 1.1602375771501046, + 1.0699312204744063, + 0.9180188972304961, + 0.879376133189205, + 0.826676641642478, + 0.7496293723357395, + 0.6894108849482833, + 0.7947654984227666, + 0.7537914046760064, + 0.714759241512971, + 0.6857459679280232, + 0.8575389194858783, + 0.6740471108590389, + 0.4954098247900116, + 0.4431566294791582 + ], + "population_cv": 0.25218310684687384 + }, + "activation_output_rms_by_block": [ + 0.09629185497760773, + 0.23656627535820007, + 0.24915747344493866, + 0.25298237800598145, + 0.06278451532125473, + 0.06907467544078827, + 0.07873155176639557, + 0.08508986234664917, + 0.020605826750397682, + 0.036092713475227356, + 0.056794870644807816, + 0.09975064545869827, + 0.04193875193595886, + 0.07166349142789841, + 0.09691529721021652, + 0.12217064946889877, + 0.03912050276994705, + 0.06314743310213089, + 0.09272204339504242, + 0.13611653447151184, + 0.08095869421958923, + 0.11476555466651917, + 0.15831638872623444, + 0.2034570872783661, + 0.06863201409578323, + 0.10555410385131836, + 0.1432904452085495, + 0.18208329379558563, + 0.0647045373916626, + 0.11017410457134247, + 0.1775582730770111, + 0.26324138045310974 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.14133482333272696, + "first_to_last_ratio": 1.0138449659237454, + "imbalance_abs_log_ratio": 0.013749999913252568, + "last_quartile_mean": 0.13940476905554533, + "mean": 0.11501416325336322, + "normalized": [ + 0.8372173674427178, + 2.0568447282190045, + 2.166319924408552, + 2.199575868310147, + 0.545885076631367, + 0.600575385560338, + 0.684537882460257, + 0.7398207311147046, + 0.17915903717879833, + 0.3138110338264965, + 0.493807623672357, + 0.8672901026889954, + 0.3646398908591155, + 0.6230840567872656, + 0.842637936657619, + 1.0622226516552629, + 0.34013639419146147, + 0.5490404948043157, + 0.8061793501969512, + 1.183476283452695, + 0.7039019537206593, + 0.9978384524148004, + 1.3764947224584967, + 1.7689741986834533, + 0.5967266304811056, + 0.9177487438550901, + 1.2458504340278234, + 1.5831380122679037, + 0.5625788647362134, + 0.9579177160002578, + 1.5437948514729467, + 2.2887735997628282 + ], + "population_cv": 0.5787632827316688 + }, + "bits_per_byte": 1.8333388418158592, + "branch_output_rms_by_sublayer": [ + 0.061832595616579056, + 0.06949058920145035, + 0.16545063257217407, + 0.016246717423200607, + 0.030131764709949493, + 0.01437400933355093, + 0.017401011660695076, + 0.009605047293007374, + 0.06230635195970535, + 0.003905204124748707, + 0.018713468685746193, + 0.006136326584964991, + 0.02388188987970352, + 0.009991052560508251, + 0.01957116276025772, + 0.012231575325131416, + 0.017069583758711815, + 0.011866510845720768, + 0.0171954408288002, + 0.013619013130664825, + 0.025862904265522957, + 0.01692933216691017, + 0.05121344327926636, + 0.03019787184894085, + 0.028961533680558205, + 0.025677748024463654, + 0.02343091554939747, + 0.030332008376717567, + 0.021057208999991417, + 0.033979352563619614, + 0.02197093330323696, + 0.03636031225323677, + 0.023018483072519302, + 0.029833514243364334, + 0.031322672963142395, + 0.027288729324936867, + 0.04649944230914116, + 0.03461595997214317, + 0.059706710278987885, + 0.05130479484796524, + 0.06799915432929993, + 0.04028605669736862, + 0.05107458308339119, + 0.042938485741615295, + 0.06013522297143936, + 0.06026872992515564, + 0.05430271103978157, + 0.07599948346614838, + 0.03256120905280113, + 0.05845339596271515, + 0.021830003708600998, + 0.06043921038508415, + 0.029674656689167023, + 0.06409842520952225, + 0.02232637256383896, + 0.07010766118764877, + 0.012769640423357487, + 0.06335301697254181, + 0.014119756408035755, + 0.06395818293094635, + 0.019985381513834, + 0.0884840339422226, + 0.016093675047159195, + 0.10494526475667953 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.000398789437371907, + 0.00028832647280452344, + 0.0001681485181753329, + 0.00016145635035848935, + 0.00021300296240266657, + 0.00013825753128248657, + 0.00019391976413960478, + 0.00020756498947656875, + 0.00019766041683496002, + 0.00021579531469259755, + 0.00031473178789132234, + 0.0004571331086904526, + 0.000314360037989259, + 0.0003441385749331872, + 0.0003435958969973368, + 0.00033581613335726405, + 0.0003684434995047589, + 0.00041405341719364053, + 0.0005223123493106576, + 0.0005914285939563847, + 0.0005364415982437965, + 0.0004944188830187058, + 0.0005263780500011639, + 0.0005290329856481236, + 0.00044960372901005536, + 0.00041242869434524477, + 0.00040716767536389106, + 0.00039803118313106795, + 0.0004639191928511152, + 0.0004011531748031933, + 0.0003542360237715795, + 0.0002959933726195236 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00022118325325144741, + "first_to_last_ratio": 0.555992978075611, + "imbalance_abs_log_ratio": 0.5869996141717176, + "last_quartile_mean": 0.00039781663073695887, + "mean": 0.00035805436625533944, + "normalized": [ + 1.1137678379475986, + 0.8052589214871048, + 0.4696172816823607, + 0.4509269138289182, + 0.5948900012875913, + 0.38613558250506275, + 0.5415930719339831, + 0.5797024391780433, + 0.5520402359623856, + 0.6026886837031541, + 0.8790055856123189, + 1.2767142416703756, + 0.8779673357343709, + 0.9611349766023284, + 0.9596193466114812, + 0.9378914628785265, + 1.0290155189505792, + 1.1563981792037874, + 1.4587515152326918, + 1.6517843369479233, + 1.4982126984069335, + 1.380848635332995, + 1.4701064966927053, + 1.4775213920191497, + 1.255685648277863, + 1.1518605363162346, + 1.1371671839173367, + 1.1116501309391096, + 1.2956669058471426, + 1.1203694539424183, + 0.9893358583399121, + 0.8266715910076118 + ], + "population_cv": 0.3440411513988422 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.2754749655723572, + "mean_weights": [ + 0.9156467914581299, + 0.08435326814651489 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5644737482070923, + "mean_weights": [ + 0.687580943107605, + 0.312419056892395 + ], + "sources": 2 + }, + { + "entropy_mean": 0.659844160079956, + "mean_weights": [ + 0.5847400426864624, + 0.4152599275112152 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6759549379348755, + "mean_weights": [ + 0.41121309995651245, + 0.5887869000434875 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6126289367675781, + "mean_weights": [ + 0.6821228265762329, + 0.31787723302841187 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6899406909942627, + "mean_weights": [ + 0.5018250942230225, + 0.4981749355792999 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6708826422691345, + "mean_weights": [ + 0.4954773187637329, + 0.5045226812362671 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6821292638778687, + "mean_weights": [ + 0.507662296295166, + 0.4923376441001892 + ], + "sources": 2 + }, + { + "entropy_mean": 0.4932137727737427, + "mean_weights": [ + 0.06824157387018204, + 0.08329407870769501, + 0.8484643697738647 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0914835929870605, + "mean_weights": [ + 0.3164386749267578, + 0.30484890937805176, + 0.3787124454975128 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9062439203262329, + "mean_weights": [ + 0.15085633099079132, + 0.24178291857242584, + 0.6073607802391052 + ], + "sources": 3 + }, + { + "entropy_mean": 1.083204746246338, + "mean_weights": [ + 0.37457406520843506, + 0.2557586431503296, + 0.36966729164123535 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0121999979019165, + "mean_weights": [ + 0.28756123781204224, + 0.20949003100395203, + 0.5029487609863281 + ], + "sources": 3 + }, + { + "entropy_mean": 1.083662986755371, + "mean_weights": [ + 0.3445448875427246, + 0.2691023349761963, + 0.3863527774810791 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0198489427566528, + "mean_weights": [ + 0.29549887776374817, + 0.21839496493339539, + 0.48610615730285645 + ], + "sources": 3 + }, + { + "entropy_mean": 1.09075927734375, + "mean_weights": [ + 0.37909889221191406, + 0.2947847247123718, + 0.3261164128780365 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3622465133666992, + "mean_weights": [ + 0.23812799155712128, + 0.18678036332130432, + 0.30133122205734253, + 0.27376043796539307 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3688647747039795, + "mean_weights": [ + 0.2817423343658447, + 0.20896364748477936, + 0.21428769826889038, + 0.29500633478164673 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3463294506072998, + "mean_weights": [ + 0.24084992706775665, + 0.18361994624137878, + 0.25363558530807495, + 0.3218945264816284 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3590599298477173, + "mean_weights": [ + 0.28448131680488586, + 0.20098814368247986, + 0.19195020198822021, + 0.32258033752441406 + ], + "sources": 4 + }, + { + "entropy_mean": 1.33388352394104, + "mean_weights": [ + 0.2506865859031677, + 0.17553305625915527, + 0.2212599515914917, + 0.3525204062461853 + ], + "sources": 4 + }, + { + "entropy_mean": 1.368056297302246, + "mean_weights": [ + 0.28030258417129517, + 0.20992082357406616, + 0.2055395543575287, + 0.30423703789711 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3193475008010864, + "mean_weights": [ + 0.331452876329422, + 0.16040930151939392, + 0.19024288654327393, + 0.31789493560791016 + ], + "sources": 4 + }, + { + "entropy_mean": 1.375154972076416, + "mean_weights": [ + 0.27968907356262207, + 0.23754991590976715, + 0.200173020362854, + 0.2825879752635956 + ], + "sources": 4 + }, + { + "entropy_mean": 1.58534574508667, + "mean_weights": [ + 0.219548299908638, + 0.14909425377845764, + 0.16230814158916473, + 0.238446444272995, + 0.23060287535190582 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5883543491363525, + "mean_weights": [ + 0.21429260075092316, + 0.17355245351791382, + 0.15443870425224304, + 0.19207395613193512, + 0.26564228534698486 + ], + "sources": 5 + }, + { + "entropy_mean": 1.557962417602539, + "mean_weights": [ + 0.18274405598640442, + 0.1345885992050171, + 0.16781184077262878, + 0.2409304976463318, + 0.2739250063896179 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5758161544799805, + "mean_weights": [ + 0.20848727226257324, + 0.1600360870361328, + 0.14673775434494019, + 0.1911737024784088, + 0.29356515407562256 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5501718521118164, + "mean_weights": [ + 0.19179648160934448, + 0.1224389374256134, + 0.1609453707933426, + 0.22299185395240784, + 0.3018273711204529 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5619536638259888, + "mean_weights": [ + 0.1851068139076233, + 0.15683910250663757, + 0.1434454321861267, + 0.19703857600688934, + 0.3175700604915619 + ], + "sources": 5 + }, + { + "entropy_mean": 1.539466381072998, + "mean_weights": [ + 0.16455797851085663, + 0.1291618049144745, + 0.16395029425621033, + 0.24075636267662048, + 0.30157357454299927 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5839322805404663, + "mean_weights": [ + 0.19887274503707886, + 0.19211483001708984, + 0.1505190134048462, + 0.1765442043542862, + 0.2819492220878601 + ], + "sources": 5 + }, + { + "entropy_mean": 1.742410659790039, + "mean_weights": [ + 0.14086930453777313, + 0.1025102511048317, + 0.13237810134887695, + 0.20144738256931305, + 0.23810720443725586, + 0.1846877634525299 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7721164226531982, + "mean_weights": [ + 0.15754514932632446, + 0.16221940517425537, + 0.13461929559707642, + 0.132829487323761, + 0.18637867271900177, + 0.2264080047607422 + ], + "sources": 6 + }, + { + "entropy_mean": 1.702785611152649, + "mean_weights": [ + 0.12611088156700134, + 0.093788743019104, + 0.11905033886432648, + 0.16815990209579468, + 0.1890314519405365, + 0.3038586974143982 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7791109085083008, + "mean_weights": [ + 0.1589602679014206, + 0.17428798973560333, + 0.14412467181682587, + 0.1346644163131714, + 0.17661315202713013, + 0.2113495171070099 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6630908250808716, + "mean_weights": [ + 0.12914776802062988, + 0.09770391881465912, + 0.10840336978435516, + 0.16123777627944946, + 0.14582033455371857, + 0.3576868176460266 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7840867042541504, + "mean_weights": [ + 0.16590633988380432, + 0.1789071261882782, + 0.1510227620601654, + 0.13796116411685944, + 0.17152813076972961, + 0.19467446208000183 + ], + "sources": 6 + }, + { + "entropy_mean": 1.683567762374878, + "mean_weights": [ + 0.13294978439807892, + 0.11526219546794891, + 0.12470671534538269, + 0.16441014409065247, + 0.12441233545541763, + 0.3382588326931 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.040186882019043, + "mean_weights": [ + 0.10968604683876038, + 0.12523329257965088, + 0.14061537384986877, + 0.14645013213157654, + 0.1277395337820053, + 0.09453088045120239, + 0.1066841408610344, + 0.14906057715415955 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0406525135040283, + "mean_weights": [ + 0.0811384841799736, + 0.10538571327924728, + 0.09852427244186401, + 0.10189516097307205, + 0.07428760826587677, + 0.07758890837430954, + 0.12111422419548035, + 0.1561078429222107, + 0.1839577853679657 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1552717685699463, + "mean_weights": [ + 0.0870475172996521, + 0.11256187409162521, + 0.12929099798202515, + 0.1428859829902649, + 0.10518459975719452, + 0.09420198202133179, + 0.09015946090221405, + 0.10239497572183609, + 0.1362726092338562 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0712037086486816, + "mean_weights": [ + 0.0744246318936348, + 0.10551024973392487, + 0.0996902659535408, + 0.09580719470977783, + 0.07981772720813751, + 0.08862842619419098, + 0.10535156726837158, + 0.16113054752349854, + 0.1896393597126007 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1501903533935547, + "mean_weights": [ + 0.11268344521522522, + 0.13761170208454132, + 0.11773395538330078, + 0.09468340873718262, + 0.12949615716934204, + 0.08656363934278488, + 0.07351287454366684, + 0.11525599658489227, + 0.13245882093906403 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0591812133789062, + "mean_weights": [ + 0.08671218156814575, + 0.11372269690036774, + 0.09408563375473022, + 0.09857165813446045, + 0.0755121260881424, + 0.07813885807991028, + 0.09759317338466644, + 0.16771414875984192, + 0.1879495084285736 + ], + "sources": 9 + }, + { + "entropy_mean": 2.156510829925537, + "mean_weights": [ + 0.08810554444789886, + 0.10587318241596222, + 0.14924302697181702, + 0.14384441077709198, + 0.10258959233760834, + 0.10161470621824265, + 0.09773251414299011, + 0.1050909161567688, + 0.10590612888336182 + ], + "sources": 9 + }, + { + "entropy_mean": 2.115041732788086, + "mean_weights": [ + 0.08408671617507935, + 0.12077361345291138, + 0.11024445295333862, + 0.12559786438941956, + 0.09111277759075165, + 0.08841434121131897, + 0.09841190278530121, + 0.15489798784255981, + 0.12646032869815826 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06193256750702858, + 0.05745238810777664, + 0.05835108086466789, + 0.10827403515577316, + 0.1462581306695938, + 0.09429900348186493, + 0.1343904584646225, + 0.13445450365543365, + 0.132965549826622, + 0.07004298269748688, + 0.10048454254865646, + 0.09444751590490341, + 0.0899040549993515, + 0.08526375144720078, + 0.09440344572067261, + 0.08712789416313171, + 0.09822046011686325, + 0.06676777452230453, + 0.06851932406425476, + 0.06369198858737946, + 0.06479933112859726, + 0.05957653373479843, + 0.06633314490318298, + 0.052995361387729645, + 0.06819401681423187, + 0.043594684451818466, + 0.04792172461748123, + 0.040562085807323456, + 0.044625043869018555, + 0.03986358270049095, + 0.047260284423828125, + 0.045404937118291855, + 0.055028751492500305, + 0.03881967440247536, + 0.04440988972783089, + 0.03645912930369377, + 0.046716880053281784, + 0.04061056300997734, + 0.04784919694066048, + 0.04656746983528137, + 0.04573614522814751, + 0.03814668208360672, + 0.038641881197690964, + 0.03834152594208717, + 0.039248377084732056, + 0.03930538147687912, + 0.04086118936538696, + 0.04054627567529678, + 0.042870234698057175, + 0.03748319298028946, + 0.038618072867393494, + 0.03864935040473938, + 0.040195826441049576, + 0.04038320854306221, + 0.042261239141225815, + 0.04236035794019699, + 0.04895058274269104, + 0.04621855914592743, + 0.0413971021771431, + 0.049570661038160324, + 0.04732445627450943, + 0.05676992982625961, + 0.04880770668387413, + 0.05727806314826012 + ], + "loss_nats": 1.2707736492156982, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.7859909534454346, + "mean_weights": [ + 0.10341720283031464, + 0.07800072431564331, + 0.07825355231761932, + 0.05820700526237488, + 0.056350164115428925, + 0.04629836976528168, + 0.03902763873338699, + 0.16005055606365204, + 0.3803947865962982 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.061832595616579056, + 0.09629185497760773, + 0.23072001338005066, + 0.23656627535820007, + 0.245231032371521, + 0.24915747344493866, + 0.25102901458740234, + 0.25298237800598145, + 0.06230635195970535, + 0.06278451532125473, + 0.06768914312124252, + 0.06907467544078827, + 0.07755908370018005, + 0.07873155176639557, + 0.08334658294916153, + 0.08508986234664917, + 0.017069583758711815, + 0.020605826750397682, + 0.029851380735635757, + 0.036092713475227356, + 0.04892096668481827, + 0.056794870644807816, + 0.08500218391418457, + 0.09975064545869827, + 0.028961533680558205, + 0.04193875193595886, + 0.0552145354449749, + 0.07166349142789841, + 0.07856409251689911, + 0.09691529721021652, + 0.1019807904958725, + 0.12217064946889877, + 0.023018483072519302, + 0.03912050276994705, + 0.051225513219833374, + 0.06314743310213089, + 0.08105792105197906, + 0.09272204339504242, + 0.11709973961114883, + 0.13611653447151184, + 0.06799915432929993, + 0.08095869421958923, + 0.10063408315181732, + 0.11476555466651917, + 0.1381969004869461, + 0.15831638872623444, + 0.1768493950366974, + 0.2034570872783661, + 0.03256120905280113, + 0.06863201409578323, + 0.07359593361616135, + 0.10555410385131836, + 0.11136370152235031, + 0.1432904452085495, + 0.1461474746465683, + 0.18208329379558563, + 0.012769640423357487, + 0.0647045373916626, + 0.06652479618787766, + 0.11017410457134247, + 0.1125815287232399, + 0.1775582730770111, + 0.17906953394412994, + 0.26324138045310974 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 8.034013546791773, + "cross_entropy_nats": 5.5687538385391235, + "step": 0 + }, + { + "bits_per_byte": 5.133024208623962, + "cross_entropy_nats": 3.557941257953644, + "step": 100 + }, + { + "bits_per_byte": 3.0919287337263097, + "cross_entropy_nats": 2.1431616842746735, + "step": 500 + }, + { + "bits_per_byte": 1.9620262576829939, + "cross_entropy_nats": 1.359972968697548, + "step": 2000 + }, + { + "bits_per_byte": 1.7942183136872292, + "cross_entropy_nats": 1.2436573654413223, + "step": 4000 + }, + { + "bits_per_byte": 1.7189223528368378, + "cross_entropy_nats": 1.1914661824703217, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.40.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.40.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.48.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.48.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "40": true, + "41": true, + "42": true, + "43": true, + "44": true, + "45": true, + "46": true, + "47": true, + "48": true, + "49": true, + "50": true, + "51": true, + "52": true, + "53": true, + "54": true, + "55": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_groups_6_7_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "ea57f192dc3b225e020f72ab0760daadcfde28c68e82e605e1d983d8387a2fea", + "final_model_state": "2059b1f77571c9e308fbb36b70b4863e890e7d4044a168e9ea37d60b9dcbe45b", + "final_optimizer_state": "4b726082605425fd042e9226a9a5c2131d3a43dd69d895482aec456392c2ed29", + "final_public_parameters": "818cfde1d28680960be5bda3f5a6762896e63d530d048ffa401f0e6615bc5a71", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "bbf2cfc5c0019fb54bf6927274ab9b185b564a77b4b8ef2dc6f5d81575f835c2" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "add6249175ac3bb3fa911069fbe8f25d17d9ab05df34b66ee89996983522ce83", + "1": "619b5ee4f70e975312ead9adf5df4f9d8584ce2839369c749442e429923405cf", + "7999": "fe22994f7750c2f8323fdb5ac51c9be1e208733abaf6a337f5b1ae1df1cd2730" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "1050c147f8c72acbe6c7271cdf6f80898558c025ee0f5a108722c012fd1a7bad", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073002, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 196.79001417892673, + "measured_steps": 7980, + "median_ms": 197.57878049858846, + "p95_ms": 198.81929501425475, + "peak_allocated_bytes": 10566180864, + "peak_reserved_bytes": 11068768256, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 8.018870646365954, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.558257579803467, + "step": 1, + "unclipped_grad_norm": 26.832857131958008 + }, + { + "bits_per_byte": 7.214403910759477, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.000643730163574, + "step": 10, + "unclipped_grad_norm": 12.742377281188965 + }, + { + "bits_per_byte": 6.684458634182628, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.6333136558532715, + "step": 20, + "unclipped_grad_norm": 3.4547319412231445 + }, + { + "bits_per_byte": 6.541276198033305, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.534067153930664, + "step": 30, + "unclipped_grad_norm": 3.060602903366089 + }, + { + "bits_per_byte": 6.411364631155161, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.444019317626953, + "step": 40, + "unclipped_grad_norm": 2.99996018409729 + }, + { + "bits_per_byte": 6.086111041843417, + "learning_rate": 3.75e-05, + "loss_nats": 4.218570709228516, + "step": 50, + "unclipped_grad_norm": 2.977649688720703 + }, + { + "bits_per_byte": 5.879963005490806, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.075679779052734, + "step": 60, + "unclipped_grad_norm": 2.708503007888794 + }, + { + "bits_per_byte": 5.710771977016803, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.9584054946899414, + "step": 70, + "unclipped_grad_norm": 2.5620598793029785 + }, + { + "bits_per_byte": 5.512682015787718, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.8210999965667725, + "step": 80, + "unclipped_grad_norm": 2.552194595336914 + }, + { + "bits_per_byte": 5.341966533096467, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.7027690410614014, + "step": 90, + "unclipped_grad_norm": 2.38407039642334 + }, + { + "bits_per_byte": 5.170387353532582, + "learning_rate": 7.5e-05, + "loss_nats": 3.5838394165039062, + "step": 100, + "unclipped_grad_norm": 2.25028920173645 + }, + { + "bits_per_byte": 5.001568839480347, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.4668233394622803, + "step": 110, + "unclipped_grad_norm": 2.1942684650421143 + }, + { + "bits_per_byte": 4.734509019741052, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.2817115783691406, + "step": 120, + "unclipped_grad_norm": 1.8807592391967773 + }, + { + "bits_per_byte": 4.585906378367137, + "learning_rate": 9.75e-05, + "loss_nats": 3.178708076477051, + "step": 130, + "unclipped_grad_norm": 1.6670387983322144 + }, + { + "bits_per_byte": 4.392610949478806, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.0447258949279785, + "step": 140, + "unclipped_grad_norm": 1.4590272903442383 + }, + { + "bits_per_byte": 4.260407197829961, + "learning_rate": 0.0001125, + "loss_nats": 2.9530892372131348, + "step": 150, + "unclipped_grad_norm": 1.2309588193893433 + }, + { + "bits_per_byte": 4.152410348255359, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.8782315254211426, + "step": 160, + "unclipped_grad_norm": 1.0431599617004395 + }, + { + "bits_per_byte": 3.994346941257481, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.7686703205108643, + "step": 170, + "unclipped_grad_norm": 0.8884556293487549 + }, + { + "bits_per_byte": 3.855331644738564, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.6723122596740723, + "step": 180, + "unclipped_grad_norm": 1.047376275062561 + }, + { + "bits_per_byte": 3.8732986722742018, + "learning_rate": 0.0001425, + "loss_nats": 2.6847660541534424, + "step": 190, + "unclipped_grad_norm": 0.9831625819206238 + }, + { + "bits_per_byte": 3.744339545316535, + "learning_rate": 0.00015, + "loss_nats": 2.5953783988952637, + "step": 200, + "unclipped_grad_norm": 0.7422687411308289 + }, + { + "bits_per_byte": 3.6965156416605103, + "learning_rate": 0.0001575, + "loss_nats": 2.5622293949127197, + "step": 210, + "unclipped_grad_norm": 0.8436761498451233 + }, + { + "bits_per_byte": 3.660975426859711, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.537594795227051, + "step": 220, + "unclipped_grad_norm": 0.8301674723625183 + }, + { + "bits_per_byte": 3.6749638077502884, + "learning_rate": 0.0001725, + "loss_nats": 2.547290802001953, + "step": 230, + "unclipped_grad_norm": 0.7788304090499878 + }, + { + "bits_per_byte": 3.670278312410182, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.5440430641174316, + "step": 240, + "unclipped_grad_norm": 0.9293244481086731 + }, + { + "bits_per_byte": 3.5213341134916534, + "learning_rate": 0.0001875, + "loss_nats": 2.440802812576294, + "step": 250, + "unclipped_grad_norm": 0.8449331521987915 + }, + { + "bits_per_byte": 3.5275378716735766, + "learning_rate": 0.000195, + "loss_nats": 2.4451029300689697, + "step": 260, + "unclipped_grad_norm": 1.0424093008041382 + }, + { + "bits_per_byte": 3.4890615690572475, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.41843318939209, + "step": 270, + "unclipped_grad_norm": 0.852695643901825 + }, + { + "bits_per_byte": 3.49760532318681, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.4243552684783936, + "step": 280, + "unclipped_grad_norm": 0.9987691640853882 + }, + { + "bits_per_byte": 3.3905089747721253, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.3501217365264893, + "step": 290, + "unclipped_grad_norm": 0.8998715877532959 + }, + { + "bits_per_byte": 3.4862923044130545, + "learning_rate": 0.000225, + "loss_nats": 2.416513681411743, + "step": 300, + "unclipped_grad_norm": 1.1805115938186646 + }, + { + "bits_per_byte": 3.39208261602752, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.351212501525879, + "step": 310, + "unclipped_grad_norm": 1.4041028022766113 + }, + { + "bits_per_byte": 3.3726688704328245, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.3377559185028076, + "step": 320, + "unclipped_grad_norm": 0.9700325131416321 + }, + { + "bits_per_byte": 3.368929623637765, + "learning_rate": 0.0002475, + "loss_nats": 2.3351640701293945, + "step": 330, + "unclipped_grad_norm": 1.7658121585845947 + }, + { + "bits_per_byte": 3.3517943042017, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.323286771774292, + "step": 340, + "unclipped_grad_norm": 0.8378285765647888 + }, + { + "bits_per_byte": 3.335395070511328, + "learning_rate": 0.0002625, + "loss_nats": 2.311919689178467, + "step": 350, + "unclipped_grad_norm": 1.0497196912765503 + }, + { + "bits_per_byte": 3.3124130288765308, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.295989751815796, + "step": 360, + "unclipped_grad_norm": 0.9907864928245544 + }, + { + "bits_per_byte": 3.3257537231039587, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.30523681640625, + "step": 370, + "unclipped_grad_norm": 1.2726961374282837 + }, + { + "bits_per_byte": 3.3559559403872785, + "learning_rate": 0.000285, + "loss_nats": 2.326171398162842, + "step": 380, + "unclipped_grad_norm": 0.8855420351028442 + }, + { + "bits_per_byte": 3.2984136410962983, + "learning_rate": 0.0002925, + "loss_nats": 2.2862861156463623, + "step": 390, + "unclipped_grad_norm": 0.9592195749282837 + }, + { + "bits_per_byte": 3.2497284483247526, + "learning_rate": 0.0003, + "loss_nats": 2.252540111541748, + "step": 400, + "unclipped_grad_norm": 1.1517537832260132 + }, + { + "bits_per_byte": 3.194011916855332, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.2139203548431396, + "step": 410, + "unclipped_grad_norm": 0.8549288511276245 + }, + { + "bits_per_byte": 3.2181262922634097, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.230635166168213, + "step": 420, + "unclipped_grad_norm": 0.9800664782524109 + }, + { + "bits_per_byte": 3.202585252000843, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.219862937927246, + "step": 430, + "unclipped_grad_norm": 1.1209087371826172 + }, + { + "bits_per_byte": 3.204228030281884, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.221001625061035, + "step": 440, + "unclipped_grad_norm": 0.9115833640098572 + }, + { + "bits_per_byte": 3.180584543371869, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.204613208770752, + "step": 450, + "unclipped_grad_norm": 1.1177111864089966 + }, + { + "bits_per_byte": 3.1362704616546524, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.1738970279693604, + "step": 460, + "unclipped_grad_norm": 0.9684751629829407 + }, + { + "bits_per_byte": 3.166254604936678, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.1946804523468018, + "step": 470, + "unclipped_grad_norm": 1.160462737083435 + }, + { + "bits_per_byte": 3.1236018756267963, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.1651158332824707, + "step": 480, + "unclipped_grad_norm": 0.8464173078536987 + }, + { + "bits_per_byte": 3.1022691476137734, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.150329113006592, + "step": 490, + "unclipped_grad_norm": 1.2556270360946655 + }, + { + "bits_per_byte": 3.1225273280242054, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.1643710136413574, + "step": 500, + "unclipped_grad_norm": 0.8427110314369202 + }, + { + "bits_per_byte": 3.097430587714399, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.14697527885437, + "step": 510, + "unclipped_grad_norm": 0.9524534344673157 + }, + { + "bits_per_byte": 2.9928661678857167, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.0744967460632324, + "step": 520, + "unclipped_grad_norm": 1.027021050453186 + }, + { + "bits_per_byte": 3.0748468578990003, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.131321430206299, + "step": 530, + "unclipped_grad_norm": 1.1862117052078247 + }, + { + "bits_per_byte": 3.028872799635656, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.099454641342163, + "step": 540, + "unclipped_grad_norm": 1.3201611042022705 + }, + { + "bits_per_byte": 2.9832537135636925, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.06783390045166, + "step": 550, + "unclipped_grad_norm": 1.5961934328079224 + }, + { + "bits_per_byte": 2.978780788780053, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.0647335052490234, + "step": 560, + "unclipped_grad_norm": 0.9054278135299683 + }, + { + "bits_per_byte": 2.9753459512770206, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.0623526573181152, + "step": 570, + "unclipped_grad_norm": 1.4001652002334595 + }, + { + "bits_per_byte": 2.8675809343157814, + "learning_rate": 0.000299626473885547, + "loss_nats": 1.9876556396484375, + "step": 580, + "unclipped_grad_norm": 1.1591882705688477 + }, + { + "bits_per_byte": 2.920566037324199, + "learning_rate": 0.00029958384005397226, + "loss_nats": 2.0243821144104004, + "step": 590, + "unclipped_grad_norm": 1.003491997718811 + }, + { + "bits_per_byte": 2.8313622479573284, + "learning_rate": 0.0002995389065559004, + "loss_nats": 1.9625507593154907, + "step": 600, + "unclipped_grad_norm": 0.7479391694068909 + }, + { + "bits_per_byte": 2.9076528919668156, + "learning_rate": 0.000299491674159121, + "loss_nats": 2.0154314041137695, + "step": 610, + "unclipped_grad_norm": 0.9462049007415771 + }, + { + "bits_per_byte": 2.8032210707619485, + "learning_rate": 0.00029944214367070535, + "loss_nats": 1.9430447816848755, + "step": 620, + "unclipped_grad_norm": 0.9233419299125671 + }, + { + "bits_per_byte": 2.8106660277351487, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.9482052326202393, + "step": 630, + "unclipped_grad_norm": 1.0161503553390503 + }, + { + "bits_per_byte": 2.806694432378774, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.945452332496643, + "step": 640, + "unclipped_grad_norm": 0.9988572597503662 + }, + { + "bits_per_byte": 2.7401051577218065, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.8992961645126343, + "step": 650, + "unclipped_grad_norm": 1.2602344751358032 + }, + { + "bits_per_byte": 2.7263491253925447, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.889761209487915, + "step": 660, + "unclipped_grad_norm": 0.9217911958694458 + }, + { + "bits_per_byte": 2.733947662872965, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.8950281143188477, + "step": 670, + "unclipped_grad_norm": 1.2217377424240112 + }, + { + "bits_per_byte": 2.73908530060219, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.8985892534255981, + "step": 680, + "unclipped_grad_norm": 0.8587907552719116 + }, + { + "bits_per_byte": 2.626848906701168, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.8207929134368896, + "step": 690, + "unclipped_grad_norm": 1.0217201709747314 + }, + { + "bits_per_byte": 2.568523742383461, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.780364990234375, + "step": 700, + "unclipped_grad_norm": 1.0057233572006226 + }, + { + "bits_per_byte": 2.6612048489681266, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.844606637954712, + "step": 710, + "unclipped_grad_norm": 1.0225130319595337 + }, + { + "bits_per_byte": 2.712764559766193, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.880345106124878, + "step": 720, + "unclipped_grad_norm": 0.98109370470047 + }, + { + "bits_per_byte": 2.5224980893248574, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.748462438583374, + "step": 730, + "unclipped_grad_norm": 1.0771642923355103 + }, + { + "bits_per_byte": 2.54348788393206, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.7630114555358887, + "step": 740, + "unclipped_grad_norm": 0.8639598488807678 + }, + { + "bits_per_byte": 2.5638458142799925, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.7771224975585938, + "step": 750, + "unclipped_grad_norm": 1.0069993734359741 + }, + { + "bits_per_byte": 2.5355746182005605, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.7575263977050781, + "step": 760, + "unclipped_grad_norm": 1.2108391523361206 + }, + { + "bits_per_byte": 2.523838866070984, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.749391794204712, + "step": 770, + "unclipped_grad_norm": 0.9761292934417725 + }, + { + "bits_per_byte": 2.4322931889473143, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.6859371662139893, + "step": 780, + "unclipped_grad_norm": 0.8456069231033325 + }, + { + "bits_per_byte": 2.511277941182568, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.740685224533081, + "step": 790, + "unclipped_grad_norm": 0.9053306579589844 + }, + { + "bits_per_byte": 2.4794197029540905, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.7186027765274048, + "step": 800, + "unclipped_grad_norm": 1.1607964038848877 + }, + { + "bits_per_byte": 2.4933105696816287, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7282311916351318, + "step": 810, + "unclipped_grad_norm": 1.0348000526428223 + }, + { + "bits_per_byte": 2.4663392184774175, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.709536075592041, + "step": 820, + "unclipped_grad_norm": 1.0680373907089233 + }, + { + "bits_per_byte": 2.454802966222842, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.7015397548675537, + "step": 830, + "unclipped_grad_norm": 0.8886773586273193 + }, + { + "bits_per_byte": 2.442313414134534, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.6928826570510864, + "step": 840, + "unclipped_grad_norm": 1.0601491928100586 + }, + { + "bits_per_byte": 2.3715612994134325, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.643841028213501, + "step": 850, + "unclipped_grad_norm": 0.8216683864593506 + }, + { + "bits_per_byte": 2.408443666821565, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.6694059371948242, + "step": 860, + "unclipped_grad_norm": 0.9449710249900818 + }, + { + "bits_per_byte": 2.365196049522349, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.639428973197937, + "step": 870, + "unclipped_grad_norm": 1.1236071586608887 + }, + { + "bits_per_byte": 2.4048370186503214, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.6669059991836548, + "step": 880, + "unclipped_grad_norm": 0.9965853691101074 + }, + { + "bits_per_byte": 2.37003392149112, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.6427823305130005, + "step": 890, + "unclipped_grad_norm": 0.8055936098098755 + }, + { + "bits_per_byte": 2.320599744275067, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.6085171699523926, + "step": 900, + "unclipped_grad_norm": 0.9317430853843689 + }, + { + "bits_per_byte": 2.3572525148442978, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.6339229345321655, + "step": 910, + "unclipped_grad_norm": 1.0633481740951538 + }, + { + "bits_per_byte": 2.322485018093825, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.6098239421844482, + "step": 920, + "unclipped_grad_norm": 1.1178252696990967 + }, + { + "bits_per_byte": 2.322306844067531, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.6097004413604736, + "step": 930, + "unclipped_grad_norm": 0.8794320225715637 + }, + { + "bits_per_byte": 2.275191852898363, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.577042818069458, + "step": 940, + "unclipped_grad_norm": 1.0244495868682861 + }, + { + "bits_per_byte": 2.2459396677784094, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.5567667484283447, + "step": 950, + "unclipped_grad_norm": 0.9427381157875061 + }, + { + "bits_per_byte": 2.2638611199115686, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.5691889524459839, + "step": 960, + "unclipped_grad_norm": 1.1330108642578125 + }, + { + "bits_per_byte": 2.275443979464528, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5772175788879395, + "step": 970, + "unclipped_grad_norm": 0.8932414054870605 + }, + { + "bits_per_byte": 2.268297756355876, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.5722641944885254, + "step": 980, + "unclipped_grad_norm": 1.0275044441223145 + }, + { + "bits_per_byte": 2.180212714132868, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.5112082958221436, + "step": 990, + "unclipped_grad_norm": 0.8848265409469604 + }, + { + "bits_per_byte": 2.239610878209309, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.5523799657821655, + "step": 1000, + "unclipped_grad_norm": 0.9493284821510315 + }, + { + "bits_per_byte": 2.2378451323329007, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.5511560440063477, + "step": 1010, + "unclipped_grad_norm": 0.8231925368309021 + }, + { + "bits_per_byte": 2.260054627900022, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.5665504932403564, + "step": 1020, + "unclipped_grad_norm": 0.7946434617042542 + }, + { + "bits_per_byte": 2.1540675675834025, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.4930858612060547, + "step": 1030, + "unclipped_grad_norm": 0.934628963470459 + }, + { + "bits_per_byte": 2.245589167135951, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.5565237998962402, + "step": 1040, + "unclipped_grad_norm": 0.8345707654953003 + }, + { + "bits_per_byte": 2.2144723461153433, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5349552631378174, + "step": 1050, + "unclipped_grad_norm": 0.9792539477348328 + }, + { + "bits_per_byte": 2.1678856856496256, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.5026638507843018, + "step": 1060, + "unclipped_grad_norm": 0.9139646291732788 + }, + { + "bits_per_byte": 2.2770767827518195, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.5783493518829346, + "step": 1070, + "unclipped_grad_norm": 0.8417990207672119 + }, + { + "bits_per_byte": 2.207630532298719, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.5302128791809082, + "step": 1080, + "unclipped_grad_norm": 0.8345304131507874 + }, + { + "bits_per_byte": 2.12152862202281, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.4705315828323364, + "step": 1090, + "unclipped_grad_norm": 0.7555098533630371 + }, + { + "bits_per_byte": 2.216691610242077, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5364935398101807, + "step": 1100, + "unclipped_grad_norm": 0.8691232204437256 + }, + { + "bits_per_byte": 2.234177774285902, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.5486140251159668, + "step": 1110, + "unclipped_grad_norm": 0.8235064744949341 + }, + { + "bits_per_byte": 2.128428393992775, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.4753141403198242, + "step": 1120, + "unclipped_grad_norm": 0.8414040207862854 + }, + { + "bits_per_byte": 2.1226958682742154, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.4713406562805176, + "step": 1130, + "unclipped_grad_norm": 0.8031512498855591 + }, + { + "bits_per_byte": 2.1362188641115583, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.4807140827178955, + "step": 1140, + "unclipped_grad_norm": 0.8719061017036438 + }, + { + "bits_per_byte": 2.127109287060657, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.4743998050689697, + "step": 1150, + "unclipped_grad_norm": 0.8182425498962402 + }, + { + "bits_per_byte": 2.0782510797423432, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.4405338764190674, + "step": 1160, + "unclipped_grad_norm": 0.7678866982460022 + }, + { + "bits_per_byte": 2.194470075889395, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.5210907459259033, + "step": 1170, + "unclipped_grad_norm": 0.8278438448905945 + }, + { + "bits_per_byte": 2.1105561288977626, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.4629260301589966, + "step": 1180, + "unclipped_grad_norm": 1.0056263208389282 + }, + { + "bits_per_byte": 2.114421610958555, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.46560537815094, + "step": 1190, + "unclipped_grad_norm": 0.8591680526733398 + }, + { + "bits_per_byte": 2.090182892093903, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.4488043785095215, + "step": 1200, + "unclipped_grad_norm": 0.917765736579895 + }, + { + "bits_per_byte": 2.212742544612965, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.5337562561035156, + "step": 1210, + "unclipped_grad_norm": 0.8319878578186035 + }, + { + "bits_per_byte": 2.070308576960197, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4350285530090332, + "step": 1220, + "unclipped_grad_norm": 0.7991510033607483 + }, + { + "bits_per_byte": 2.019974070566877, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.400139331817627, + "step": 1230, + "unclipped_grad_norm": 0.7604983448982239 + }, + { + "bits_per_byte": 2.0318589316547517, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.4083772897720337, + "step": 1240, + "unclipped_grad_norm": 0.9534946084022522 + }, + { + "bits_per_byte": 2.183396800930942, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.5134153366088867, + "step": 1250, + "unclipped_grad_norm": 0.7991413474082947 + }, + { + "bits_per_byte": 2.1301045369080835, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.4764759540557861, + "step": 1260, + "unclipped_grad_norm": 0.9281015396118164 + }, + { + "bits_per_byte": 1.9921252638779645, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.380836009979248, + "step": 1270, + "unclipped_grad_norm": 0.771392285823822 + }, + { + "bits_per_byte": 2.1320455331057215, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.4778213500976562, + "step": 1280, + "unclipped_grad_norm": 0.7563143372535706 + }, + { + "bits_per_byte": 2.0373350312408745, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4121730327606201, + "step": 1290, + "unclipped_grad_norm": 0.7661232352256775 + }, + { + "bits_per_byte": 2.0719695854022304, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.4361798763275146, + "step": 1300, + "unclipped_grad_norm": 0.7675361633300781 + }, + { + "bits_per_byte": 2.1444258762107316, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.4864027500152588, + "step": 1310, + "unclipped_grad_norm": 0.8049575090408325 + }, + { + "bits_per_byte": 2.0809224862581956, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.4423855543136597, + "step": 1320, + "unclipped_grad_norm": 0.9051185846328735 + }, + { + "bits_per_byte": 1.966023284973865, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.3627434968948364, + "step": 1330, + "unclipped_grad_norm": 0.7623180747032166 + }, + { + "bits_per_byte": 1.9803427324673535, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.372668981552124, + "step": 1340, + "unclipped_grad_norm": 0.6947795748710632 + }, + { + "bits_per_byte": 2.1662188297974634, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.5015084743499756, + "step": 1350, + "unclipped_grad_norm": 0.8032646179199219 + }, + { + "bits_per_byte": 2.0020988817668006, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.387749195098877, + "step": 1360, + "unclipped_grad_norm": 0.7354259490966797 + }, + { + "bits_per_byte": 2.0845504602781464, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4449002742767334, + "step": 1370, + "unclipped_grad_norm": 0.7366811633110046 + }, + { + "bits_per_byte": 2.0702659252627833, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.4349989891052246, + "step": 1380, + "unclipped_grad_norm": 0.6824614405632019 + }, + { + "bits_per_byte": 2.0773908225227276, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.4399375915527344, + "step": 1390, + "unclipped_grad_norm": 0.7256225943565369 + }, + { + "bits_per_byte": 2.065367515400909, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.4316036701202393, + "step": 1400, + "unclipped_grad_norm": 0.7995309829711914 + }, + { + "bits_per_byte": 1.9996553522633416, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.3860554695129395, + "step": 1410, + "unclipped_grad_norm": 0.7026796936988831 + }, + { + "bits_per_byte": 2.0381708669240677, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.412752389907837, + "step": 1420, + "unclipped_grad_norm": 0.7526254653930664 + }, + { + "bits_per_byte": 2.0373769950076848, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.4122021198272705, + "step": 1430, + "unclipped_grad_norm": 0.7283347845077515 + }, + { + "bits_per_byte": 2.0109293309753515, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.3938699960708618, + "step": 1440, + "unclipped_grad_norm": 0.906847357749939 + }, + { + "bits_per_byte": 2.0604617102850473, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.4282032251358032, + "step": 1450, + "unclipped_grad_norm": 0.9333901405334473 + }, + { + "bits_per_byte": 1.999319126180905, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.3858224153518677, + "step": 1460, + "unclipped_grad_norm": 0.7090993523597717 + }, + { + "bits_per_byte": 1.9999628572430843, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.3862686157226562, + "step": 1470, + "unclipped_grad_norm": 0.6991606950759888 + }, + { + "bits_per_byte": 1.8990007860193743, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.3162870407104492, + "step": 1480, + "unclipped_grad_norm": 0.6645700931549072 + }, + { + "bits_per_byte": 2.000543814637699, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.3866713047027588, + "step": 1490, + "unclipped_grad_norm": 0.7391359210014343 + }, + { + "bits_per_byte": 1.9105236236271823, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.3242740631103516, + "step": 1500, + "unclipped_grad_norm": 0.7324264049530029 + }, + { + "bits_per_byte": 1.9783522052662672, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.3712892532348633, + "step": 1510, + "unclipped_grad_norm": 0.7448433041572571 + }, + { + "bits_per_byte": 1.9770492647033306, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.3703861236572266, + "step": 1520, + "unclipped_grad_norm": 0.7791208028793335 + }, + { + "bits_per_byte": 1.9549718518120718, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.3550832271575928, + "step": 1530, + "unclipped_grad_norm": 0.7815102934837341 + }, + { + "bits_per_byte": 2.017270159330012, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3982651233673096, + "step": 1540, + "unclipped_grad_norm": 0.7528116703033447 + }, + { + "bits_per_byte": 1.9870283860369942, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.377303123474121, + "step": 1550, + "unclipped_grad_norm": 0.720079779624939 + }, + { + "bits_per_byte": 2.069979402166446, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.434800386428833, + "step": 1560, + "unclipped_grad_norm": 0.7640688419342041 + }, + { + "bits_per_byte": 2.014800488463622, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.3965532779693604, + "step": 1570, + "unclipped_grad_norm": 0.7967034578323364 + }, + { + "bits_per_byte": 1.9837679389419371, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.3750431537628174, + "step": 1580, + "unclipped_grad_norm": 0.6793249845504761 + }, + { + "bits_per_byte": 2.013798001591743, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3958584070205688, + "step": 1590, + "unclipped_grad_norm": 0.7808796763420105 + }, + { + "bits_per_byte": 2.0107666353876352, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.3937572240829468, + "step": 1600, + "unclipped_grad_norm": 0.7448055148124695 + }, + { + "bits_per_byte": 2.0185538378360524, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.3991549015045166, + "step": 1610, + "unclipped_grad_norm": 0.8283863067626953 + }, + { + "bits_per_byte": 2.035551915116729, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.4109370708465576, + "step": 1620, + "unclipped_grad_norm": 0.7735480666160583 + }, + { + "bits_per_byte": 1.913315933946591, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.326209545135498, + "step": 1630, + "unclipped_grad_norm": 0.7558816075325012 + }, + { + "bits_per_byte": 2.0144580710057545, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.3963159322738647, + "step": 1640, + "unclipped_grad_norm": 0.6372897624969482 + }, + { + "bits_per_byte": 1.9643466261106035, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.3615813255310059, + "step": 1650, + "unclipped_grad_norm": 0.740619957447052 + }, + { + "bits_per_byte": 1.9373945369632894, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.3428995609283447, + "step": 1660, + "unclipped_grad_norm": 0.6587967276573181 + }, + { + "bits_per_byte": 2.1028098583202506, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.4575567245483398, + "step": 1670, + "unclipped_grad_norm": 0.7939133644104004 + }, + { + "bits_per_byte": 2.0018959422387823, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.387608528137207, + "step": 1680, + "unclipped_grad_norm": 0.7275616526603699 + }, + { + "bits_per_byte": 1.899749598481231, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.3168060779571533, + "step": 1690, + "unclipped_grad_norm": 0.8117929697036743 + }, + { + "bits_per_byte": 1.9215979304815405, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.3319501876831055, + "step": 1700, + "unclipped_grad_norm": 0.6590717434883118 + }, + { + "bits_per_byte": 2.0062030757469906, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3905940055847168, + "step": 1710, + "unclipped_grad_norm": 0.7091744542121887 + }, + { + "bits_per_byte": 1.995371264430349, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.3830859661102295, + "step": 1720, + "unclipped_grad_norm": 0.6968663334846497 + }, + { + "bits_per_byte": 1.915485667069876, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.3277134895324707, + "step": 1730, + "unclipped_grad_norm": 0.711622953414917 + }, + { + "bits_per_byte": 1.9506143273871608, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.3520628213882446, + "step": 1740, + "unclipped_grad_norm": 0.6855903267860413 + }, + { + "bits_per_byte": 2.04537900378703, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.4177486896514893, + "step": 1750, + "unclipped_grad_norm": 0.6923466920852661 + }, + { + "bits_per_byte": 1.9171900151399266, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.328894853591919, + "step": 1760, + "unclipped_grad_norm": 0.7452006340026855 + }, + { + "bits_per_byte": 1.9802333515014046, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.3725931644439697, + "step": 1770, + "unclipped_grad_norm": 0.7692703008651733 + }, + { + "bits_per_byte": 1.9165595267418636, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.3284578323364258, + "step": 1780, + "unclipped_grad_norm": 0.8586394786834717 + }, + { + "bits_per_byte": 1.896442372105139, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3145136833190918, + "step": 1790, + "unclipped_grad_norm": 0.7278003692626953 + }, + { + "bits_per_byte": 1.9739018102098904, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.3682044744491577, + "step": 1800, + "unclipped_grad_norm": 0.7333357930183411 + }, + { + "bits_per_byte": 1.9500858247010595, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.351696491241455, + "step": 1810, + "unclipped_grad_norm": 0.6984512805938721 + }, + { + "bits_per_byte": 2.028312133446008, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.405918836593628, + "step": 1820, + "unclipped_grad_norm": 0.7212170958518982 + }, + { + "bits_per_byte": 1.868449787920105, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.2951107025146484, + "step": 1830, + "unclipped_grad_norm": 0.7083231210708618 + }, + { + "bits_per_byte": 1.968316329657819, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.364332914352417, + "step": 1840, + "unclipped_grad_norm": 0.677427351474762 + }, + { + "bits_per_byte": 1.8363894700768635, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.27288818359375, + "step": 1850, + "unclipped_grad_norm": 0.6434873342514038 + }, + { + "bits_per_byte": 2.000464874600953, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.386616587638855, + "step": 1860, + "unclipped_grad_norm": 0.7411749362945557 + }, + { + "bits_per_byte": 2.0079070798517393, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.391775131225586, + "step": 1870, + "unclipped_grad_norm": 0.7058617472648621 + }, + { + "bits_per_byte": 1.8676419854090023, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.2945507764816284, + "step": 1880, + "unclipped_grad_norm": 0.6586018800735474 + }, + { + "bits_per_byte": 1.9429705584695633, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3467645645141602, + "step": 1890, + "unclipped_grad_norm": 0.6531692743301392 + }, + { + "bits_per_byte": 1.866154163496388, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.2935194969177246, + "step": 1900, + "unclipped_grad_norm": 0.7685879468917847 + }, + { + "bits_per_byte": 1.8582224956212463, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.2880216836929321, + "step": 1910, + "unclipped_grad_norm": 0.6435375809669495 + }, + { + "bits_per_byte": 1.9271306123597969, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.335785150527954, + "step": 1920, + "unclipped_grad_norm": 0.6694431304931641 + }, + { + "bits_per_byte": 1.877656019220791, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.3014919757843018, + "step": 1930, + "unclipped_grad_norm": 0.6606138944625854 + }, + { + "bits_per_byte": 1.8513371701939538, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.2832491397857666, + "step": 1940, + "unclipped_grad_norm": 0.6543250679969788 + }, + { + "bits_per_byte": 1.9126519089316096, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.3257492780685425, + "step": 1950, + "unclipped_grad_norm": 0.7224748134613037 + }, + { + "bits_per_byte": 1.8867329195680285, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.307783603668213, + "step": 1960, + "unclipped_grad_norm": 0.6546120643615723 + }, + { + "bits_per_byte": 1.82966735618524, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.2682287693023682, + "step": 1970, + "unclipped_grad_norm": 0.6692179441452026 + }, + { + "bits_per_byte": 1.8179455346503834, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.2601038217544556, + "step": 1980, + "unclipped_grad_norm": 0.6375955939292908 + }, + { + "bits_per_byte": 1.9237587205069808, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.3334479331970215, + "step": 1990, + "unclipped_grad_norm": 0.7180453538894653 + }, + { + "bits_per_byte": 1.8680232709459652, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.2948150634765625, + "step": 2000, + "unclipped_grad_norm": 0.661660373210907 + }, + { + "bits_per_byte": 1.943437319384005, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.347088098526001, + "step": 2010, + "unclipped_grad_norm": 0.7082369923591614 + }, + { + "bits_per_byte": 1.9107379140101575, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.3244225978851318, + "step": 2020, + "unclipped_grad_norm": 0.6783074140548706 + }, + { + "bits_per_byte": 1.9120566769769736, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.3253366947174072, + "step": 2030, + "unclipped_grad_norm": 0.7567960619926453 + }, + { + "bits_per_byte": 1.9199861090776542, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.3308329582214355, + "step": 2040, + "unclipped_grad_norm": 0.6270517110824585 + }, + { + "bits_per_byte": 1.8635001272282732, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.291679859161377, + "step": 2050, + "unclipped_grad_norm": 0.7828612923622131 + }, + { + "bits_per_byte": 1.7576238237388093, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.218291997909546, + "step": 2060, + "unclipped_grad_norm": 0.6288750767707825 + }, + { + "bits_per_byte": 1.869910092608581, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.2961229085922241, + "step": 2070, + "unclipped_grad_norm": 0.6638445854187012 + }, + { + "bits_per_byte": 1.965530554679143, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.3624019622802734, + "step": 2080, + "unclipped_grad_norm": 0.6700616478919983 + }, + { + "bits_per_byte": 1.8244614413436226, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.264620304107666, + "step": 2090, + "unclipped_grad_norm": 0.6495924592018127 + }, + { + "bits_per_byte": 1.9164788668786088, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.3284019231796265, + "step": 2100, + "unclipped_grad_norm": 0.693890392780304 + }, + { + "bits_per_byte": 1.8188984905188141, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.2607643604278564, + "step": 2110, + "unclipped_grad_norm": 0.6431348919868469 + }, + { + "bits_per_byte": 1.8724841569440447, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.2979071140289307, + "step": 2120, + "unclipped_grad_norm": 0.7186147570610046 + }, + { + "bits_per_byte": 1.8906917882082395, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.3105276823043823, + "step": 2130, + "unclipped_grad_norm": 0.7031733989715576 + }, + { + "bits_per_byte": 1.77546788367908, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.2306605577468872, + "step": 2140, + "unclipped_grad_norm": 0.6101313233375549 + }, + { + "bits_per_byte": 1.8356633593249208, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2723848819732666, + "step": 2150, + "unclipped_grad_norm": 0.6404749155044556 + }, + { + "bits_per_byte": 1.8759986224144263, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.3003431558609009, + "step": 2160, + "unclipped_grad_norm": 0.6671034097671509 + }, + { + "bits_per_byte": 1.852071536113138, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.2837581634521484, + "step": 2170, + "unclipped_grad_norm": 0.6445640921592712 + }, + { + "bits_per_byte": 1.779679910781361, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.2335801124572754, + "step": 2180, + "unclipped_grad_norm": 0.6398025155067444 + }, + { + "bits_per_byte": 1.900652163432959, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.3174316883087158, + "step": 2190, + "unclipped_grad_norm": 0.6908156871795654 + }, + { + "bits_per_byte": 1.9469320068495373, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.3495104312896729, + "step": 2200, + "unclipped_grad_norm": 0.6775866746902466 + }, + { + "bits_per_byte": 1.966904524076886, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.3633543252944946, + "step": 2210, + "unclipped_grad_norm": 0.7027010321617126 + }, + { + "bits_per_byte": 1.8561585318282416, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2865910530090332, + "step": 2220, + "unclipped_grad_norm": 0.6675288677215576 + }, + { + "bits_per_byte": 1.8637964533357099, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.291885256767273, + "step": 2230, + "unclipped_grad_norm": 0.6956242322921753 + }, + { + "bits_per_byte": 1.8208453341265811, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.2621138095855713, + "step": 2240, + "unclipped_grad_norm": 0.6335778832435608 + }, + { + "bits_per_byte": 1.8361132659395778, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2726967334747314, + "step": 2250, + "unclipped_grad_norm": 0.6491182446479797 + }, + { + "bits_per_byte": 1.8553932090319023, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.2860605716705322, + "step": 2260, + "unclipped_grad_norm": 0.6391105651855469 + }, + { + "bits_per_byte": 1.8993306487437291, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.3165156841278076, + "step": 2270, + "unclipped_grad_norm": 0.6586543917655945 + }, + { + "bits_per_byte": 1.8028796824521505, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.2496609687805176, + "step": 2280, + "unclipped_grad_norm": 0.6979043483734131 + }, + { + "bits_per_byte": 1.8175587456685929, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.2598357200622559, + "step": 2290, + "unclipped_grad_norm": 0.6497045755386353 + }, + { + "bits_per_byte": 1.876450076872941, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.3006560802459717, + "step": 2300, + "unclipped_grad_norm": 0.6925584673881531 + }, + { + "bits_per_byte": 1.8264801736994503, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.266019582748413, + "step": 2310, + "unclipped_grad_norm": 0.6329606771469116 + }, + { + "bits_per_byte": 1.8394875655495062, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.2750356197357178, + "step": 2320, + "unclipped_grad_norm": 0.6535547375679016 + }, + { + "bits_per_byte": 1.7895314209880844, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.2404086589813232, + "step": 2330, + "unclipped_grad_norm": 0.6539317965507507 + }, + { + "bits_per_byte": 1.814682851780865, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.2578423023223877, + "step": 2340, + "unclipped_grad_norm": 0.608658492565155 + }, + { + "bits_per_byte": 1.7517715980952497, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.214235544204712, + "step": 2350, + "unclipped_grad_norm": 0.6089392304420471 + }, + { + "bits_per_byte": 1.841215303260074, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.276233196258545, + "step": 2360, + "unclipped_grad_norm": 0.6478467583656311 + }, + { + "bits_per_byte": 1.8727567494456623, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.2980960607528687, + "step": 2370, + "unclipped_grad_norm": 0.7012330293655396 + }, + { + "bits_per_byte": 1.7773048303729457, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.231933832168579, + "step": 2380, + "unclipped_grad_norm": 0.6178461909294128 + }, + { + "bits_per_byte": 1.7761807517669064, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.2311546802520752, + "step": 2390, + "unclipped_grad_norm": 0.6248509287834167 + }, + { + "bits_per_byte": 1.8835620754340707, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.3055857419967651, + "step": 2400, + "unclipped_grad_norm": 0.6689409017562866 + }, + { + "bits_per_byte": 1.8394308112747215, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.274996280670166, + "step": 2410, + "unclipped_grad_norm": 0.6848244071006775 + }, + { + "bits_per_byte": 1.9410624109582404, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.3454419374465942, + "step": 2420, + "unclipped_grad_norm": 0.7015742659568787 + }, + { + "bits_per_byte": 1.8422162422880954, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.2769269943237305, + "step": 2430, + "unclipped_grad_norm": 0.6618350148200989 + }, + { + "bits_per_byte": 1.7594248260586447, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.2195403575897217, + "step": 2440, + "unclipped_grad_norm": 0.6089354753494263 + }, + { + "bits_per_byte": 1.8629284568968052, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.2912836074829102, + "step": 2450, + "unclipped_grad_norm": 0.7045013904571533 + }, + { + "bits_per_byte": 1.8746934460770284, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.2994384765625, + "step": 2460, + "unclipped_grad_norm": 0.6667384505271912 + }, + { + "bits_per_byte": 1.8141024103342032, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.2574399709701538, + "step": 2470, + "unclipped_grad_norm": 0.6361277103424072 + }, + { + "bits_per_byte": 1.8153903884065146, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.2583327293395996, + "step": 2480, + "unclipped_grad_norm": 0.6309748291969299 + }, + { + "bits_per_byte": 1.7490704385807985, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.2123632431030273, + "step": 2490, + "unclipped_grad_norm": 0.6707090139389038 + }, + { + "bits_per_byte": 1.81991026445384, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2614656686782837, + "step": 2500, + "unclipped_grad_norm": 0.6535080671310425 + }, + { + "bits_per_byte": 1.7964764243952265, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.245222568511963, + "step": 2510, + "unclipped_grad_norm": 0.63571697473526 + }, + { + "bits_per_byte": 1.7730957269757293, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.2290163040161133, + "step": 2520, + "unclipped_grad_norm": 0.6286839246749878 + }, + { + "bits_per_byte": 1.7889358450681465, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.2399958372116089, + "step": 2530, + "unclipped_grad_norm": 0.6449724435806274 + }, + { + "bits_per_byte": 1.805525291570373, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.2514947652816772, + "step": 2540, + "unclipped_grad_norm": 0.6508165001869202 + }, + { + "bits_per_byte": 1.8357693006378524, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.2724583148956299, + "step": 2550, + "unclipped_grad_norm": 0.6523280143737793 + }, + { + "bits_per_byte": 1.7843949871374158, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.2368483543395996, + "step": 2560, + "unclipped_grad_norm": 0.6234970688819885 + }, + { + "bits_per_byte": 1.7678992711799095, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2254143953323364, + "step": 2570, + "unclipped_grad_norm": 0.6816107630729675 + }, + { + "bits_per_byte": 1.8151564920013412, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.2581706047058105, + "step": 2580, + "unclipped_grad_norm": 0.6391043066978455 + }, + { + "bits_per_byte": 1.7777144930473008, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.232217788696289, + "step": 2590, + "unclipped_grad_norm": 0.6147258281707764 + }, + { + "bits_per_byte": 1.8219353601677497, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.2628693580627441, + "step": 2600, + "unclipped_grad_norm": 0.6263659596443176 + }, + { + "bits_per_byte": 1.7260897728072082, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.1964342594146729, + "step": 2610, + "unclipped_grad_norm": 0.6208634376525879 + }, + { + "bits_per_byte": 1.776598669608503, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.2314443588256836, + "step": 2620, + "unclipped_grad_norm": 0.6409498453140259 + }, + { + "bits_per_byte": 1.7928606611434867, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.2427163124084473, + "step": 2630, + "unclipped_grad_norm": 0.6330344676971436 + }, + { + "bits_per_byte": 1.8162043822930483, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.2588969469070435, + "step": 2640, + "unclipped_grad_norm": 0.6721590757369995 + }, + { + "bits_per_byte": 1.7578326106769568, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2184367179870605, + "step": 2650, + "unclipped_grad_norm": 0.6089538931846619 + }, + { + "bits_per_byte": 1.756062221268975, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.2172095775604248, + "step": 2660, + "unclipped_grad_norm": 0.6342610716819763 + }, + { + "bits_per_byte": 1.7819191248955948, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.2351322174072266, + "step": 2670, + "unclipped_grad_norm": 0.6434658765792847 + }, + { + "bits_per_byte": 1.7670290389665437, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.2248111963272095, + "step": 2680, + "unclipped_grad_norm": 0.6400484442710876 + }, + { + "bits_per_byte": 1.8319156853799694, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.2697871923446655, + "step": 2690, + "unclipped_grad_norm": 0.6114118695259094 + }, + { + "bits_per_byte": 1.83115638197641, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.2692608833312988, + "step": 2700, + "unclipped_grad_norm": 0.6913010478019714 + }, + { + "bits_per_byte": 1.7083645528786837, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.1841480731964111, + "step": 2710, + "unclipped_grad_norm": 0.6150210499763489 + }, + { + "bits_per_byte": 1.8339803371035774, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.2712182998657227, + "step": 2720, + "unclipped_grad_norm": 0.679834246635437 + }, + { + "bits_per_byte": 1.743354595179373, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.2084013223648071, + "step": 2730, + "unclipped_grad_norm": 0.6028276681900024 + }, + { + "bits_per_byte": 1.773320680283058, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.2291722297668457, + "step": 2740, + "unclipped_grad_norm": 0.6363300681114197 + }, + { + "bits_per_byte": 1.7904618471292522, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.241053581237793, + "step": 2750, + "unclipped_grad_norm": 0.6628854870796204 + }, + { + "bits_per_byte": 1.8616392749459376, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.2903900146484375, + "step": 2760, + "unclipped_grad_norm": 0.6769214272499084 + }, + { + "bits_per_byte": 1.781435165716067, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.2347967624664307, + "step": 2770, + "unclipped_grad_norm": 0.718517541885376 + }, + { + "bits_per_byte": 1.7843648901735147, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.2368274927139282, + "step": 2780, + "unclipped_grad_norm": 0.6826419830322266 + }, + { + "bits_per_byte": 1.7681489899889624, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2255874872207642, + "step": 2790, + "unclipped_grad_norm": 0.7190320491790771 + }, + { + "bits_per_byte": 1.7227371430112886, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.19411039352417, + "step": 2800, + "unclipped_grad_norm": 0.6588876843452454 + }, + { + "bits_per_byte": 1.784764749836771, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.2371046543121338, + "step": 2810, + "unclipped_grad_norm": 0.6112179160118103 + }, + { + "bits_per_byte": 1.7925197915294766, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2424800395965576, + "step": 2820, + "unclipped_grad_norm": 0.6516722440719604 + }, + { + "bits_per_byte": 1.7167070873067367, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.1899306774139404, + "step": 2830, + "unclipped_grad_norm": 0.6079142689704895 + }, + { + "bits_per_byte": 1.8028568087595858, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.2496451139450073, + "step": 2840, + "unclipped_grad_norm": 0.6816202402114868 + }, + { + "bits_per_byte": 1.7174362937463952, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.1904361248016357, + "step": 2850, + "unclipped_grad_norm": 0.6232561469078064 + }, + { + "bits_per_byte": 1.698484493551917, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.1772997379302979, + "step": 2860, + "unclipped_grad_norm": 0.6568719744682312 + }, + { + "bits_per_byte": 1.8119818642490646, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.2559701204299927, + "step": 2870, + "unclipped_grad_norm": 0.6315737366676331 + }, + { + "bits_per_byte": 1.7399543261891643, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2060444355010986, + "step": 2880, + "unclipped_grad_norm": 0.6461299061775208 + }, + { + "bits_per_byte": 1.7955126336197913, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.2445545196533203, + "step": 2890, + "unclipped_grad_norm": 0.6371240615844727 + }, + { + "bits_per_byte": 1.6978051620810088, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.1768288612365723, + "step": 2900, + "unclipped_grad_norm": 0.6048023104667664 + }, + { + "bits_per_byte": 1.7388536372236423, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.2052814960479736, + "step": 2910, + "unclipped_grad_norm": 0.6422730684280396 + }, + { + "bits_per_byte": 1.8189935969247413, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.260830283164978, + "step": 2920, + "unclipped_grad_norm": 0.6604676842689514 + }, + { + "bits_per_byte": 1.7516567136844734, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.214155912399292, + "step": 2930, + "unclipped_grad_norm": 0.6407750844955444 + }, + { + "bits_per_byte": 1.7999293200765991, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.247615933418274, + "step": 2940, + "unclipped_grad_norm": 0.6448708176612854 + }, + { + "bits_per_byte": 1.7624513767485284, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2216382026672363, + "step": 2950, + "unclipped_grad_norm": 0.6607219576835632 + }, + { + "bits_per_byte": 1.74244704273077, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.2077722549438477, + "step": 2960, + "unclipped_grad_norm": 0.6431576609611511 + }, + { + "bits_per_byte": 1.7998355895318787, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.2475509643554688, + "step": 2970, + "unclipped_grad_norm": 0.662564218044281 + }, + { + "bits_per_byte": 1.735044565472333, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.202641248703003, + "step": 2980, + "unclipped_grad_norm": 0.6156101822853088 + }, + { + "bits_per_byte": 1.8660346355540385, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.2934366464614868, + "step": 2990, + "unclipped_grad_norm": 0.6657274961471558 + }, + { + "bits_per_byte": 1.8038383137480598, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.2503254413604736, + "step": 3000, + "unclipped_grad_norm": 0.6519805788993835 + }, + { + "bits_per_byte": 1.7641859937651305, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.2228405475616455, + "step": 3010, + "unclipped_grad_norm": 0.6597804427146912 + }, + { + "bits_per_byte": 1.736222646630743, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.2034578323364258, + "step": 3020, + "unclipped_grad_norm": 0.6512488722801208 + }, + { + "bits_per_byte": 1.7697855768945727, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.2267218828201294, + "step": 3030, + "unclipped_grad_norm": 0.7177257537841797 + }, + { + "bits_per_byte": 1.7396571401684735, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.2058384418487549, + "step": 3040, + "unclipped_grad_norm": 0.648729681968689 + }, + { + "bits_per_byte": 1.750393501113886, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.2132803201675415, + "step": 3050, + "unclipped_grad_norm": 0.6819168329238892 + }, + { + "bits_per_byte": 1.7528870775687457, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.2150087356567383, + "step": 3060, + "unclipped_grad_norm": 0.6625254154205322 + }, + { + "bits_per_byte": 1.7217516824218446, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.193427324295044, + "step": 3070, + "unclipped_grad_norm": 0.6461355686187744 + }, + { + "bits_per_byte": 1.7354877647636064, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.2029484510421753, + "step": 3080, + "unclipped_grad_norm": 0.6848534345626831 + }, + { + "bits_per_byte": 1.7102998736488428, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.185489535331726, + "step": 3090, + "unclipped_grad_norm": 0.6524667739868164 + }, + { + "bits_per_byte": 1.8038328103032322, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.250321626663208, + "step": 3100, + "unclipped_grad_norm": 0.6857631802558899 + }, + { + "bits_per_byte": 1.7776011564803822, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.232139229774475, + "step": 3110, + "unclipped_grad_norm": 0.6425225734710693 + }, + { + "bits_per_byte": 1.6614246400480939, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.1516118049621582, + "step": 3120, + "unclipped_grad_norm": 0.6101730465888977 + }, + { + "bits_per_byte": 1.7549701314359962, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.2164525985717773, + "step": 3130, + "unclipped_grad_norm": 0.6334211230278015 + }, + { + "bits_per_byte": 1.7012076668456788, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.179187297821045, + "step": 3140, + "unclipped_grad_norm": 0.6345035433769226 + }, + { + "bits_per_byte": 1.7011780858297303, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.1791667938232422, + "step": 3150, + "unclipped_grad_norm": 0.6423520445823669 + }, + { + "bits_per_byte": 1.7687597003821762, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.226010799407959, + "step": 3160, + "unclipped_grad_norm": 0.62806636095047 + }, + { + "bits_per_byte": 1.8016169858295157, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.2487857341766357, + "step": 3170, + "unclipped_grad_norm": 0.6600594520568848 + }, + { + "bits_per_byte": 1.6798027061192935, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.1643505096435547, + "step": 3180, + "unclipped_grad_norm": 0.6483400464057922 + }, + { + "bits_per_byte": 1.757403685945705, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.218139410018921, + "step": 3190, + "unclipped_grad_norm": 0.6186169385910034 + }, + { + "bits_per_byte": 1.6701751173239936, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.157677173614502, + "step": 3200, + "unclipped_grad_norm": 0.602679967880249 + }, + { + "bits_per_byte": 1.7034881567961195, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.1807680130004883, + "step": 3210, + "unclipped_grad_norm": 0.6303826570510864 + }, + { + "bits_per_byte": 1.701971957746113, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.1797170639038086, + "step": 3220, + "unclipped_grad_norm": 0.5931465029716492 + }, + { + "bits_per_byte": 1.6794496257370721, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.164105772972107, + "step": 3230, + "unclipped_grad_norm": 0.6347519755363464 + }, + { + "bits_per_byte": 1.744860131305026, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.2094448804855347, + "step": 3240, + "unclipped_grad_norm": 0.6984133124351501 + }, + { + "bits_per_byte": 1.738909359602522, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.205320119857788, + "step": 3250, + "unclipped_grad_norm": 0.6158328056335449 + }, + { + "bits_per_byte": 1.8355668770577869, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.2723180055618286, + "step": 3260, + "unclipped_grad_norm": 0.6927392482757568 + }, + { + "bits_per_byte": 1.698512182758706, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.1773189306259155, + "step": 3270, + "unclipped_grad_norm": 0.6319116353988647 + }, + { + "bits_per_byte": 1.6225218204576248, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.1246464252471924, + "step": 3280, + "unclipped_grad_norm": 0.6093255877494812 + }, + { + "bits_per_byte": 1.6754737308144259, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.161349892616272, + "step": 3290, + "unclipped_grad_norm": 0.6271039247512817 + }, + { + "bits_per_byte": 1.7556056073309343, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.2168930768966675, + "step": 3300, + "unclipped_grad_norm": 0.651100754737854 + }, + { + "bits_per_byte": 1.7421115045789366, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.207539677619934, + "step": 3310, + "unclipped_grad_norm": 0.6893958449363708 + }, + { + "bits_per_byte": 1.749813059667224, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.2128779888153076, + "step": 3320, + "unclipped_grad_norm": 0.642137348651886 + }, + { + "bits_per_byte": 1.6928255764079272, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.173377275466919, + "step": 3330, + "unclipped_grad_norm": 0.6155524253845215 + }, + { + "bits_per_byte": 1.7121750004912, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.1867892742156982, + "step": 3340, + "unclipped_grad_norm": 0.6620399355888367 + }, + { + "bits_per_byte": 1.7088519517112288, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.184485912322998, + "step": 3350, + "unclipped_grad_norm": 0.6751689910888672 + }, + { + "bits_per_byte": 1.706591411748288, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.1829190254211426, + "step": 3360, + "unclipped_grad_norm": 0.6241607069969177 + }, + { + "bits_per_byte": 1.6499554610273548, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.1436619758605957, + "step": 3370, + "unclipped_grad_norm": 0.616858720779419 + }, + { + "bits_per_byte": 1.7488181400319827, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.2121883630752563, + "step": 3380, + "unclipped_grad_norm": 0.6998610496520996 + }, + { + "bits_per_byte": 1.7225412547719559, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.1939746141433716, + "step": 3390, + "unclipped_grad_norm": 0.6383993029594421 + }, + { + "bits_per_byte": 1.7310700464108935, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.1998863220214844, + "step": 3400, + "unclipped_grad_norm": 0.6605661511421204 + }, + { + "bits_per_byte": 1.7513547121495583, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.2139465808868408, + "step": 3410, + "unclipped_grad_norm": 0.6599247455596924 + }, + { + "bits_per_byte": 1.7249777329867293, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.1956634521484375, + "step": 3420, + "unclipped_grad_norm": 0.6639347672462463 + }, + { + "bits_per_byte": 1.65335899768793, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1460211277008057, + "step": 3430, + "unclipped_grad_norm": 0.5962919592857361 + }, + { + "bits_per_byte": 1.6667925625468238, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.1553325653076172, + "step": 3440, + "unclipped_grad_norm": 0.6136701107025146 + }, + { + "bits_per_byte": 1.6700273842269024, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.1575747728347778, + "step": 3450, + "unclipped_grad_norm": 0.6937346458435059 + }, + { + "bits_per_byte": 1.6672555398429465, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.155653476715088, + "step": 3460, + "unclipped_grad_norm": 0.6436060667037964 + }, + { + "bits_per_byte": 1.7216875328930727, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.1933828592300415, + "step": 3470, + "unclipped_grad_norm": 0.6449083089828491 + }, + { + "bits_per_byte": 1.766103600322251, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.2241697311401367, + "step": 3480, + "unclipped_grad_norm": 0.6719608902931213 + }, + { + "bits_per_byte": 1.7067905676579873, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.1830570697784424, + "step": 3490, + "unclipped_grad_norm": 0.6102010607719421 + }, + { + "bits_per_byte": 1.6611136954153338, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1513962745666504, + "step": 3500, + "unclipped_grad_norm": 0.6064846515655518 + }, + { + "bits_per_byte": 1.669534137984228, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.1572328805923462, + "step": 3510, + "unclipped_grad_norm": 0.617793619632721 + }, + { + "bits_per_byte": 1.7151011133129799, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.1888175010681152, + "step": 3520, + "unclipped_grad_norm": 0.6516768932342529 + }, + { + "bits_per_byte": 1.6797051919562545, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.1642829179763794, + "step": 3530, + "unclipped_grad_norm": 0.6367366909980774 + }, + { + "bits_per_byte": 1.6748155532095739, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.1608936786651611, + "step": 3540, + "unclipped_grad_norm": 0.6410197615623474 + }, + { + "bits_per_byte": 1.7350614197721175, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.202652931213379, + "step": 3550, + "unclipped_grad_norm": 0.6610242128372192 + }, + { + "bits_per_byte": 1.653766252605173, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.146303415298462, + "step": 3560, + "unclipped_grad_norm": 0.6675723791122437 + }, + { + "bits_per_byte": 1.698406757393727, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.177245855331421, + "step": 3570, + "unclipped_grad_norm": 0.6639978289604187 + }, + { + "bits_per_byte": 1.7344293835301967, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.2022148370742798, + "step": 3580, + "unclipped_grad_norm": 0.6603791117668152 + }, + { + "bits_per_byte": 1.6983835397358606, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.1772297620773315, + "step": 3590, + "unclipped_grad_norm": 0.6535188555717468 + }, + { + "bits_per_byte": 1.7488331025226076, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.2121987342834473, + "step": 3600, + "unclipped_grad_norm": 0.6535741686820984 + }, + { + "bits_per_byte": 1.7188835277534056, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.1914392709732056, + "step": 3610, + "unclipped_grad_norm": 0.6745624542236328 + }, + { + "bits_per_byte": 1.6643256434028477, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.1536226272583008, + "step": 3620, + "unclipped_grad_norm": 0.614556610584259 + }, + { + "bits_per_byte": 1.6653341496675071, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.1543216705322266, + "step": 3630, + "unclipped_grad_norm": 0.6602646708488464 + }, + { + "bits_per_byte": 1.6109921035437822, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.116654634475708, + "step": 3640, + "unclipped_grad_norm": 0.607775866985321 + }, + { + "bits_per_byte": 1.68876179835069, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.1705604791641235, + "step": 3650, + "unclipped_grad_norm": 0.6529019474983215 + }, + { + "bits_per_byte": 1.690797385006302, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.1719714403152466, + "step": 3660, + "unclipped_grad_norm": 0.6075798273086548 + }, + { + "bits_per_byte": 1.714149533305756, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.1881579160690308, + "step": 3670, + "unclipped_grad_norm": 0.6986024379730225 + }, + { + "bits_per_byte": 1.6661156388330278, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.1548633575439453, + "step": 3680, + "unclipped_grad_norm": 0.6382054090499878 + }, + { + "bits_per_byte": 1.7279294712234876, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.1977094411849976, + "step": 3690, + "unclipped_grad_norm": 0.6791408061981201 + }, + { + "bits_per_byte": 1.6309630729272733, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.1304974555969238, + "step": 3700, + "unclipped_grad_norm": 0.6656120419502258 + }, + { + "bits_per_byte": 1.6975607747341328, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.1766594648361206, + "step": 3710, + "unclipped_grad_norm": 0.6638663411140442 + }, + { + "bits_per_byte": 1.7115804564671673, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.1863771677017212, + "step": 3720, + "unclipped_grad_norm": 0.686505913734436 + }, + { + "bits_per_byte": 1.6779865693261824, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.1630916595458984, + "step": 3730, + "unclipped_grad_norm": 0.6658623814582825 + }, + { + "bits_per_byte": 1.6618549406405525, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.1519100666046143, + "step": 3740, + "unclipped_grad_norm": 0.6103088855743408 + }, + { + "bits_per_byte": 1.6787431210073278, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1636160612106323, + "step": 3750, + "unclipped_grad_norm": 0.6425120830535889 + }, + { + "bits_per_byte": 1.720990487209126, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.1928997039794922, + "step": 3760, + "unclipped_grad_norm": 0.7003195285797119 + }, + { + "bits_per_byte": 1.7130969994824756, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.18742835521698, + "step": 3770, + "unclipped_grad_norm": 0.6268595457077026 + }, + { + "bits_per_byte": 1.753244113551937, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.2152562141418457, + "step": 3780, + "unclipped_grad_norm": 0.6847130656242371 + }, + { + "bits_per_byte": 1.6579560939454927, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.149207592010498, + "step": 3790, + "unclipped_grad_norm": 0.6192958950996399 + }, + { + "bits_per_byte": 1.6900098764480012, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.1714255809783936, + "step": 3800, + "unclipped_grad_norm": 0.6738025546073914 + }, + { + "bits_per_byte": 1.6220395811046056, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.124312162399292, + "step": 3810, + "unclipped_grad_norm": 0.6109552979469299 + }, + { + "bits_per_byte": 1.6345922508257802, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1330130100250244, + "step": 3820, + "unclipped_grad_norm": 0.6535995006561279 + }, + { + "bits_per_byte": 1.696745748951694, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.1760945320129395, + "step": 3830, + "unclipped_grad_norm": 0.6457944512367249 + }, + { + "bits_per_byte": 1.6471569593325153, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.1417222023010254, + "step": 3840, + "unclipped_grad_norm": 0.6749499440193176 + }, + { + "bits_per_byte": 1.6019778048814588, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1104063987731934, + "step": 3850, + "unclipped_grad_norm": 0.6017903089523315 + }, + { + "bits_per_byte": 1.5923278583415466, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.103717565536499, + "step": 3860, + "unclipped_grad_norm": 0.6315345764160156 + }, + { + "bits_per_byte": 1.5646482825810828, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.084531545639038, + "step": 3870, + "unclipped_grad_norm": 0.6246797442436218 + }, + { + "bits_per_byte": 1.6670811494349715, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1555325984954834, + "step": 3880, + "unclipped_grad_norm": 0.6677500009536743 + }, + { + "bits_per_byte": 1.645123780434015, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.140312910079956, + "step": 3890, + "unclipped_grad_norm": 0.6457604169845581 + }, + { + "bits_per_byte": 1.6348758502170528, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.1332095861434937, + "step": 3900, + "unclipped_grad_norm": 0.6628963351249695 + }, + { + "bits_per_byte": 1.6681883737412264, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1563000679016113, + "step": 3910, + "unclipped_grad_norm": 0.6977989077568054 + }, + { + "bits_per_byte": 1.6973328977217397, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.1765015125274658, + "step": 3920, + "unclipped_grad_norm": 0.7063660621643066 + }, + { + "bits_per_byte": 1.6746361753047239, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.1607693433761597, + "step": 3930, + "unclipped_grad_norm": 0.6622145771980286 + }, + { + "bits_per_byte": 1.608798292849376, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.1151340007781982, + "step": 3940, + "unclipped_grad_norm": 0.6348296999931335 + }, + { + "bits_per_byte": 1.7196904703512539, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1919986009597778, + "step": 3950, + "unclipped_grad_norm": 0.7217053174972534 + }, + { + "bits_per_byte": 1.6881366414148036, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.1701271533966064, + "step": 3960, + "unclipped_grad_norm": 0.6808131337165833 + }, + { + "bits_per_byte": 1.6454921672721632, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1405682563781738, + "step": 3970, + "unclipped_grad_norm": 0.6680302023887634 + }, + { + "bits_per_byte": 1.638767989588729, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.1359074115753174, + "step": 3980, + "unclipped_grad_norm": 0.6691642999649048 + }, + { + "bits_per_byte": 1.668301538325494, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1563785076141357, + "step": 3990, + "unclipped_grad_norm": 0.6827875375747681 + }, + { + "bits_per_byte": 1.5894203196460601, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.1017022132873535, + "step": 4000, + "unclipped_grad_norm": 0.6496227383613586 + }, + { + "bits_per_byte": 1.6673331040184856, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.1557072401046753, + "step": 4010, + "unclipped_grad_norm": 0.6678752303123474 + }, + { + "bits_per_byte": 1.6876217253581203, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.1697702407836914, + "step": 4020, + "unclipped_grad_norm": 0.688493549823761 + }, + { + "bits_per_byte": 1.6368569183723416, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.134582757949829, + "step": 4030, + "unclipped_grad_norm": 0.7154926061630249 + }, + { + "bits_per_byte": 1.615401222763952, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1197108030319214, + "step": 4040, + "unclipped_grad_norm": 0.660170316696167 + }, + { + "bits_per_byte": 1.6411334389686962, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.1375470161437988, + "step": 4050, + "unclipped_grad_norm": 0.678307294845581 + }, + { + "bits_per_byte": 1.6164319147905728, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1204252243041992, + "step": 4060, + "unclipped_grad_norm": 0.66500324010849 + }, + { + "bits_per_byte": 1.7425645068813091, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.2078536748886108, + "step": 4070, + "unclipped_grad_norm": 0.6793113946914673 + }, + { + "bits_per_byte": 1.5848681108603724, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.0985468626022339, + "step": 4080, + "unclipped_grad_norm": 0.6559226512908936 + }, + { + "bits_per_byte": 1.6144109466602838, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.119024395942688, + "step": 4090, + "unclipped_grad_norm": 0.6595537662506104 + }, + { + "bits_per_byte": 1.7310557718508717, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.1998764276504517, + "step": 4100, + "unclipped_grad_norm": 0.6686861515045166 + }, + { + "bits_per_byte": 1.6299851795744673, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.129819631576538, + "step": 4110, + "unclipped_grad_norm": 0.6534212827682495 + }, + { + "bits_per_byte": 1.633285010696572, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.132106900215149, + "step": 4120, + "unclipped_grad_norm": 0.6697815656661987 + }, + { + "bits_per_byte": 1.6929194789352981, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.1734423637390137, + "step": 4130, + "unclipped_grad_norm": 0.6689978837966919 + }, + { + "bits_per_byte": 1.674332625925951, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.1605589389801025, + "step": 4140, + "unclipped_grad_norm": 0.6499730944633484 + }, + { + "bits_per_byte": 1.6289295500634713, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1290879249572754, + "step": 4150, + "unclipped_grad_norm": 0.6851745247840881 + }, + { + "bits_per_byte": 1.642519103186698, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.1385074853897095, + "step": 4160, + "unclipped_grad_norm": 0.7024235129356384 + }, + { + "bits_per_byte": 1.6592466517575672, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.150102138519287, + "step": 4170, + "unclipped_grad_norm": 0.6990675330162048 + }, + { + "bits_per_byte": 1.6290158853542045, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.1291477680206299, + "step": 4180, + "unclipped_grad_norm": 0.6769554615020752 + }, + { + "bits_per_byte": 1.6624596316409863, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.1523292064666748, + "step": 4190, + "unclipped_grad_norm": 0.6870110630989075 + }, + { + "bits_per_byte": 1.643196026900494, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1389766931533813, + "step": 4200, + "unclipped_grad_norm": 0.718636691570282 + }, + { + "bits_per_byte": 1.6763578936225114, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.1619627475738525, + "step": 4210, + "unclipped_grad_norm": 0.665073037147522 + }, + { + "bits_per_byte": 1.6464858830288485, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.1412570476531982, + "step": 4220, + "unclipped_grad_norm": 0.6534920334815979 + }, + { + "bits_per_byte": 1.6566755111271683, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.148319959640503, + "step": 4230, + "unclipped_grad_norm": 0.6859180927276611 + }, + { + "bits_per_byte": 1.6780268132664842, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.1631195545196533, + "step": 4240, + "unclipped_grad_norm": 0.64006108045578 + }, + { + "bits_per_byte": 1.608748417880626, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1150994300842285, + "step": 4250, + "unclipped_grad_norm": 0.6516591310501099 + }, + { + "bits_per_byte": 1.6356482243020778, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1337449550628662, + "step": 4260, + "unclipped_grad_norm": 0.6675782203674316 + }, + { + "bits_per_byte": 1.5674168592946722, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.0864505767822266, + "step": 4270, + "unclipped_grad_norm": 0.6488245725631714 + }, + { + "bits_per_byte": 1.6384080299004733, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1356579065322876, + "step": 4280, + "unclipped_grad_norm": 0.666085422039032 + }, + { + "bits_per_byte": 1.612649500350147, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.1178034543991089, + "step": 4290, + "unclipped_grad_norm": 0.6461021900177002 + }, + { + "bits_per_byte": 1.5810767533221017, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.095918893814087, + "step": 4300, + "unclipped_grad_norm": 0.6691118478775024 + }, + { + "bits_per_byte": 1.5563184748691936, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.0787577629089355, + "step": 4310, + "unclipped_grad_norm": 0.6182403564453125 + }, + { + "bits_per_byte": 1.6958627900221643, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.1754825115203857, + "step": 4320, + "unclipped_grad_norm": 0.710037350654602 + }, + { + "bits_per_byte": 1.6432519212620242, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.1390154361724854, + "step": 4330, + "unclipped_grad_norm": 0.6684199571609497 + }, + { + "bits_per_byte": 1.593839241877329, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.1047651767730713, + "step": 4340, + "unclipped_grad_norm": 0.7022923231124878 + }, + { + "bits_per_byte": 1.5457828176773378, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.0714550018310547, + "step": 4350, + "unclipped_grad_norm": 0.6899289488792419 + }, + { + "bits_per_byte": 1.6521984587599077, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.145216703414917, + "step": 4360, + "unclipped_grad_norm": 0.719445526599884 + }, + { + "bits_per_byte": 1.6143125725839904, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.118956208229065, + "step": 4370, + "unclipped_grad_norm": 0.6690176725387573 + }, + { + "bits_per_byte": 1.5596614736366647, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.0810749530792236, + "step": 4380, + "unclipped_grad_norm": 0.6416013836860657 + }, + { + "bits_per_byte": 1.7011741302287604, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1791640520095825, + "step": 4390, + "unclipped_grad_norm": 0.6886416673660278 + }, + { + "bits_per_byte": 1.6470823188620407, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.1416704654693604, + "step": 4400, + "unclipped_grad_norm": 0.6624540090560913 + }, + { + "bits_per_byte": 1.6557540280838454, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.1476812362670898, + "step": 4410, + "unclipped_grad_norm": 0.7250106930732727 + }, + { + "bits_per_byte": 1.6743157716261665, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.1605472564697266, + "step": 4420, + "unclipped_grad_norm": 0.70937579870224 + }, + { + "bits_per_byte": 1.6202545731713003, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.1230748891830444, + "step": 4430, + "unclipped_grad_norm": 0.6804994940757751 + }, + { + "bits_per_byte": 1.6200650482900496, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.1229435205459595, + "step": 4440, + "unclipped_grad_norm": 0.6455305814743042 + }, + { + "bits_per_byte": 1.6548672855359967, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.147066593170166, + "step": 4450, + "unclipped_grad_norm": 0.6803700923919678 + }, + { + "bits_per_byte": 1.6333722059005593, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.1321673393249512, + "step": 4460, + "unclipped_grad_norm": 0.7055696249008179 + }, + { + "bits_per_byte": 1.6093336748415121, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1155050992965698, + "step": 4470, + "unclipped_grad_norm": 0.7044621109962463 + }, + { + "bits_per_byte": 1.6521430803463297, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.1451783180236816, + "step": 4480, + "unclipped_grad_norm": 0.717522919178009 + }, + { + "bits_per_byte": 1.6882831706333388, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.1702287197113037, + "step": 4490, + "unclipped_grad_norm": 0.6851462125778198 + }, + { + "bits_per_byte": 1.5972596328376885, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.1071360111236572, + "step": 4500, + "unclipped_grad_norm": 0.6727108955383301 + }, + { + "bits_per_byte": 1.5868809958060706, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.0999420881271362, + "step": 4510, + "unclipped_grad_norm": 0.673721969127655 + }, + { + "bits_per_byte": 1.6634034724289213, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.1529834270477295, + "step": 4520, + "unclipped_grad_norm": 0.7126460075378418 + }, + { + "bits_per_byte": 1.6365516491670602, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.1343711614608765, + "step": 4530, + "unclipped_grad_norm": 0.7139353156089783 + }, + { + "bits_per_byte": 1.6428592848701045, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.138743281364441, + "step": 4540, + "unclipped_grad_norm": 0.6903482675552368 + }, + { + "bits_per_byte": 1.6469521279953376, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.1415802240371704, + "step": 4550, + "unclipped_grad_norm": 0.6733337640762329 + }, + { + "bits_per_byte": 1.6447951215882164, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.1400851011276245, + "step": 4560, + "unclipped_grad_norm": 0.7175610661506653 + }, + { + "bits_per_byte": 1.635643064822552, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.1337413787841797, + "step": 4570, + "unclipped_grad_norm": 0.7035735249519348 + }, + { + "bits_per_byte": 1.6124235151469133, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.1176468133926392, + "step": 4580, + "unclipped_grad_norm": 0.6848859786987305 + }, + { + "bits_per_byte": 1.5835818526145693, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.0976552963256836, + "step": 4590, + "unclipped_grad_norm": 0.6959241628646851 + }, + { + "bits_per_byte": 1.5960186060290624, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.1062757968902588, + "step": 4600, + "unclipped_grad_norm": 0.7097700238227844 + }, + { + "bits_per_byte": 1.6048268694756522, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.1123812198638916, + "step": 4610, + "unclipped_grad_norm": 0.6661962270736694 + }, + { + "bits_per_byte": 1.6559490564099237, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.1478164196014404, + "step": 4620, + "unclipped_grad_norm": 0.6830936670303345 + }, + { + "bits_per_byte": 1.6545855779538834, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.1468713283538818, + "step": 4630, + "unclipped_grad_norm": 0.7361748814582825 + }, + { + "bits_per_byte": 1.6499609644721824, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.1436657905578613, + "step": 4640, + "unclipped_grad_norm": 0.7230168581008911 + }, + { + "bits_per_byte": 1.6797445759833018, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.1643102169036865, + "step": 4650, + "unclipped_grad_norm": 0.7130787372589111 + }, + { + "bits_per_byte": 1.6089793905807346, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.1152595281600952, + "step": 4660, + "unclipped_grad_norm": 0.6770996451377869 + }, + { + "bits_per_byte": 1.6145978918017718, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.1191539764404297, + "step": 4670, + "unclipped_grad_norm": 0.6875596642494202 + }, + { + "bits_per_byte": 1.5872010555193263, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.1001639366149902, + "step": 4680, + "unclipped_grad_norm": 0.7211877703666687 + }, + { + "bits_per_byte": 1.5995029745355431, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.1086909770965576, + "step": 4690, + "unclipped_grad_norm": 0.6830824017524719 + }, + { + "bits_per_byte": 1.615236979332378, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.1195969581604004, + "step": 4700, + "unclipped_grad_norm": 0.6662686467170715 + }, + { + "bits_per_byte": 1.5971631505705546, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.1070691347122192, + "step": 4710, + "unclipped_grad_norm": 0.6915621161460876 + }, + { + "bits_per_byte": 1.6281631953712266, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.128556728363037, + "step": 4720, + "unclipped_grad_norm": 0.7106578946113586 + }, + { + "bits_per_byte": 1.6613228263187831, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.1515412330627441, + "step": 4730, + "unclipped_grad_norm": 0.7514477372169495 + }, + { + "bits_per_byte": 1.6096174462154356, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.1157017946243286, + "step": 4740, + "unclipped_grad_norm": 0.7302504181861877 + }, + { + "bits_per_byte": 1.575867742792769, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.0923082828521729, + "step": 4750, + "unclipped_grad_norm": 0.7527251839637756 + }, + { + "bits_per_byte": 1.5433951825354095, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.0698000192642212, + "step": 4760, + "unclipped_grad_norm": 0.6936410069465637 + }, + { + "bits_per_byte": 1.5583320477454954, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.080153465270996, + "step": 4770, + "unclipped_grad_norm": 0.6624149084091187 + }, + { + "bits_per_byte": 1.5995043503967499, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.108691930770874, + "step": 4780, + "unclipped_grad_norm": 0.7143124341964722 + }, + { + "bits_per_byte": 1.5959776741581573, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.1062474250793457, + "step": 4790, + "unclipped_grad_norm": 0.7112782001495361 + }, + { + "bits_per_byte": 1.616335260540788, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.1203582286834717, + "step": 4800, + "unclipped_grad_norm": 0.7171105742454529 + }, + { + "bits_per_byte": 1.5889982742208426, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.101409673690796, + "step": 4810, + "unclipped_grad_norm": 0.6826947331428528 + }, + { + "bits_per_byte": 1.5799485471324417, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.0951368808746338, + "step": 4820, + "unclipped_grad_norm": 0.6934098601341248 + }, + { + "bits_per_byte": 1.576572183730703, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.0927965641021729, + "step": 4830, + "unclipped_grad_norm": 0.642882764339447 + }, + { + "bits_per_byte": 1.5256281708227253, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.0574848651885986, + "step": 4840, + "unclipped_grad_norm": 0.6699715256690979 + }, + { + "bits_per_byte": 1.5907193046080268, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.1026026010513306, + "step": 4850, + "unclipped_grad_norm": 0.6837528347969055 + }, + { + "bits_per_byte": 1.5800562082718819, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.0952115058898926, + "step": 4860, + "unclipped_grad_norm": 0.6885666847229004 + }, + { + "bits_per_byte": 1.5946076603613841, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.1052978038787842, + "step": 4870, + "unclipped_grad_norm": 0.6755294799804688 + }, + { + "bits_per_byte": 1.5533662206844825, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.0767114162445068, + "step": 4880, + "unclipped_grad_norm": 0.6481236815452576 + }, + { + "bits_per_byte": 1.5291947470363183, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.0599570274353027, + "step": 4890, + "unclipped_grad_norm": 0.6977331638336182 + }, + { + "bits_per_byte": 1.5147387453180448, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.0499368906021118, + "step": 4900, + "unclipped_grad_norm": 0.6663004159927368 + }, + { + "bits_per_byte": 1.5877115000270872, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.100517749786377, + "step": 4910, + "unclipped_grad_norm": 0.7221488356590271 + }, + { + "bits_per_byte": 1.5841899832680202, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.0980768203735352, + "step": 4920, + "unclipped_grad_norm": 0.7138182520866394 + }, + { + "bits_per_byte": 1.565747595685398, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.0852935314178467, + "step": 4930, + "unclipped_grad_norm": 0.7141739726066589 + }, + { + "bits_per_byte": 1.658866914064462, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.149838924407959, + "step": 4940, + "unclipped_grad_norm": 0.7705830931663513 + }, + { + "bits_per_byte": 1.5082266222431246, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.0454230308532715, + "step": 4950, + "unclipped_grad_norm": 0.6872978806495667 + }, + { + "bits_per_byte": 1.594800624895652, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.1054315567016602, + "step": 4960, + "unclipped_grad_norm": 0.7526467442512512 + }, + { + "bits_per_byte": 1.672291879790813, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.159144401550293, + "step": 4970, + "unclipped_grad_norm": 0.7568042874336243 + }, + { + "bits_per_byte": 1.5354246465811727, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.0642752647399902, + "step": 4980, + "unclipped_grad_norm": 0.7262822389602661 + }, + { + "bits_per_byte": 1.5933059236770035, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.1043955087661743, + "step": 4990, + "unclipped_grad_norm": 0.7073372602462769 + }, + { + "bits_per_byte": 1.6293450601479558, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.12937593460083, + "step": 5000, + "unclipped_grad_norm": 0.741706132888794 + }, + { + "bits_per_byte": 1.5896232591740782, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.1018428802490234, + "step": 5010, + "unclipped_grad_norm": 0.6887884140014648 + }, + { + "bits_per_byte": 1.5749681015461057, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.0916846990585327, + "step": 5020, + "unclipped_grad_norm": 0.6972888112068176 + }, + { + "bits_per_byte": 1.5606562212892552, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.0817644596099854, + "step": 5030, + "unclipped_grad_norm": 0.7181957364082336 + }, + { + "bits_per_byte": 1.5631546132583394, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.0834962129592896, + "step": 5040, + "unclipped_grad_norm": 0.7281494736671448 + }, + { + "bits_per_byte": 1.5968547856775575, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.1068553924560547, + "step": 5050, + "unclipped_grad_norm": 0.7250375747680664 + }, + { + "bits_per_byte": 1.544679376989402, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.0706901550292969, + "step": 5060, + "unclipped_grad_norm": 0.7048619389533997 + }, + { + "bits_per_byte": 1.6120329425468038, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.1173760890960693, + "step": 5070, + "unclipped_grad_norm": 0.7317821383476257 + }, + { + "bits_per_byte": 1.6327541002533585, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.1317389011383057, + "step": 5080, + "unclipped_grad_norm": 0.7896588444709778 + }, + { + "bits_per_byte": 1.6237800455313371, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.125518560409546, + "step": 5090, + "unclipped_grad_norm": 0.7764332294464111 + }, + { + "bits_per_byte": 1.5469977031230326, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.0722970962524414, + "step": 5100, + "unclipped_grad_norm": 0.729245662689209 + }, + { + "bits_per_byte": 1.5710204117782007, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.088948369026184, + "step": 5110, + "unclipped_grad_norm": 0.7094104886054993 + }, + { + "bits_per_byte": 1.5501618399336068, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0744903087615967, + "step": 5120, + "unclipped_grad_norm": 0.7421026229858398 + }, + { + "bits_per_byte": 1.59801738839739, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.107661247253418, + "step": 5130, + "unclipped_grad_norm": 0.7430219054222107 + }, + { + "bits_per_byte": 1.545712476773135, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.0714062452316284, + "step": 5140, + "unclipped_grad_norm": 0.7433828711509705 + }, + { + "bits_per_byte": 1.5456263134650527, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.0713465213775635, + "step": 5150, + "unclipped_grad_norm": 0.7395268082618713 + }, + { + "bits_per_byte": 1.5712259310459817, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.0890908241271973, + "step": 5160, + "unclipped_grad_norm": 0.7406699657440186 + }, + { + "bits_per_byte": 1.5875770095941124, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.1004245281219482, + "step": 5170, + "unclipped_grad_norm": 0.7371104955673218 + }, + { + "bits_per_byte": 1.6181173447690282, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.1215934753417969, + "step": 5180, + "unclipped_grad_norm": 0.771541178226471 + }, + { + "bits_per_byte": 1.5541208805564686, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.0772345066070557, + "step": 5190, + "unclipped_grad_norm": 0.692523181438446 + }, + { + "bits_per_byte": 1.5258262948365193, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.0576221942901611, + "step": 5200, + "unclipped_grad_norm": 0.7665137052536011 + }, + { + "bits_per_byte": 1.5979719849775622, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.1076297760009766, + "step": 5210, + "unclipped_grad_norm": 0.7058459520339966 + }, + { + "bits_per_byte": 1.564745796744122, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.0845991373062134, + "step": 5220, + "unclipped_grad_norm": 0.7257795929908752 + }, + { + "bits_per_byte": 1.605369646721775, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.1127574443817139, + "step": 5230, + "unclipped_grad_norm": 0.7407286763191223 + }, + { + "bits_per_byte": 1.572461798375082, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0899474620819092, + "step": 5240, + "unclipped_grad_norm": 0.7420238852500916 + }, + { + "bits_per_byte": 1.526845807990834, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0583288669586182, + "step": 5250, + "unclipped_grad_norm": 0.6995547413825989 + }, + { + "bits_per_byte": 1.507830030250235, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.0451481342315674, + "step": 5260, + "unclipped_grad_norm": 0.7092004418373108 + }, + { + "bits_per_byte": 1.6009154680470792, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.1096700429916382, + "step": 5270, + "unclipped_grad_norm": 0.7264013886451721 + }, + { + "bits_per_byte": 1.5878023068667426, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.1005806922912598, + "step": 5280, + "unclipped_grad_norm": 0.731412947177887 + }, + { + "bits_per_byte": 1.5747574227987986, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.091538667678833, + "step": 5290, + "unclipped_grad_norm": 0.7513731718063354 + }, + { + "bits_per_byte": 1.5056150656897729, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.0436128377914429, + "step": 5300, + "unclipped_grad_norm": 0.7147036790847778 + }, + { + "bits_per_byte": 1.5807904022084152, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.0957204103469849, + "step": 5310, + "unclipped_grad_norm": 0.7368376851081848 + }, + { + "bits_per_byte": 1.5437480909349799, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0700446367263794, + "step": 5320, + "unclipped_grad_norm": 0.7258135080337524 + }, + { + "bits_per_byte": 1.634141312315218, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.1327004432678223, + "step": 5330, + "unclipped_grad_norm": 0.7758851051330566 + }, + { + "bits_per_byte": 1.5389238055956274, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.0667006969451904, + "step": 5340, + "unclipped_grad_norm": 0.7344443202018738 + }, + { + "bits_per_byte": 1.5814767849680087, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.096196174621582, + "step": 5350, + "unclipped_grad_norm": 0.7485818862915039 + }, + { + "bits_per_byte": 1.4998032559691659, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.0395843982696533, + "step": 5360, + "unclipped_grad_norm": 0.6871463060379028 + }, + { + "bits_per_byte": 1.5294386184352418, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.0601260662078857, + "step": 5370, + "unclipped_grad_norm": 0.7182185053825378 + }, + { + "bits_per_byte": 1.5906888636788241, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.10258150100708, + "step": 5380, + "unclipped_grad_norm": 0.7666484713554382 + }, + { + "bits_per_byte": 1.5091233397847232, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.0460445880889893, + "step": 5390, + "unclipped_grad_norm": 0.7180011868476868 + }, + { + "bits_per_byte": 1.5795863516697246, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.0948858261108398, + "step": 5400, + "unclipped_grad_norm": 0.7400155067443848 + }, + { + "bits_per_byte": 1.536605135496695, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.0650935173034668, + "step": 5410, + "unclipped_grad_norm": 0.7344176173210144 + }, + { + "bits_per_byte": 1.4930890532794818, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.0349304676055908, + "step": 5420, + "unclipped_grad_norm": 0.7356988787651062 + }, + { + "bits_per_byte": 1.5627624928143722, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.0832244157791138, + "step": 5430, + "unclipped_grad_norm": 0.7437497973442078 + }, + { + "bits_per_byte": 1.5292869297371807, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.060020923614502, + "step": 5440, + "unclipped_grad_norm": 0.7088740468025208 + }, + { + "bits_per_byte": 1.5469412928135498, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.0722579956054688, + "step": 5450, + "unclipped_grad_norm": 0.7210080027580261 + }, + { + "bits_per_byte": 1.5345666251360182, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.0636805295944214, + "step": 5460, + "unclipped_grad_norm": 0.7137321829795837 + }, + { + "bits_per_byte": 1.5422895060730124, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0690336227416992, + "step": 5470, + "unclipped_grad_norm": 0.7130851149559021 + }, + { + "bits_per_byte": 1.5248621600957823, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.0569539070129395, + "step": 5480, + "unclipped_grad_norm": 0.7526648044586182 + }, + { + "bits_per_byte": 1.5883399246333398, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.1009533405303955, + "step": 5490, + "unclipped_grad_norm": 0.8593043684959412 + }, + { + "bits_per_byte": 1.5656519733315182, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.0852272510528564, + "step": 5500, + "unclipped_grad_norm": 0.7626182436943054 + }, + { + "bits_per_byte": 1.523241739559353, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.055830717086792, + "step": 5510, + "unclipped_grad_norm": 0.7108238339424133 + }, + { + "bits_per_byte": 1.4944408369152635, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.03586745262146, + "step": 5520, + "unclipped_grad_norm": 0.7117043137550354 + }, + { + "bits_per_byte": 1.4972702954872585, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.0378286838531494, + "step": 5530, + "unclipped_grad_norm": 0.7184089422225952 + }, + { + "bits_per_byte": 1.616333884679581, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.1203572750091553, + "step": 5540, + "unclipped_grad_norm": 0.759319007396698 + }, + { + "bits_per_byte": 1.5693796972889695, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.0878111124038696, + "step": 5550, + "unclipped_grad_norm": 0.7310317754745483 + }, + { + "bits_per_byte": 1.59896019728942, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.1083147525787354, + "step": 5560, + "unclipped_grad_norm": 0.7927998304367065 + }, + { + "bits_per_byte": 1.5313543331832025, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.061453938484192, + "step": 5570, + "unclipped_grad_norm": 0.7150455713272095 + }, + { + "bits_per_byte": 1.5681379825497401, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.086950421333313, + "step": 5580, + "unclipped_grad_norm": 0.7254865169525146 + }, + { + "bits_per_byte": 1.5639214838985365, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.0840277671813965, + "step": 5590, + "unclipped_grad_norm": 0.7706523537635803 + }, + { + "bits_per_byte": 1.5591563605910808, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.080724835395813, + "step": 5600, + "unclipped_grad_norm": 0.7836406230926514 + }, + { + "bits_per_byte": 1.5001788660786504, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.0398447513580322, + "step": 5610, + "unclipped_grad_norm": 0.7128682732582092 + }, + { + "bits_per_byte": 1.5433616459184913, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.0697767734527588, + "step": 5620, + "unclipped_grad_norm": 0.760458767414093 + }, + { + "bits_per_byte": 1.484657431838282, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0290861129760742, + "step": 5630, + "unclipped_grad_norm": 0.6990724205970764 + }, + { + "bits_per_byte": 1.4650980169383054, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.0155285596847534, + "step": 5640, + "unclipped_grad_norm": 0.7067834138870239 + }, + { + "bits_per_byte": 1.5401789349816242, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.067570686340332, + "step": 5650, + "unclipped_grad_norm": 0.759252667427063 + }, + { + "bits_per_byte": 1.479384443762828, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.0254311561584473, + "step": 5660, + "unclipped_grad_norm": 0.7858567237854004 + }, + { + "bits_per_byte": 1.5478569284467434, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.072892665863037, + "step": 5670, + "unclipped_grad_norm": 0.7649872303009033 + }, + { + "bits_per_byte": 1.5613524070599478, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.082247018814087, + "step": 5680, + "unclipped_grad_norm": 0.8485968112945557 + }, + { + "bits_per_byte": 1.5772632100218698, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.093275547027588, + "step": 5690, + "unclipped_grad_norm": 0.7517440915107727 + }, + { + "bits_per_byte": 1.5855020389114527, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.098986268043518, + "step": 5700, + "unclipped_grad_norm": 0.755023717880249 + }, + { + "bits_per_byte": 1.5683263035524349, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.087080955505371, + "step": 5710, + "unclipped_grad_norm": 0.8086130023002625 + }, + { + "bits_per_byte": 1.5789663542133643, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0944560766220093, + "step": 5720, + "unclipped_grad_norm": 0.7476592659950256 + }, + { + "bits_per_byte": 1.5294810981500049, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.0601555109024048, + "step": 5730, + "unclipped_grad_norm": 0.7624625563621521 + }, + { + "bits_per_byte": 1.5928070020068505, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.1040496826171875, + "step": 5740, + "unclipped_grad_norm": 0.7598878145217896 + }, + { + "bits_per_byte": 1.4759626769412617, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.023059368133545, + "step": 5750, + "unclipped_grad_norm": 0.7400259971618652 + }, + { + "bits_per_byte": 1.5034964114137939, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.0421442985534668, + "step": 5760, + "unclipped_grad_norm": 0.7557976245880127 + }, + { + "bits_per_byte": 1.5122719981567196, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.048227071762085, + "step": 5770, + "unclipped_grad_norm": 0.7882905006408691 + }, + { + "bits_per_byte": 1.5950058001981315, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.1055737733840942, + "step": 5780, + "unclipped_grad_norm": 0.7740811705589294 + }, + { + "bits_per_byte": 1.447340292288768, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 1.0032198429107666, + "step": 5790, + "unclipped_grad_norm": 0.7437180876731873 + }, + { + "bits_per_byte": 1.5529183778616358, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.0764009952545166, + "step": 5800, + "unclipped_grad_norm": 0.761067807674408 + }, + { + "bits_per_byte": 1.4799820834745765, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.0258454084396362, + "step": 5810, + "unclipped_grad_norm": 0.7384158372879028 + }, + { + "bits_per_byte": 1.4757451188879203, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.0229085683822632, + "step": 5820, + "unclipped_grad_norm": 0.7456022500991821 + }, + { + "bits_per_byte": 1.5308984071757652, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.0611379146575928, + "step": 5830, + "unclipped_grad_norm": 0.7736859321594238 + }, + { + "bits_per_byte": 1.5101812050701804, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.0467778444290161, + "step": 5840, + "unclipped_grad_norm": 0.7637408375740051 + }, + { + "bits_per_byte": 1.4801612893967755, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.0259696245193481, + "step": 5850, + "unclipped_grad_norm": 0.7267042994499207 + }, + { + "bits_per_byte": 1.5046289171697256, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.0429292917251587, + "step": 5860, + "unclipped_grad_norm": 0.7446306347846985 + }, + { + "bits_per_byte": 1.5555887524815826, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0782519578933716, + "step": 5870, + "unclipped_grad_norm": 0.7623310685157776 + }, + { + "bits_per_byte": 1.5003634034630262, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.0399726629257202, + "step": 5880, + "unclipped_grad_norm": 0.7461996674537659 + }, + { + "bits_per_byte": 1.549820454371644, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.0742536783218384, + "step": 5890, + "unclipped_grad_norm": 0.8418232798576355 + }, + { + "bits_per_byte": 1.5716478044885482, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.0893832445144653, + "step": 5900, + "unclipped_grad_norm": 0.8174726366996765 + }, + { + "bits_per_byte": 1.5065370646810483, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.0442519187927246, + "step": 5910, + "unclipped_grad_norm": 0.7397674322128296 + }, + { + "bits_per_byte": 1.4784677762337293, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.0247957706451416, + "step": 5920, + "unclipped_grad_norm": 0.7461287379264832 + }, + { + "bits_per_byte": 1.593945871120864, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.1048390865325928, + "step": 5930, + "unclipped_grad_norm": 0.8132187724113464 + }, + { + "bits_per_byte": 1.5228630337621532, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.0555682182312012, + "step": 5940, + "unclipped_grad_norm": 0.7938750386238098 + }, + { + "bits_per_byte": 1.5349143740560627, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.063921570777893, + "step": 5950, + "unclipped_grad_norm": 0.7796859741210938 + }, + { + "bits_per_byte": 1.4863856854968023, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.03028404712677, + "step": 5960, + "unclipped_grad_norm": 0.7745685577392578 + }, + { + "bits_per_byte": 1.5014804307803802, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.0407469272613525, + "step": 5970, + "unclipped_grad_norm": 0.8056560754776001 + }, + { + "bits_per_byte": 1.4897254165939071, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.0325989723205566, + "step": 5980, + "unclipped_grad_norm": 0.7700846791267395 + }, + { + "bits_per_byte": 1.4925400846579278, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0345499515533447, + "step": 5990, + "unclipped_grad_norm": 0.7688166499137878 + }, + { + "bits_per_byte": 1.5595305948393583, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.0809842348098755, + "step": 6000, + "unclipped_grad_norm": 0.7695974707603455 + }, + { + "bits_per_byte": 1.542890757420429, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0694503784179688, + "step": 6010, + "unclipped_grad_norm": 0.8104206919670105 + }, + { + "bits_per_byte": 1.5033098102376077, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0420149564743042, + "step": 6020, + "unclipped_grad_norm": 0.7977633476257324 + }, + { + "bits_per_byte": 1.488771428829571, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.0319377183914185, + "step": 6030, + "unclipped_grad_norm": 0.757161021232605 + }, + { + "bits_per_byte": 1.5226195063285313, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.0553994178771973, + "step": 6040, + "unclipped_grad_norm": 0.7623481750488281 + }, + { + "bits_per_byte": 1.5189906723953261, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.0528841018676758, + "step": 6050, + "unclipped_grad_norm": 0.7636798620223999 + }, + { + "bits_per_byte": 1.507766052704114, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.0451037883758545, + "step": 6060, + "unclipped_grad_norm": 0.789476752281189 + }, + { + "bits_per_byte": 1.409173386461341, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 0.9767645597457886, + "step": 6070, + "unclipped_grad_norm": 0.7549622654914856 + }, + { + "bits_per_byte": 1.48704902258118, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.0307438373565674, + "step": 6080, + "unclipped_grad_norm": 0.7766039967536926 + }, + { + "bits_per_byte": 1.4976606961047172, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.0380992889404297, + "step": 6090, + "unclipped_grad_norm": 0.8215247392654419 + }, + { + "bits_per_byte": 1.5794394784858878, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.0947840213775635, + "step": 6100, + "unclipped_grad_norm": 0.8410324454307556 + }, + { + "bits_per_byte": 1.5224927551148455, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.0553115606307983, + "step": 6110, + "unclipped_grad_norm": 0.8147144913673401 + }, + { + "bits_per_byte": 1.4801377277736072, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0259532928466797, + "step": 6120, + "unclipped_grad_norm": 0.7981242537498474 + }, + { + "bits_per_byte": 1.5445381792330437, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.0705922842025757, + "step": 6130, + "unclipped_grad_norm": 0.7598981261253357 + }, + { + "bits_per_byte": 1.4947529854265795, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.0360838174819946, + "step": 6140, + "unclipped_grad_norm": 0.7671735882759094 + }, + { + "bits_per_byte": 1.476655595041588, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.023539662361145, + "step": 6150, + "unclipped_grad_norm": 0.7648332118988037 + }, + { + "bits_per_byte": 1.5348151400665149, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 1.0638527870178223, + "step": 6160, + "unclipped_grad_norm": 0.7746754288673401 + }, + { + "bits_per_byte": 1.4744416623770311, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 1.0220050811767578, + "step": 6170, + "unclipped_grad_norm": 0.7370507121086121 + }, + { + "bits_per_byte": 1.4699374367509341, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0188829898834229, + "step": 6180, + "unclipped_grad_norm": 0.7552403211593628 + }, + { + "bits_per_byte": 1.4978233916924333, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.0382120609283447, + "step": 6190, + "unclipped_grad_norm": 0.8160126209259033 + }, + { + "bits_per_byte": 1.4769711832059211, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.0237584114074707, + "step": 6200, + "unclipped_grad_norm": 0.7619742751121521 + }, + { + "bits_per_byte": 1.5025568701921304, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.0414930582046509, + "step": 6210, + "unclipped_grad_norm": 0.7690407037734985 + }, + { + "bits_per_byte": 1.5282577855544177, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.05930757522583, + "step": 6220, + "unclipped_grad_norm": 0.8082229495048523 + }, + { + "bits_per_byte": 1.4785606468651953, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.024860143661499, + "step": 6230, + "unclipped_grad_norm": 0.7651602625846863 + }, + { + "bits_per_byte": 1.534109323267374, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.0633635520935059, + "step": 6240, + "unclipped_grad_norm": 0.8183982968330383 + }, + { + "bits_per_byte": 1.477424185508294, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.0240724086761475, + "step": 6250, + "unclipped_grad_norm": 0.8113808631896973 + }, + { + "bits_per_byte": 1.4349362155779386, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 0.994621992111206, + "step": 6260, + "unclipped_grad_norm": 0.7736802697181702 + }, + { + "bits_per_byte": 1.4878031665052136, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.0312665700912476, + "step": 6270, + "unclipped_grad_norm": 0.7538067698478699 + }, + { + "bits_per_byte": 1.5012530697159394, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.0405893325805664, + "step": 6280, + "unclipped_grad_norm": 0.7801132798194885 + }, + { + "bits_per_byte": 1.4987971574616186, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 1.0388870239257812, + "step": 6290, + "unclipped_grad_norm": 0.8576039671897888 + }, + { + "bits_per_byte": 1.465903411692296, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.0160868167877197, + "step": 6300, + "unclipped_grad_norm": 0.7885270714759827 + }, + { + "bits_per_byte": 1.5269913053134638, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.0584297180175781, + "step": 6310, + "unclipped_grad_norm": 0.8002225756645203 + }, + { + "bits_per_byte": 1.4833125275085346, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 1.028153896331787, + "step": 6320, + "unclipped_grad_norm": 0.7923094630241394 + }, + { + "bits_per_byte": 1.4998316331065584, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 1.0396040678024292, + "step": 6330, + "unclipped_grad_norm": 0.7795959115028381 + }, + { + "bits_per_byte": 1.417350903535891, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 0.9824327826499939, + "step": 6340, + "unclipped_grad_norm": 0.7254495024681091 + }, + { + "bits_per_byte": 1.5246251680028935, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.0567896366119385, + "step": 6350, + "unclipped_grad_norm": 0.7819901704788208 + }, + { + "bits_per_byte": 1.5326743000285745, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.0623688697814941, + "step": 6360, + "unclipped_grad_norm": 0.8296079039573669 + }, + { + "bits_per_byte": 1.4638351483330199, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 1.014653205871582, + "step": 6370, + "unclipped_grad_norm": 0.7792860865592957 + }, + { + "bits_per_byte": 1.4443538135565352, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.0011497735977173, + "step": 6380, + "unclipped_grad_norm": 0.7530481219291687 + }, + { + "bits_per_byte": 1.5690776957540544, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0876017808914185, + "step": 6390, + "unclipped_grad_norm": 0.7972773313522339 + }, + { + "bits_per_byte": 1.5404424124027458, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.0677533149719238, + "step": 6400, + "unclipped_grad_norm": 0.8417203426361084 + }, + { + "bits_per_byte": 1.5479594301066575, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.0729637145996094, + "step": 6410, + "unclipped_grad_norm": 0.8366388082504272 + }, + { + "bits_per_byte": 1.472646507467325, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0207607746124268, + "step": 6420, + "unclipped_grad_norm": 0.8455454707145691 + }, + { + "bits_per_byte": 1.513690683043687, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.0492104291915894, + "step": 6430, + "unclipped_grad_norm": 0.7943284511566162 + }, + { + "bits_per_byte": 1.4389344682451972, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 0.9973933696746826, + "step": 6440, + "unclipped_grad_norm": 0.795547604560852 + }, + { + "bits_per_byte": 1.4463345377465222, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0025227069854736, + "step": 6450, + "unclipped_grad_norm": 0.7833832502365112 + }, + { + "bits_per_byte": 1.4825721421965703, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.027640700340271, + "step": 6460, + "unclipped_grad_norm": 0.7794166803359985 + }, + { + "bits_per_byte": 1.4890472890015551, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.032128930091858, + "step": 6470, + "unclipped_grad_norm": 0.8213272094726562 + }, + { + "bits_per_byte": 1.4542797922510822, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 1.0080299377441406, + "step": 6480, + "unclipped_grad_norm": 0.8203483819961548 + }, + { + "bits_per_byte": 1.5288009067658423, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0596840381622314, + "step": 6490, + "unclipped_grad_norm": 0.831902265548706 + }, + { + "bits_per_byte": 1.5548401120023767, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.077733039855957, + "step": 6500, + "unclipped_grad_norm": 0.8567390441894531 + }, + { + "bits_per_byte": 1.5324465949988322, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 1.062211036682129, + "step": 6510, + "unclipped_grad_norm": 0.8450196981430054 + }, + { + "bits_per_byte": 1.5193554475978062, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 1.053136944770813, + "step": 6520, + "unclipped_grad_norm": 0.8159236907958984 + }, + { + "bits_per_byte": 1.4815101493274925, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.026904582977295, + "step": 6530, + "unclipped_grad_norm": 0.7863933444023132 + }, + { + "bits_per_byte": 1.4746263717440578, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.0221331119537354, + "step": 6540, + "unclipped_grad_norm": 0.8281742930412292 + }, + { + "bits_per_byte": 1.506139956740206, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 1.0439766645431519, + "step": 6550, + "unclipped_grad_norm": 0.7693076133728027 + }, + { + "bits_per_byte": 1.523745820709032, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.0561801195144653, + "step": 6560, + "unclipped_grad_norm": 0.8091615438461304 + }, + { + "bits_per_byte": 1.4941292043519, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 1.035651445388794, + "step": 6570, + "unclipped_grad_norm": 0.8045541048049927 + }, + { + "bits_per_byte": 1.4898459764321619, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.0326825380325317, + "step": 6580, + "unclipped_grad_norm": 0.8213433623313904 + }, + { + "bits_per_byte": 1.5230542784699126, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.0557007789611816, + "step": 6590, + "unclipped_grad_norm": 0.8289300799369812 + }, + { + "bits_per_byte": 1.4195692217580451, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9839704036712646, + "step": 6600, + "unclipped_grad_norm": 0.7690811157226562 + }, + { + "bits_per_byte": 1.460909207493891, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.0126250982284546, + "step": 6610, + "unclipped_grad_norm": 0.7890901565551758 + }, + { + "bits_per_byte": 1.5810279102492566, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 1.0958850383758545, + "step": 6620, + "unclipped_grad_norm": 0.8683408498764038 + }, + { + "bits_per_byte": 1.5114910249391516, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.0476857423782349, + "step": 6630, + "unclipped_grad_norm": 0.8106812834739685 + }, + { + "bits_per_byte": 1.482018530043443, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.027256965637207, + "step": 6640, + "unclipped_grad_norm": 0.8215150237083435 + }, + { + "bits_per_byte": 1.4773591760662677, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.0240273475646973, + "step": 6650, + "unclipped_grad_norm": 0.8189572691917419 + }, + { + "bits_per_byte": 1.5120166039201883, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.048050045967102, + "step": 6660, + "unclipped_grad_norm": 0.8496000170707703 + }, + { + "bits_per_byte": 1.4830884341144606, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.0279985666275024, + "step": 6670, + "unclipped_grad_norm": 0.7834759950637817 + }, + { + "bits_per_byte": 1.499375879081772, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.0392881631851196, + "step": 6680, + "unclipped_grad_norm": 0.7986477017402649 + }, + { + "bits_per_byte": 1.4836231281759928, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.0283691883087158, + "step": 6690, + "unclipped_grad_norm": 0.8349180817604065 + }, + { + "bits_per_byte": 1.4949118973959767, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.0361939668655396, + "step": 6700, + "unclipped_grad_norm": 0.8228501081466675 + }, + { + "bits_per_byte": 1.514613197982915, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.0498498678207397, + "step": 6710, + "unclipped_grad_norm": 0.8309299945831299 + }, + { + "bits_per_byte": 1.5793448880279133, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.0947184562683105, + "step": 6720, + "unclipped_grad_norm": 0.8643161654472351 + }, + { + "bits_per_byte": 1.4391264008835603, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 0.9975264072418213, + "step": 6730, + "unclipped_grad_norm": 0.7649608254432678 + }, + { + "bits_per_byte": 1.4696622645095536, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 1.0186922550201416, + "step": 6740, + "unclipped_grad_norm": 0.8079668283462524 + }, + { + "bits_per_byte": 1.4559390808666066, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 1.0091800689697266, + "step": 6750, + "unclipped_grad_norm": 0.8132560849189758 + }, + { + "bits_per_byte": 1.441940724982279, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 0.9994771480560303, + "step": 6760, + "unclipped_grad_norm": 0.8272307515144348 + }, + { + "bits_per_byte": 1.4738834066923303, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 1.021618127822876, + "step": 6770, + "unclipped_grad_norm": 0.8277111649513245 + }, + { + "bits_per_byte": 1.442480664514663, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 0.999851405620575, + "step": 6780, + "unclipped_grad_norm": 0.8028788566589355 + }, + { + "bits_per_byte": 1.4851632328144695, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.029436707496643, + "step": 6790, + "unclipped_grad_norm": 0.7982462644577026 + }, + { + "bits_per_byte": 1.4703782282850955, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0191885232925415, + "step": 6800, + "unclipped_grad_norm": 0.8199721574783325 + }, + { + "bits_per_byte": 1.4693898439905868, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0185034275054932, + "step": 6810, + "unclipped_grad_norm": 0.8096816539764404 + }, + { + "bits_per_byte": 1.479111163330607, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 1.025241732597351, + "step": 6820, + "unclipped_grad_norm": 0.7861770987510681 + }, + { + "bits_per_byte": 1.4842697829432372, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.0288174152374268, + "step": 6830, + "unclipped_grad_norm": 0.8372291326522827 + }, + { + "bits_per_byte": 1.4464014389977078, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 1.0025690793991089, + "step": 6840, + "unclipped_grad_norm": 0.8044474720954895 + }, + { + "bits_per_byte": 1.5325680147503413, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.0622951984405518, + "step": 6850, + "unclipped_grad_norm": 0.8673999309539795 + }, + { + "bits_per_byte": 1.5006537101776825, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.040173888206482, + "step": 6860, + "unclipped_grad_norm": 0.862133264541626 + }, + { + "bits_per_byte": 1.43319231149819, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 0.9934132099151611, + "step": 6870, + "unclipped_grad_norm": 0.80036860704422 + }, + { + "bits_per_byte": 1.468490718691876, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 1.0178802013397217, + "step": 6880, + "unclipped_grad_norm": 0.8777946829795837 + }, + { + "bits_per_byte": 1.4511889200497758, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 1.005887508392334, + "step": 6890, + "unclipped_grad_norm": 0.8468692898750305 + }, + { + "bits_per_byte": 1.5072036694357926, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 1.0447139739990234, + "step": 6900, + "unclipped_grad_norm": 0.8150814771652222 + }, + { + "bits_per_byte": 1.4986095243895272, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.0387569665908813, + "step": 6910, + "unclipped_grad_norm": 0.828105628490448 + }, + { + "bits_per_byte": 1.4454732486310014, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 1.0019257068634033, + "step": 6920, + "unclipped_grad_norm": 0.79502272605896 + }, + { + "bits_per_byte": 1.4145174893629264, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.9804688096046448, + "step": 6930, + "unclipped_grad_norm": 0.828509509563446 + }, + { + "bits_per_byte": 1.5029957699171324, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 1.0417972803115845, + "step": 6940, + "unclipped_grad_norm": 0.8663814663887024 + }, + { + "bits_per_byte": 1.4178889512591153, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 0.9828057289123535, + "step": 6950, + "unclipped_grad_norm": 0.806262731552124 + }, + { + "bits_per_byte": 1.4227103988847287, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 0.9861477017402649, + "step": 6960, + "unclipped_grad_norm": 0.8236929178237915 + }, + { + "bits_per_byte": 1.4056539334940843, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 0.9743250608444214, + "step": 6970, + "unclipped_grad_norm": 0.8532453775405884 + }, + { + "bits_per_byte": 1.517700802513855, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.051990032196045, + "step": 6980, + "unclipped_grad_norm": 0.9018015265464783 + }, + { + "bits_per_byte": 1.4807167933590624, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.0263546705245972, + "step": 6990, + "unclipped_grad_norm": 0.8347485661506653 + }, + { + "bits_per_byte": 1.4909444296232226, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 1.0334439277648926, + "step": 7000, + "unclipped_grad_norm": 0.9078467488288879 + }, + { + "bits_per_byte": 1.3961774314575672, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 0.967756450176239, + "step": 7010, + "unclipped_grad_norm": 0.8078724145889282 + }, + { + "bits_per_byte": 1.4983813034118323, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 1.0385987758636475, + "step": 7020, + "unclipped_grad_norm": 0.8892475366592407 + }, + { + "bits_per_byte": 1.4547002898324417, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 1.0083214044570923, + "step": 7030, + "unclipped_grad_norm": 0.8249509930610657 + }, + { + "bits_per_byte": 1.4505936880951398, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.0054749250411987, + "step": 7040, + "unclipped_grad_norm": 0.8168858289718628 + }, + { + "bits_per_byte": 1.4524932364739194, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.006791591644287, + "step": 7050, + "unclipped_grad_norm": 0.8396674990653992 + }, + { + "bits_per_byte": 1.4267951588253713, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 0.9889790415763855, + "step": 7060, + "unclipped_grad_norm": 0.8129274845123291 + }, + { + "bits_per_byte": 1.4790917292910597, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 1.0252282619476318, + "step": 7070, + "unclipped_grad_norm": 0.8441708087921143 + }, + { + "bits_per_byte": 1.4681171723742021, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 1.0176212787628174, + "step": 7080, + "unclipped_grad_norm": 0.8331723213195801 + }, + { + "bits_per_byte": 1.4411883868760798, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 0.9989556670188904, + "step": 7090, + "unclipped_grad_norm": 0.8706258535385132 + }, + { + "bits_per_byte": 1.4278659228096433, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 0.9897212386131287, + "step": 7100, + "unclipped_grad_norm": 0.8339611291885376 + }, + { + "bits_per_byte": 1.4113611777629667, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 0.9782810211181641, + "step": 7110, + "unclipped_grad_norm": 0.8087135553359985 + }, + { + "bits_per_byte": 1.4662300067462846, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.0163131952285767, + "step": 7120, + "unclipped_grad_norm": 0.8630717396736145 + }, + { + "bits_per_byte": 1.495092307196732, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 1.0363190174102783, + "step": 7130, + "unclipped_grad_norm": 0.8377772569656372 + }, + { + "bits_per_byte": 1.4547083730170323, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.0083270072937012, + "step": 7140, + "unclipped_grad_norm": 0.840854287147522 + }, + { + "bits_per_byte": 1.478726438140627, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 1.024975061416626, + "step": 7150, + "unclipped_grad_norm": 0.8724638819694519 + }, + { + "bits_per_byte": 1.4841982381604781, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0287678241729736, + "step": 7160, + "unclipped_grad_norm": 0.8433313965797424 + }, + { + "bits_per_byte": 1.4970965930098872, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.0377082824707031, + "step": 7170, + "unclipped_grad_norm": 0.8823571801185608 + }, + { + "bits_per_byte": 1.4508581974121666, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 1.0056582689285278, + "step": 7180, + "unclipped_grad_norm": 0.8181361556053162 + }, + { + "bits_per_byte": 1.506407045796996, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 1.0441617965698242, + "step": 7190, + "unclipped_grad_norm": 0.8949211239814758 + }, + { + "bits_per_byte": 1.4327776613269596, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 0.9931257963180542, + "step": 7200, + "unclipped_grad_norm": 0.8277369141578674 + }, + { + "bits_per_byte": 1.4302049728527029, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 0.9913425445556641, + "step": 7210, + "unclipped_grad_norm": 0.8151546716690063 + }, + { + "bits_per_byte": 1.4415479166077085, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 0.9992048740386963, + "step": 7220, + "unclipped_grad_norm": 0.8505395650863647 + }, + { + "bits_per_byte": 1.4741217746464261, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 1.0217833518981934, + "step": 7230, + "unclipped_grad_norm": 0.8731361627578735 + }, + { + "bits_per_byte": 1.4951263597616027, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 1.0363426208496094, + "step": 7240, + "unclipped_grad_norm": 0.8732264041900635 + }, + { + "bits_per_byte": 1.4910892390152493, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 1.0335443019866943, + "step": 7250, + "unclipped_grad_norm": 0.9077531695365906 + }, + { + "bits_per_byte": 1.4464908699761565, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 1.0026310682296753, + "step": 7260, + "unclipped_grad_norm": 0.866310715675354 + }, + { + "bits_per_byte": 1.4755265289386736, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 1.0227570533752441, + "step": 7270, + "unclipped_grad_norm": 0.8879220485687256 + }, + { + "bits_per_byte": 1.4585126292541177, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.0109639167785645, + "step": 7280, + "unclipped_grad_norm": 0.8492760062217712 + }, + { + "bits_per_byte": 1.4511491920574264, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 1.0058599710464478, + "step": 7290, + "unclipped_grad_norm": 0.8843321204185486 + }, + { + "bits_per_byte": 1.445528627044579, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 1.0019640922546387, + "step": 7300, + "unclipped_grad_norm": 0.817732572555542 + }, + { + "bits_per_byte": 1.4403970087081346, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 0.9984071254730225, + "step": 7310, + "unclipped_grad_norm": 0.8735653758049011 + }, + { + "bits_per_byte": 1.4129260479031673, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 0.9793657064437866, + "step": 7320, + "unclipped_grad_norm": 0.8592900037765503 + }, + { + "bits_per_byte": 1.468236528333901, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 1.0177040100097656, + "step": 7330, + "unclipped_grad_norm": 0.8909156918525696 + }, + { + "bits_per_byte": 1.4289090835784515, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.9904443025588989, + "step": 7340, + "unclipped_grad_norm": 0.8425762057304382 + }, + { + "bits_per_byte": 1.4445665560956524, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 1.0012972354888916, + "step": 7350, + "unclipped_grad_norm": 0.8770133256912231 + }, + { + "bits_per_byte": 1.4415752618491957, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 0.9992238283157349, + "step": 7360, + "unclipped_grad_norm": 0.83420330286026 + }, + { + "bits_per_byte": 1.4083028102826733, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 0.9761611223220825, + "step": 7370, + "unclipped_grad_norm": 0.8744100332260132 + }, + { + "bits_per_byte": 1.3897802787847489, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 0.9633222818374634, + "step": 7380, + "unclipped_grad_norm": 0.8288680911064148 + }, + { + "bits_per_byte": 1.413930426584206, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 0.9800618886947632, + "step": 7390, + "unclipped_grad_norm": 0.8405190110206604 + }, + { + "bits_per_byte": 1.4240541853272453, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.9870791435241699, + "step": 7400, + "unclipped_grad_norm": 0.8632912635803223 + }, + { + "bits_per_byte": 1.4738995730615114, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 1.0216293334960938, + "step": 7410, + "unclipped_grad_norm": 0.8763834834098816 + }, + { + "bits_per_byte": 1.4578408650198476, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.010498285293579, + "step": 7420, + "unclipped_grad_norm": 0.8779239058494568 + }, + { + "bits_per_byte": 1.406993162396353, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 0.9752533435821533, + "step": 7430, + "unclipped_grad_norm": 0.8496518731117249 + }, + { + "bits_per_byte": 1.3809137132195166, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 0.9571764469146729, + "step": 7440, + "unclipped_grad_norm": 0.8649865984916687 + }, + { + "bits_per_byte": 1.4943858024669876, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 1.0358293056488037, + "step": 7450, + "unclipped_grad_norm": 0.8558982610702515 + }, + { + "bits_per_byte": 1.4135341785566182, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 0.9797872304916382, + "step": 7460, + "unclipped_grad_norm": 0.8545086979866028 + }, + { + "bits_per_byte": 1.3959241010128463, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 0.9675808548927307, + "step": 7470, + "unclipped_grad_norm": 0.8330896496772766 + }, + { + "bits_per_byte": 1.4852968633341899, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 1.029529333114624, + "step": 7480, + "unclipped_grad_norm": 0.8309849500656128 + }, + { + "bits_per_byte": 1.449001300730801, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 1.004371166229248, + "step": 7490, + "unclipped_grad_norm": 0.856985330581665 + }, + { + "bits_per_byte": 1.4795997660417084, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.0255804061889648, + "step": 7500, + "unclipped_grad_norm": 0.8804658055305481 + }, + { + "bits_per_byte": 1.4570958361763098, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 1.0099818706512451, + "step": 7510, + "unclipped_grad_norm": 0.8468979001045227 + }, + { + "bits_per_byte": 1.4713509621583756, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 1.0198627710342407, + "step": 7520, + "unclipped_grad_norm": 0.884891927242279 + }, + { + "bits_per_byte": 1.4497497692273558, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.004889965057373, + "step": 7530, + "unclipped_grad_norm": 0.8724031448364258 + }, + { + "bits_per_byte": 1.4649561312513437, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.015430212020874, + "step": 7540, + "unclipped_grad_norm": 0.8533750772476196 + }, + { + "bits_per_byte": 1.4334140831264774, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 0.9935669302940369, + "step": 7550, + "unclipped_grad_norm": 0.8301107883453369 + }, + { + "bits_per_byte": 1.3763201426062965, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 0.9539924263954163, + "step": 7560, + "unclipped_grad_norm": 0.8443491458892822 + }, + { + "bits_per_byte": 1.4010540855141076, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 0.9711366891860962, + "step": 7570, + "unclipped_grad_norm": 0.844425618648529 + }, + { + "bits_per_byte": 1.3988938114366198, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 0.9696393013000488, + "step": 7580, + "unclipped_grad_norm": 0.8635607361793518 + }, + { + "bits_per_byte": 1.4539557769368567, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 1.007805347442627, + "step": 7590, + "unclipped_grad_norm": 0.8628139495849609 + }, + { + "bits_per_byte": 1.4960316764357446, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 1.0369701385498047, + "step": 7600, + "unclipped_grad_norm": 0.8755514025688171 + }, + { + "bits_per_byte": 1.45712576115756, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 1.010002613067627, + "step": 7610, + "unclipped_grad_norm": 0.8845657110214233 + }, + { + "bits_per_byte": 1.4641213274640557, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 1.0148515701293945, + "step": 7620, + "unclipped_grad_norm": 0.860925555229187 + }, + { + "bits_per_byte": 1.422379332281818, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 0.9859182238578796, + "step": 7630, + "unclipped_grad_norm": 0.8529493808746338 + }, + { + "bits_per_byte": 1.409826232604016, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.9772170782089233, + "step": 7640, + "unclipped_grad_norm": 0.8849031329154968 + }, + { + "bits_per_byte": 1.3750410216405045, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 0.9531058073043823, + "step": 7650, + "unclipped_grad_norm": 0.8356350660324097 + }, + { + "bits_per_byte": 1.3941935255885394, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 0.966381311416626, + "step": 7660, + "unclipped_grad_norm": 0.8818058371543884 + }, + { + "bits_per_byte": 1.4632259857836638, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 1.0142309665679932, + "step": 7670, + "unclipped_grad_norm": 0.8807849884033203 + }, + { + "bits_per_byte": 1.4441820028883232, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 1.001030683517456, + "step": 7680, + "unclipped_grad_norm": 0.8890030384063721 + }, + { + "bits_per_byte": 1.4307842104208088, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 0.9917440414428711, + "step": 7690, + "unclipped_grad_norm": 0.8743600845336914 + }, + { + "bits_per_byte": 1.4648470942506966, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 1.0153546333312988, + "step": 7700, + "unclipped_grad_norm": 0.8784114718437195 + }, + { + "bits_per_byte": 1.398571687931554, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 0.9694160223007202, + "step": 7710, + "unclipped_grad_norm": 0.8419727683067322 + }, + { + "bits_per_byte": 1.4118487485781628, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 0.9786189794540405, + "step": 7720, + "unclipped_grad_norm": 0.8376843333244324 + }, + { + "bits_per_byte": 1.4173476358655246, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 0.9824305176734924, + "step": 7730, + "unclipped_grad_norm": 0.8294816017150879 + }, + { + "bits_per_byte": 1.457065395247107, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 1.0099607706069946, + "step": 7740, + "unclipped_grad_norm": 0.8761158585548401 + }, + { + "bits_per_byte": 1.390177386725591, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9635975360870361, + "step": 7750, + "unclipped_grad_norm": 0.8553816676139832 + }, + { + "bits_per_byte": 1.3997380742697054, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 0.9702244997024536, + "step": 7760, + "unclipped_grad_norm": 0.8842922449111938 + }, + { + "bits_per_byte": 1.454595896363368, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 1.008249044418335, + "step": 7770, + "unclipped_grad_norm": 0.8904503583908081 + }, + { + "bits_per_byte": 1.434321721566406, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 0.9941960573196411, + "step": 7780, + "unclipped_grad_norm": 0.9097840785980225 + }, + { + "bits_per_byte": 1.4461262667563273, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 1.0023783445358276, + "step": 7790, + "unclipped_grad_norm": 0.8700852990150452 + }, + { + "bits_per_byte": 1.4464286122565442, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 1.002587914466858, + "step": 7800, + "unclipped_grad_norm": 0.8869295120239258 + }, + { + "bits_per_byte": 1.422014385096687, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9856652617454529, + "step": 7810, + "unclipped_grad_norm": 0.8851216435432434 + }, + { + "bits_per_byte": 1.4434567520496349, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 1.0005279779434204, + "step": 7820, + "unclipped_grad_norm": 0.9227840900421143 + }, + { + "bits_per_byte": 1.4492475798868365, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 1.0045418739318848, + "step": 7830, + "unclipped_grad_norm": 0.8756785988807678 + }, + { + "bits_per_byte": 1.422440558105525, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.9859606623649597, + "step": 7840, + "unclipped_grad_norm": 0.8852615356445312 + }, + { + "bits_per_byte": 1.41144355745273, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.9783381223678589, + "step": 7850, + "unclipped_grad_norm": 0.8239513635635376 + }, + { + "bits_per_byte": 1.460244838513608, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 1.01216459274292, + "step": 7860, + "unclipped_grad_norm": 0.9041352868080139 + }, + { + "bits_per_byte": 1.4572076248993706, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0100593566894531, + "step": 7870, + "unclipped_grad_norm": 0.9027304649353027 + }, + { + "bits_per_byte": 1.409934667665385, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9772922396659851, + "step": 7880, + "unclipped_grad_norm": 0.8465965390205383 + }, + { + "bits_per_byte": 1.4298815594777554, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 0.9911183714866638, + "step": 7890, + "unclipped_grad_norm": 0.8582502007484436 + }, + { + "bits_per_byte": 1.3946472158215155, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 0.9666957855224609, + "step": 7900, + "unclipped_grad_norm": 0.8593489527702332 + }, + { + "bits_per_byte": 1.4539912053629345, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.0078299045562744, + "step": 7910, + "unclipped_grad_norm": 0.9641585350036621 + }, + { + "bits_per_byte": 1.4634609140847423, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 1.0143938064575195, + "step": 7920, + "unclipped_grad_norm": 0.8732636570930481 + }, + { + "bits_per_byte": 1.4218497117084858, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.985551118850708, + "step": 7930, + "unclipped_grad_norm": 0.8945783376693726 + }, + { + "bits_per_byte": 1.4504538661999882, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 1.005378007888794, + "step": 7940, + "unclipped_grad_norm": 0.9048901796340942 + }, + { + "bits_per_byte": 1.3842384818259965, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 0.9594810009002686, + "step": 7950, + "unclipped_grad_norm": 0.8598849773406982 + }, + { + "bits_per_byte": 1.46090903551124, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 1.012624979019165, + "step": 7960, + "unclipped_grad_norm": 0.9215705990791321 + }, + { + "bits_per_byte": 1.418839671353085, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 0.9834647178649902, + "step": 7970, + "unclipped_grad_norm": 0.860109806060791 + }, + { + "bits_per_byte": 1.4586677576051958, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 1.0110714435577393, + "step": 7980, + "unclipped_grad_norm": 0.9471375942230225 + }, + { + "bits_per_byte": 1.4152665597987593, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 0.9809880256652832, + "step": 7990, + "unclipped_grad_norm": 0.9314482808113098 + }, + { + "bits_per_byte": 1.4567645975907482, + "learning_rate": 3e-05, + "loss_nats": 1.0097522735595703, + "step": 8000, + "unclipped_grad_norm": 0.9127020239830017 + } + ], + "variant": "uniform_groups_6_7_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073003.json b/experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073003.json new file mode 100644 index 0000000..c54e9fe --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073003.json @@ -0,0 +1,14077 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "e697e7d99286c1c6ad7eca3cf424fa7d6384efd2a79c0e2a3871cae67a74535f", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.00027819047681987286, + 0.0002560806169640273, + 0.0002331122086616233, + 0.00022053472639527172, + 0.00020666928321588784, + 0.00019684624567162246, + 0.0001890837011160329, + 0.00018263237143401057, + 0.00017469045997131616, + 0.0001688334159553051, + 0.0001649101759539917, + 0.00016157960635609925, + 0.00015717606584075838, + 0.00015400603297166526, + 0.00015046694898046553, + 0.00014767753600608557, + 0.00014588078192900866, + 0.00014366162940859795, + 0.00014116008242126554, + 0.00013849612150806934, + 0.0001361419854220003, + 0.00013430001854430884, + 0.00013224364374764264, + 0.00013045054220128804, + 0.00012888714263681322, + 0.00012767451698891819, + 0.0001263424928765744, + 0.00012512138346210122, + 0.00012419422273524106, + 0.00012343283742666245, + 0.0001227961911354214, + 0.00012195061572128907 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00022039370378479362, + "first_to_last_ratio": 1.7624457042066763, + "imbalance_abs_log_ratio": 0.5667024491058967, + "last_quartile_mean": 0.00012504992537287762, + "mean": 0.0001607882525149762, + "normalized": [ + 1.7301666786506156, + 1.592657504255016, + 1.4498087081325215, + 1.3715848200709229, + 1.2853506396348091, + 1.224257634451794, + 1.175979576607571, + 1.1358564358860717, + 1.0864628307036615, + 1.050035766385231, + 1.0256357250890054, + 1.0049217142965674, + 0.9775345112735686, + 0.9578189361646354, + 0.9358080993289649, + 0.9184597363064851, + 0.9072850761620224, + 0.8934833681037547, + 0.8779253472396409, + 0.8613572157279928, + 0.8467159963027754, + 0.8352601414820391, + 0.8224708066612338, + 0.81131886291965, + 0.8015955184587156, + 0.7940537632065271, + 0.7857694259399116, + 0.7781749070905981, + 0.7724085609032495, + 0.7676732316943716, + 0.7637136993200662, + 0.75845476155001 + ], + "population_cv": 0.2498932502086342 + }, + "activation_output_rms_by_block": [ + 0.0034565660171210766, + 0.004928664769977331, + 0.0060537648387253284, + 0.006960673723369837, + 0.0036145171616226435, + 0.0051584807224571705, + 0.006370538379997015, + 0.007546419743448496, + 0.00395671371370554, + 0.005665866658091545, + 0.006755253300070763, + 0.007946331985294819, + 0.004383862018585205, + 0.006652849726378918, + 0.007932198233902454, + 0.009347881190478802, + 0.004880861379206181, + 0.006906712893396616, + 0.008132148534059525, + 0.009816600009799004, + 0.005453306715935469, + 0.007622495759278536, + 0.009448722936213017, + 0.010670131072402, + 0.0055701653473079205, + 0.007738742511719465, + 0.009509840048849583, + 0.010865328833460808, + 0.005987230688333511, + 0.00815181527286768, + 0.009874933399260044, + 0.011666664853692055 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.005511203169589862, + "first_to_last_ratio": 0.6356203088456116, + "imbalance_abs_log_ratio": 0.45315389258709987, + "last_quartile_mean": 0.008670590119436383, + "mean": 0.007157071326219011, + "normalized": [ + 0.4829581625738996, + 0.6886426787339398, + 0.8458438602600115, + 0.9725589429115095, + 0.5050274053272774, + 0.7207530129760953, + 0.8901040788372985, + 1.0544005221525679, + 0.5528397766850007, + 0.7916459680002588, + 0.9438571997073384, + 1.1102770425361632, + 0.6125217730505899, + 0.9295491721602742, + 1.1083022471557418, + 1.3061042379491232, + 0.6819634955048963, + 0.9650194302374437, + 1.1362396940587196, + 1.3715945478756304, + 0.7619466772614072, + 1.0650300118366156, + 1.3201940438400879, + 1.490851576861243, + 0.7782743937317399, + 1.0812722353862223, + 1.3287334463206348, + 1.5181249897087195, + 0.8365475786723071, + 1.138987569259587, + 1.3797450031110483, + 1.6300892253166077 + ], + "population_cv": 0.3025375276561286 + }, + "bits_per_byte": 8.06433666787865, + "branch_output_rms_by_sublayer": [ + 0.0021227982360869646, + 0.0027448106557130814, + 0.00224129855632782, + 0.0027243648655712605, + 0.0024903982412070036, + 0.0027103275060653687, + 0.0023722455371171236, + 0.0027234479784965515, + 0.0023577234242111444, + 0.0027595730498433113, + 0.0023993616923689842, + 0.0027272943407297134, + 0.002688874490559101, + 0.002728298306465149, + 0.0029664887115359306, + 0.0027007071767002344, + 0.0029547461308538914, + 0.002684502862393856, + 0.003052416956052184, + 0.002683063969016075, + 0.0028214550111442804, + 0.0027139110025018454, + 0.0032544767018407583, + 0.002666700165718794, + 0.00348988501355052, + 0.002679243916645646, + 0.003914494998753071, + 0.0026584581937640905, + 0.0035071265883743763, + 0.002684473292902112, + 0.004211232997477055, + 0.0026407993864268064, + 0.004072706215083599, + 0.002663099905475974, + 0.004050548188388348, + 0.0026283918414264917, + 0.003929397091269493, + 0.002597011858597398, + 0.004410331137478352, + 0.0026434636674821377, + 0.004824656527489424, + 0.0025708195753395557, + 0.00433384021744132, + 0.002691218862310052, + 0.004475373309105635, + 0.0026598076801747084, + 0.004755842965096235, + 0.0026596153620630503, + 0.004843301139771938, + 0.0026765079237520695, + 0.0050001065246760845, + 0.002582729561254382, + 0.005346097983419895, + 0.002614464145153761, + 0.004958339501172304, + 0.0025760054122656584, + 0.0054801348596811295, + 0.002634295728057623, + 0.005455824080854654, + 0.00259007909335196, + 0.005286100786179304, + 0.00257018837146461, + 0.005402065813541412, + 0.0025643527042120695 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.006159982634052032, + 0.005514663551817784, + 0.005191681274696802, + 0.00489831575692433, + 0.004666415075367917, + 0.004399813239791133, + 0.004483878567781538, + 0.004868281936291656, + 0.00443828064080646, + 0.004266304440803418, + 0.0039471188591477255, + 0.004494388758593749, + 0.0042334459038140095, + 0.00446434979681851, + 0.004322339302329385, + 0.004638471384339534, + 0.004165330760400444, + 0.004499937764754925, + 0.003981478974932938, + 0.004372386299982334, + 0.00458495381977442, + 0.00429501466057645, + 0.003963900484885007, + 0.004271473918508079, + 0.004041799650627023, + 0.004206875397956982, + 0.003955591472942362, + 0.003916464042690177, + 0.003989587235588314, + 0.003922309599766117, + 0.003679536678174494, + 0.003763244111925236 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.005022879004590398, + "first_to_last_ratio": 1.2766484804448084, + "imbalance_abs_log_ratio": 0.24423826934058726, + "last_quartile_mean": 0.003934426023708838, + "mean": 0.0043936754999019155, + "normalized": [ + 1.4020112851278042, + 1.255136741878387, + 1.1816260155791434, + 1.1148560600421402, + 1.0620754936207943, + 1.0013969488391563, + 1.0205302070855338, + 1.108020366183241, + 1.0101521245493756, + 0.9710103627130996, + 0.8983637638318627, + 1.0229223252136126, + 0.9635317637610054, + 1.0160854612313022, + 0.9837638902613262, + 1.0557155130011497, + 0.9480287655502622, + 1.0241852783291305, + 0.9061841219320409, + 0.995154580733134, + 1.0435349219296635, + 0.9775448051801577, + 0.9021832597727114, + 0.9721869351988865, + 0.9199131002545027, + 0.9574843199164109, + 0.9002921296829196, + 0.891386731400079, + 0.9080295610536958, + 0.892717179467095, + 0.8374620925593245, + 0.8565138941210488 + ], + "population_cv": 0.11556590700625036 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028097106143832207, + 0.014079716056585312, + 0.014136734418570995, + 0.014167090877890587, + 0.014224153943359852, + 0.014255538582801819, + 0.014337832108139992, + 0.014355218037962914, + 0.014408928342163563, + 0.009655896574258804, + 0.009704872034490108, + 0.009734091348946095, + 0.009773381985723972, + 0.009828771464526653, + 0.00987160298973322, + 0.009915035218000412, + 0.00994801800698042, + 0.007479202933609486, + 0.007515520788729191, + 0.007559895049780607, + 0.007602147292345762, + 0.0076358383521437645, + 0.007658426649868488, + 0.007706487085670233, + 0.007734050042927265, + 0.0062490664422512054, + 0.0062688966281712055, + 0.006315518636256456, + 0.006336068734526634, + 0.0063744354993104935, + 0.006399397272616625, + 0.006439714692533016, + 0.006462901830673218, + 0.0054116579703986645, + 0.005432929843664169, + 0.005481995176523924, + 0.005504137370735407, + 0.005541813559830189, + 0.005560056772083044, + 0.005614723544567823, + 0.005631812382489443, + 0.004862982779741287, + 0.004873365629464388, + 0.004923288244754076, + 0.004937449004501104, + 0.0049925679340958595, + 0.005005750339478254, + 0.005004179198294878, + 0.005022916477173567, + 0.004448203835636377, + 0.0044684382155537605, + 0.004490054678171873, + 0.004500201437622309, + 0.00455933902412653, + 0.004573327023535967, + 0.004625064320862293, + 0.00463611725717783, + 0.004160413984209299, + 0.004166214261204004, + 0.004242690280079842, + 0.0042496914975345135, + 0.004270957782864571, + 0.004283483140170574, + 0.004303553607314825 + ], + "loss_nats": 5.5897722244262695, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0021227982360869646, + 0.0034565660171210766, + 0.00413061399012804, + 0.004928664769977331, + 0.005416445434093475, + 0.0060537648387253284, + 0.006388718727976084, + 0.006960673723369837, + 0.0023577234242111444, + 0.0036145171616226435, + 0.004388594534248114, + 0.0051584807224571705, + 0.00574506726115942, + 0.006370538379997015, + 0.007066869642585516, + 0.007546419743448496, + 0.0029547461308538914, + 0.00395671371370554, + 0.004991721361875534, + 0.005665866658091545, + 0.006190058775246143, + 0.006755253300070763, + 0.0074937604367733, + 0.007946331985294819, + 0.00348988501355052, + 0.004383862018585205, + 0.006057556718587875, + 0.006652849726378918, + 0.0074848332442343235, + 0.007932198233902454, + 0.00897275935858488, + 0.009347881190478802, + 0.004072706215083599, + 0.004880861379206181, + 0.0063449000008404255, + 0.006906712893396616, + 0.007735534105449915, + 0.008132148534059525, + 0.00946879480034113, + 0.009816600009799004, + 0.004824656527489424, + 0.005453306715935469, + 0.007074959110468626, + 0.007622495759278536, + 0.009095527231693268, + 0.009448722936213017, + 0.010346688330173492, + 0.010670131072402, + 0.004843301139771938, + 0.0055701653473079205, + 0.007334613241255283, + 0.007738742511719465, + 0.009146629832684994, + 0.009509840048849583, + 0.010587968863546848, + 0.010865328833460808, + 0.0054801348596811295, + 0.005987230688333511, + 0.0077637480571866035, + 0.00815181527286768, + 0.009536821395158768, + 0.009874933399260044, + 0.011367839761078358, + 0.011666664853692055 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.816401582909748e-05, + 6.459993164753541e-05, + 6.122804916230962e-05, + 5.838384822709486e-05, + 5.393978790380061e-05, + 5.085195880383253e-05, + 4.795097993337549e-05, + 4.5285396481631324e-05, + 4.145973434788175e-05, + 3.8650578062515706e-05, + 3.622193617047742e-05, + 3.392690632608719e-05, + 3.170019408571534e-05, + 2.976237374241464e-05, + 2.807834061968606e-05, + 2.642651816131547e-05, + 2.5217916117981076e-05, + 2.388454959145747e-05, + 2.279889304190874e-05, + 2.191068597312551e-05, + 2.1405352526926436e-05, + 2.0655908883782104e-05, + 2.0067092918907292e-05, + 1.9527635231497698e-05, + 1.920874638017267e-05, + 1.878538023447618e-05, + 1.8405704395263456e-05, + 1.8111097233486362e-05, + 1.7792881408240646e-05, + 1.758169491949957e-05, + 1.7379523342242464e-05, + 1.7233598555321805e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.6300495998584665e-05, + "first_to_last_ratio": 3.1170121058986675, + "imbalance_abs_log_ratio": 1.136874884607837, + "last_quartile_mean": 1.8062328308587894e-05, + "mean": 3.239366282059564e-05, + "normalized": [ + 2.104239221313353, + 1.9942151032844389, + 1.890124296885047, + 1.8023231442038368, + 1.6651339554447084, + 1.5698119439429756, + 1.4802580430295964, + 1.3979708541276543, + 1.2798717631129375, + 1.1931524470255945, + 1.1181796998716613, + 1.0473315880943457, + 0.9785924568419168, + 0.9187714864862999, + 0.8667849873967963, + 0.815792839101656, + 0.7784830093973727, + 0.7373216707149234, + 0.7038071973575456, + 0.6763880359708773, + 0.6607882734803636, + 0.637652771722043, + 0.6194758842198231, + 0.6028226983668602, + 0.5929785244279292, + 0.5799091118072817, + 0.5681884292368832, + 0.5590938367726501, + 0.5492704393072855, + 0.5427510626652959, + 0.536509978463834, + 0.532005245926213 + ], + "population_cv": 0.48150343707480214 + }, + "activation_output_rms_by_block": [ + 0.005520942155271769, + 0.009554436430335045, + 0.014000519178807735, + 0.01878626085817814, + 0.00663581769913435, + 0.011810574680566788, + 0.01722543127834797, + 0.022368814796209335, + 0.007465824019163847, + 0.013207847252488136, + 0.019116781651973724, + 0.025761326774954796, + 0.008932394906878471, + 0.016149893403053284, + 0.02359013631939888, + 0.031918227672576904, + 0.009814132936298847, + 0.01907377876341343, + 0.028344878926873207, + 0.038373641669750214, + 0.012559778057038784, + 0.022438690066337585, + 0.03435983881354332, + 0.045627936720848083, + 0.014813034795224667, + 0.02957596629858017, + 0.04641536623239517, + 0.06272666901350021, + 0.01975581981241703, + 0.03762444108724594, + 0.05590876564383507, + 0.07515289634466171 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013237849634606391, + "first_to_last_ratio": 0.3096817868756899, + "imbalance_abs_log_ratio": 1.1722100026580597, + "last_quartile_mean": 0.0427466199034825, + "mean": 0.025144089508103207, + "normalized": [ + 0.21957216440438337, + 0.37998736948720807, + 0.556811539121183, + 0.7471442086667673, + 0.26391163207543544, + 0.4697157428094815, + 0.685068802065659, + 0.8896251657472234, + 0.29692162910722336, + 0.5252863599706655, + 0.7602892777570229, + 1.02454800626814, + 0.3552482941965276, + 0.6422938240753814, + 0.9381980728232958, + 1.2694127445851873, + 0.3903157015542773, + 0.7585790194258777, + 1.1272978851645623, + 1.526149581092746, + 0.499512143917207, + 0.8924041595980737, + 1.3665175190563232, + 1.8146585385859182, + 0.5891259172638109, + 1.1762591876332884, + 1.8459752228227175, + 2.4946884234279088, + 0.7857043225228062, + 1.4963532911032902, + 2.223535102586129, + 2.9888891510842788 + ], + "population_cv": 0.6676401977209104 + }, + "bits_per_byte": 5.183521324613673, + "branch_output_rms_by_sublayer": [ + 0.002715682378038764, + 0.004832939710468054, + 0.002531828358769417, + 0.004951633978635073, + 0.002760748378932476, + 0.005507093388587236, + 0.002654546871781349, + 0.005907066166400909, + 0.0028051594272255898, + 0.006011204794049263, + 0.002543412381783128, + 0.006060866639018059, + 0.0034209867008030415, + 0.0061915963888168335, + 0.0033688985276967287, + 0.005953880492597818, + 0.00356606999412179, + 0.006386091932654381, + 0.0038514393381774426, + 0.006217809859663248, + 0.0033933203667402267, + 0.006497394293546677, + 0.004616235848516226, + 0.006915244273841381, + 0.004843748174607754, + 0.007105018477886915, + 0.0050963591784238815, + 0.006856550462543964, + 0.005340212024748325, + 0.0068215481005609035, + 0.00606229854747653, + 0.007103092037141323, + 0.005526870954781771, + 0.007104143965989351, + 0.00629366235807538, + 0.0076282392255961895, + 0.007119832560420036, + 0.0074433255940675735, + 0.008592812344431877, + 0.0074295224621891975, + 0.008309490978717804, + 0.0074363769963383675, + 0.00745787937194109, + 0.007729838136583567, + 0.009424961172044277, + 0.007436051499098539, + 0.00886266678571701, + 0.007267844397574663, + 0.009580768644809723, + 0.00839823018759489, + 0.011882693506777287, + 0.008350460790097713, + 0.014197822660207748, + 0.008164937607944012, + 0.01370962243527174, + 0.00773513363674283, + 0.014600876718759537, + 0.00796277541667223, + 0.014649493619799614, + 0.007810473907738924, + 0.013789324089884758, + 0.008753451518714428, + 0.015275243669748306, + 0.008414690382778645 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00023382090770529182, + 0.00020269094053736167, + 0.00019548698676906582, + 0.00017994854744274953, + 0.000169740479874054, + 0.0001588205325648179, + 0.00017304038729024809, + 0.00015967262597043526, + 0.00016455094048465097, + 0.00016204102929579444, + 0.00014746741973125247, + 0.00016984722099153268, + 0.00016403348468129258, + 0.00016709380664266086, + 0.0001543646431301034, + 0.00016072289497675644, + 0.00014197616914625325, + 0.0001551789683188849, + 0.00014598305269794662, + 0.0001642616492663035, + 0.00015562271939256807, + 0.0001363750267399419, + 0.00015163610752086596, + 0.0001435036650953136, + 0.0001437199503211265, + 0.00015313097913818114, + 0.00015642282317903313, + 0.00014417670713509623, + 0.00014429395973481832, + 0.00014114189847674579, + 0.00013600349476350978, + 0.00013478366544580949 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.000184152676019253, + "first_to_last_ratio": 1.2769829904210386, + "imbalance_abs_log_ratio": 0.2445002570094893, + "last_quartile_mean": 0.00014420918477429004, + "mean": 0.00015973605263938957, + "normalized": [ + 1.4637954540741767, + 1.268911665138893, + 1.223812555393399, + 1.126536836671371, + 1.0626309907460265, + 0.9942685445180088, + 1.083289491827456, + 0.9996029282813349, + 1.0301427746942715, + 1.014429908704508, + 0.9231943402543318, + 1.0632992250971012, + 1.0269033319084493, + 1.0460619495830519, + 0.9663732174388185, + 1.0061779562037552, + 0.8888173133135451, + 0.9714711597963895, + 0.9139017165242533, + 1.02833171693012, + 0.9742491868375669, + 0.8537523275838917, + 0.9492916909821882, + 0.8983799381801351, + 0.8997339545229651, + 0.9586500768482139, + 0.9792580985594017, + 0.9025934017574656, + 0.9033274414296917, + 0.8835945057148694, + 0.8514264157418677, + 0.8437898847424815 + ], + "population_cv": 0.1283119608426292 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6920350193977356, + "mean_weights": [ + 0.5232011079788208, + 0.4767988920211792 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6922361850738525, + "mean_weights": [ + 0.5206986665725708, + 0.4793013334274292 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6930614709854126, + "mean_weights": [ + 0.502338171005249, + 0.49766185879707336 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6924488544464111, + "mean_weights": [ + 0.5175743103027344, + 0.4824257493019104 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929870247840881, + "mean_weights": [ + 0.5013582706451416, + 0.4986417293548584 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929033994674683, + "mean_weights": [ + 0.509894609451294, + 0.49010539054870605 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927410960197449, + "mean_weights": [ + 0.5128090977668762, + 0.4871909022331238 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928887367248535, + "mean_weights": [ + 0.5100314617156982, + 0.48996856808662415 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0981125831604004, + "mean_weights": [ + 0.3356848955154419, + 0.32094132900238037, + 0.3433738052845001 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985171794891357, + "mean_weights": [ + 0.33795008063316345, + 0.33096984028816223, + 0.3310800790786743 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984188318252563, + "mean_weights": [ + 0.34082239866256714, + 0.33128002285957336, + 0.3278975486755371 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985372066497803, + "mean_weights": [ + 0.3372408151626587, + 0.33113664388656616, + 0.33162254095077515 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098381519317627, + "mean_weights": [ + 0.34112292528152466, + 0.3331053853034973, + 0.325771689414978 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098473072052002, + "mean_weights": [ + 0.33783212304115295, + 0.33313754200935364, + 0.3290303647518158 + ], + "sources": 3 + }, + { + "entropy_mean": 1.098205327987671, + "mean_weights": [ + 0.34278345108032227, + 0.33263900876045227, + 0.32457756996154785 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985093116760254, + "mean_weights": [ + 0.3366430401802063, + 0.33530697226524353, + 0.32805001735687256 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3860595226287842, + "mean_weights": [ + 0.25572261214256287, + 0.25286489725112915, + 0.24698469042778015, + 0.24442780017852783 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861671686172485, + "mean_weights": [ + 0.2540628910064697, + 0.2526942491531372, + 0.2465653270483017, + 0.24667754769325256 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859889507293701, + "mean_weights": [ + 0.25525251030921936, + 0.2542743682861328, + 0.24826151132583618, + 0.24221158027648926 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862321376800537, + "mean_weights": [ + 0.2518759071826935, + 0.2524644732475281, + 0.24843870103359222, + 0.2472209334373474 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859431743621826, + "mean_weights": [ + 0.254672646522522, + 0.25507205724716187, + 0.24962930381298065, + 0.2406260371208191 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861368894577026, + "mean_weights": [ + 0.25204890966415405, + 0.25428909063339233, + 0.24923911690711975, + 0.24442288279533386 + ], + "sources": 4 + }, + { + "entropy_mean": 1.385894536972046, + "mean_weights": [ + 0.25420743227005005, + 0.25656363368034363, + 0.24983736872673035, + 0.23939155042171478 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861730098724365, + "mean_weights": [ + 0.2493215799331665, + 0.2550925016403198, + 0.2504051923751831, + 0.24518072605133057 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6091580390930176, + "mean_weights": [ + 0.20300956070423126, + 0.2057613581418991, + 0.2007068544626236, + 0.19484838843345642, + 0.19567382335662842 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093809604644775, + "mean_weights": [ + 0.19999122619628906, + 0.20289385318756104, + 0.19902992248535156, + 0.19802528619766235, + 0.200059711933136 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6088714599609375, + "mean_weights": [ + 0.20446722209453583, + 0.20714402198791504, + 0.20309042930603027, + 0.1947326958179474, + 0.19056564569473267 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093981266021729, + "mean_weights": [ + 0.20017988979816437, + 0.2025301158428192, + 0.2002895325422287, + 0.19928382337093353, + 0.1977166384458542 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085047721862793, + "mean_weights": [ + 0.20296458899974823, + 0.2106347382068634, + 0.20383580029010773, + 0.19454464316368103, + 0.1880202442407608 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6092844009399414, + "mean_weights": [ + 0.20029163360595703, + 0.2046186327934265, + 0.20172445476055145, + 0.19854769110679626, + 0.19481757283210754 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085381507873535, + "mean_weights": [ + 0.2030726969242096, + 0.21117369830608368, + 0.20326058566570282, + 0.19477629661560059, + 0.18771672248840332 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093015670776367, + "mean_weights": [ + 0.2016163170337677, + 0.20403525233268738, + 0.2002500295639038, + 0.19875164330005646, + 0.19534678757190704 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7907962799072266, + "mean_weights": [ + 0.16954033076763153, + 0.1769932508468628, + 0.1704968363046646, + 0.16254760324954987, + 0.15731333196163177, + 0.16310864686965942 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917119264602661, + "mean_weights": [ + 0.1674097180366516, + 0.16901224851608276, + 0.16775229573249817, + 0.1657467484474182, + 0.1647152304649353, + 0.16536374390125275 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7902342081069946, + "mean_weights": [ + 0.1706220507621765, + 0.18031282722949982, + 0.1723010540008545, + 0.16421601176261902, + 0.15758152306079865, + 0.1549665331840515 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917349338531494, + "mean_weights": [ + 0.16662749648094177, + 0.16852226853370667, + 0.16664135456085205, + 0.16634400188922882, + 0.16624683141708374, + 0.16561803221702576 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7899137735366821, + "mean_weights": [ + 0.1718863993883133, + 0.18010222911834717, + 0.17452453076839447, + 0.16424649953842163, + 0.1564745455980301, + 0.15276579558849335 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917156219482422, + "mean_weights": [ + 0.1659335494041443, + 0.16858306527137756, + 0.16695700585842133, + 0.16677077114582062, + 0.16570396721363068, + 0.16605164110660553 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7884749174118042, + "mean_weights": [ + 0.17287473380565643, + 0.18481233716011047, + 0.17555414140224457, + 0.16396528482437134, + 0.15405462682247162, + 0.14873886108398438 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079286575317383, + "mean_weights": [ + 0.12387315183877945, + 0.12212847918272018, + 0.12293092906475067, + 0.12412279844284058, + 0.12562230229377747, + 0.1255335658788681, + 0.12667343020439148, + 0.12911532819271088 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1897544860839844, + "mean_weights": [ + 0.12040543556213379, + 0.13437892496585846, + 0.12441074848175049, + 0.1165996789932251, + 0.10945087671279907, + 0.1053101047873497, + 0.09913021326065063, + 0.09472149610519409, + 0.09559252858161926 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197054386138916, + "mean_weights": [ + 0.10992924869060516, + 0.10842838883399963, + 0.10941662639379501, + 0.10996294021606445, + 0.11065961420536041, + 0.11117805540561676, + 0.11222304403781891, + 0.11507813632488251, + 0.11312393844127655 + ], + "sources": 9 + }, + { + "entropy_mean": 2.190793514251709, + "mean_weights": [ + 0.11953991651535034, + 0.13204032182693481, + 0.12330931425094604, + 0.11671796441078186, + 0.11062805354595184, + 0.10582989454269409, + 0.10042969882488251, + 0.09672330319881439, + 0.0947815477848053 + ], + "sources": 9 + }, + { + "entropy_mean": 2.19716739654541, + "mean_weights": [ + 0.11096567660570145, + 0.11005344986915588, + 0.10995280742645264, + 0.10996513813734055, + 0.10989762842655182, + 0.11260464042425156, + 0.11269719153642654, + 0.11196349561214447, + 0.1118999719619751 + ], + "sources": 9 + }, + { + "entropy_mean": 2.19034743309021, + "mean_weights": [ + 0.11990296840667725, + 0.1333274096250534, + 0.12371348589658737, + 0.11619628965854645, + 0.11001677811145782, + 0.10621090233325958, + 0.10072537511587143, + 0.0956655889749527, + 0.0942411869764328 + ], + "sources": 9 + }, + { + "entropy_mean": 2.196920156478882, + "mean_weights": [ + 0.1095680519938469, + 0.10721082985401154, + 0.10891026258468628, + 0.10958534479141235, + 0.11024601757526398, + 0.11101588606834412, + 0.11389359086751938, + 0.11441180109977722, + 0.11515821516513824 + ], + "sources": 9 + }, + { + "entropy_mean": 2.18877911567688, + "mean_weights": [ + 0.12050613760948181, + 0.13492467999458313, + 0.12486425042152405, + 0.11728397011756897, + 0.11076700687408447, + 0.106074258685112, + 0.09963071346282959, + 0.09463612735271454, + 0.09131284058094025 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.02861778251826763, + 0.01496149506419897, + 0.015276923775672913, + 0.014784772880375385, + 0.015796421095728874, + 0.015450108796358109, + 0.01660514622926712, + 0.016709795221686363, + 0.01784764975309372, + 0.011800500564277172, + 0.012908930890262127, + 0.013010304421186447, + 0.014014357700943947, + 0.01416988018900156, + 0.015305271372199059, + 0.015453503467142582, + 0.016497977077960968, + 0.012539377436041832, + 0.013501109555363655, + 0.013629280962049961, + 0.014522580429911613, + 0.014704737812280655, + 0.015671445056796074, + 0.015808571130037308, + 0.01695968024432659, + 0.01380107831209898, + 0.014685645699501038, + 0.014929752796888351, + 0.01580941677093506, + 0.016086125746369362, + 0.01699533686041832, + 0.01724478416144848, + 0.01825549080967903, + 0.015461772680282593, + 0.01630270853638649, + 0.01665503904223442, + 0.01759832352399826, + 0.017755307257175446, + 0.018836453557014465, + 0.01891903206706047, + 0.020232973620295525, + 0.017715876922011375, + 0.018473053351044655, + 0.01887219399213791, + 0.01965896598994732, + 0.02023034542798996, + 0.021011387929320335, + 0.021628815680742264, + 0.022398309782147408, + 0.020178601145744324, + 0.020946361124515533, + 0.02167428284883499, + 0.02244539000093937, + 0.02344209887087345, + 0.024203453212976456, + 0.02518332190811634, + 0.026152627542614937, + 0.022846585139632225, + 0.024925731122493744, + 0.024386556819081306, + 0.026462100446224213, + 0.02564980834722519, + 0.028442492708563805, + 0.02694111131131649 + ], + "loss_nats": 3.5929431915283203, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1923866271972656, + "mean_weights": [ + 0.09996331483125687, + 0.09331953525543213, + 0.10059553384780884, + 0.1079796776175499, + 0.11491774022579193, + 0.11774426698684692, + 0.12089118361473083, + 0.12261839956045151, + 0.12197032570838928 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.002715682378038764, + 0.005520942155271769, + 0.00624072877690196, + 0.009554436430335045, + 0.009989554062485695, + 0.014000519178807735, + 0.014332035556435585, + 0.01878626085817814, + 0.0028051594272255898, + 0.00663581769913435, + 0.007202635984867811, + 0.011810574680566788, + 0.012382718734443188, + 0.01722543127834797, + 0.017774248495697975, + 0.022368814796209335, + 0.00356606999412179, + 0.007465824019163847, + 0.008660274557769299, + 0.013207847252488136, + 0.014126150868833065, + 0.019116781651973724, + 0.020290663465857506, + 0.025761326774954796, + 0.004843748174607754, + 0.008932394906878471, + 0.011116033419966698, + 0.016149893403053284, + 0.018401717767119408, + 0.02359013631939888, + 0.026343002915382385, + 0.031918227672576904, + 0.005526870954781771, + 0.009814132936298847, + 0.013462314382195473, + 0.01907377876341343, + 0.022680213674902916, + 0.028344878926873207, + 0.03258388116955757, + 0.038373641669750214, + 0.008309490978717804, + 0.012559778057038784, + 0.016885953024029732, + 0.022438690066337585, + 0.028723804280161858, + 0.03435983881354332, + 0.04001648724079132, + 0.045627936720848083, + 0.009580768644809723, + 0.014813034795224667, + 0.023388538509607315, + 0.02957596629858017, + 0.04012707248330116, + 0.04641536623239517, + 0.0566704161465168, + 0.06272666901350021, + 0.014600876718759537, + 0.01975581981241703, + 0.031985681504011154, + 0.03762444108724594, + 0.04894806072115898, + 0.05590876564383507, + 0.06842334568500519, + 0.07515289634466171 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.0001455797319067642, + 0.00015071721281856298, + 0.0001452456635888666, + 0.00013979071809444577, + 0.00014201068552210927, + 0.00013072651927359402, + 0.00012021356815239415, + 0.00011013842595275491, + 0.00010502194345463067, + 9.624040831113234e-05, + 8.855360647430643e-05, + 8.2683443906717e-05, + 8.231480023823678e-05, + 7.691894279560074e-05, + 6.996515730861574e-05, + 6.454550748458132e-05, + 6.13390511716716e-05, + 5.61736014788039e-05, + 5.073993816040456e-05, + 4.684478699346073e-05, + 4.334143886808306e-05, + 3.9756814658176154e-05, + 3.70069537893869e-05, + 3.467940041446127e-05, + 3.317321898066439e-05, + 3.084823038079776e-05, + 2.933880386990495e-05, + 2.8611797461053357e-05, + 2.8641657991101965e-05, + 2.820031477313023e-05, + 2.8049316824763082e-05, + 2.7998732548439875e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.0001355528156636865, + "first_to_last_ratio": 4.617273926961784, + "imbalance_abs_log_ratio": 1.5298044718410857, + "last_quartile_mean": 2.935775910373195e-05, + "mean": 7.360657480148802e-05, + "normalized": [ + 1.9778088071532054, + 2.047605301904581, + 1.9732702408797633, + 1.8991607539333535, + 1.9293206606217248, + 1.7760168792822468, + 1.633190628372561, + 1.496312336904561, + 1.4268011212023897, + 1.3074974425951247, + 1.2030665292214664, + 1.123316009876953, + 1.1183077117802895, + 1.0450009799130846, + 0.9505286381998763, + 0.8768986691563384, + 0.8333365781127419, + 0.7631601066929188, + 0.6893397539179993, + 0.6364212316603234, + 0.5888256447874664, + 0.5401258619273837, + 0.5027669591906995, + 0.47114541748463745, + 0.45068282378483615, + 0.4190961264532871, + 0.39858944597041407, + 0.388712523823007, + 0.38911820130669833, + 0.38312222582268746, + 0.38107080651980074, + 0.3803835815475801 + ], + "population_cv": 0.5702118885298645 + }, + "activation_output_rms_by_block": [ + 0.02133006602525711, + 0.03659679740667343, + 0.04965122416615486, + 0.05993568152189255, + 0.022419068962335587, + 0.02709372714161873, + 0.034259479492902756, + 0.03876027837395668, + 0.019337469711899757, + 0.025743016973137856, + 0.03297785297036171, + 0.039158884435892105, + 0.02106749825179577, + 0.03277015686035156, + 0.037133507430553436, + 0.04253266379237175, + 0.027558336034417152, + 0.033867884427309036, + 0.04354614391922951, + 0.05029674991965294, + 0.02994200587272644, + 0.048489294946193695, + 0.05383354052901268, + 0.06795433908700943, + 0.024078167974948883, + 0.045469481498003006, + 0.06551749259233475, + 0.0873798280954361, + 0.029591713100671768, + 0.06267120689153671, + 0.09506357461214066, + 0.13159257173538208 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.03625579038634896, + "first_to_last_ratio": 0.5357694703285827, + "imbalance_abs_log_ratio": 0.6240513030982704, + "last_quartile_mean": 0.06767050456255674, + "mean": 0.044925615773536265, + "normalized": [ + 0.47478628078864815, + 0.814608698768933, + 1.1051873927881084, + 1.3341092935491397, + 0.4990264145815379, + 0.603079705756164, + 0.762582301945509, + 0.8627656562203134, + 0.4304330475819682, + 0.5730142265139959, + 0.7340545566831727, + 0.871638234927854, + 0.4689417805199171, + 0.729431445648766, + 0.8265553357740992, + 0.946735243580699, + 0.6134214425314695, + 0.7538657811176657, + 0.969294313933047, + 1.1195561608591367, + 0.6664795875845063, + 1.0793239916982205, + 1.1982816395968843, + 1.5125967205337314, + 0.5359563260373226, + 1.0121059158589676, + 1.4583549154362012, + 1.9449889910447877, + 0.6586824151690974, + 1.3949993965013965, + 2.1160216276465262, + 2.929121158822214 + ], + "population_cv": 0.5333175379316631 + }, + "bits_per_byte": 3.083476947319296, + "branch_output_rms_by_sublayer": [ + 0.010896719060838223, + 0.016794223338365555, + 0.014081712812185287, + 0.012681919150054455, + 0.012359644286334515, + 0.013004478067159653, + 0.011513753794133663, + 0.012683904729783535, + 0.020123755559325218, + 0.008086138404905796, + 0.022705350071191788, + 0.008010311983525753, + 0.021436627954244614, + 0.007301283534616232, + 0.01400331687182188, + 0.008281868882477283, + 0.016638703644275665, + 0.008350465446710587, + 0.014064709655940533, + 0.007866447791457176, + 0.013398478738963604, + 0.007607573177665472, + 0.012602395378053188, + 0.008093936368823051, + 0.018125703558325768, + 0.00911905150860548, + 0.017135752364993095, + 0.008371320553123951, + 0.025901980698108673, + 0.009295077994465828, + 0.01743973419070244, + 0.00945811066776514, + 0.02475162222981453, + 0.010629010386765003, + 0.0171822402626276, + 0.010581896640360355, + 0.018213940784335136, + 0.012264817953109741, + 0.024904746562242508, + 0.013786884024739265, + 0.026277849450707436, + 0.0134026063606143, + 0.03628547117114067, + 0.015776431187987328, + 0.022936983034014702, + 0.015481598675251007, + 0.014343642629683018, + 0.018496496602892876, + 0.01471907738596201, + 0.01834065094590187, + 0.01647946797311306, + 0.019849255681037903, + 0.010129584930837154, + 0.02140134572982788, + 0.006027875933796167, + 0.024515407159924507, + 0.006824173498898745, + 0.026826053857803345, + 0.012643430382013321, + 0.028400201350450516, + 0.005394645035266876, + 0.03210867568850517, + 0.00889973621815443, + 0.03472597897052765 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0005159239783740132, + 0.00043347167237452845, + 0.00040282854843988106, + 0.0003789152955276376, + 0.00035476660664310344, + 0.0003506193371636615, + 0.0002854770405222703, + 0.00025447974738463996, + 0.0002736322195591077, + 0.0002505824295091003, + 0.000238031421065728, + 0.00021150600719448855, + 0.00024471327654747234, + 0.00019985921869072445, + 0.00018622669896129938, + 0.00016581119851786372, + 0.00016598707939561121, + 0.00014052446496112863, + 0.00013033170146838866, + 0.00013169682088079308, + 0.00012589945588973363, + 0.00012699397951932152, + 0.00011183250343326783, + 0.00010227608789980666, + 0.00010571051132372853, + 0.00010396497552429689, + 9.077847606649772e-05, + 8.679720908353474e-05, + 9.385788683328275e-05, + 9.726661447068656e-05, + 9.697092314291889e-05, + 0.00010482840319752068 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00037206027830371694, + "first_to_last_ratio": 3.8151468937017685, + "imbalance_abs_log_ratio": 1.3389791682420102, + "last_quartile_mean": 9.752187495530835e-05, + "mean": 0.0002050800559239387, + "normalized": [ + 2.5157199028917865, + 2.1136705391542168, + 1.9642502369381276, + 1.8476457587283472, + 1.729893260681972, + 1.7096705751518875, + 1.3920273194588444, + 1.2408800370086837, + 1.3342702601007386, + 1.2218761516333787, + 1.1606756200320645, + 1.031333867360233, + 1.19325731332077, + 0.9745424429026361, + 0.9080683059222894, + 0.8085193744015788, + 0.8093769949876232, + 0.6852176060125865, + 0.6355162177092777, + 0.6421727373120942, + 0.6139039475219763, + 0.6192410029692103, + 0.5453114537609886, + 0.498712990100507, + 0.5154597352115755, + 0.5069482503108739, + 0.4426489726537142, + 0.42323573929454456, + 0.4576646247263231, + 0.4742860734676286, + 0.4728442398069369, + 0.5111584484665835 + ], + "population_cv": 0.5607428945577665 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6436527967453003, + "mean_weights": [ + 0.36712443828582764, + 0.6328755617141724 + ], + "sources": 2 + }, + { + "entropy_mean": 0.678328275680542, + "mean_weights": [ + 0.44874346256256104, + 0.5512565970420837 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6893893480300903, + "mean_weights": [ + 0.486998051404953, + 0.5130019187927246 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6784281730651855, + "mean_weights": [ + 0.46703025698661804, + 0.5329697728157043 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6797459125518799, + "mean_weights": [ + 0.44392699003219604, + 0.5560730695724487 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6810081005096436, + "mean_weights": [ + 0.4665757417678833, + 0.5334242582321167 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6859228610992432, + "mean_weights": [ + 0.5150086879730225, + 0.4849913418292999 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6815803050994873, + "mean_weights": [ + 0.456489622592926, + 0.543510377407074 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0884934663772583, + "mean_weights": [ + 0.28649747371673584, + 0.37324267625808716, + 0.3402598798274994 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0830835103988647, + "mean_weights": [ + 0.30145174264907837, + 0.3134004771709442, + 0.3851478099822998 + ], + "sources": 3 + }, + { + "entropy_mean": 1.092095136642456, + "mean_weights": [ + 0.3024594187736511, + 0.33196404576301575, + 0.36557650566101074 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0697039365768433, + "mean_weights": [ + 0.29316943883895874, + 0.2949412763118744, + 0.41188931465148926 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0871790647506714, + "mean_weights": [ + 0.2912317216396332, + 0.31717509031295776, + 0.39159318804740906 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0581157207489014, + "mean_weights": [ + 0.28815174102783203, + 0.281956285238266, + 0.4298919439315796 + ], + "sources": 3 + }, + { + "entropy_mean": 1.084208369255066, + "mean_weights": [ + 0.2831770181655884, + 0.33562830090522766, + 0.3811946511268616 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0418733358383179, + "mean_weights": [ + 0.2701767086982727, + 0.2820456027984619, + 0.4477776885032654 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3799623250961304, + "mean_weights": [ + 0.22686973214149475, + 0.239097461104393, + 0.2619684636592865, + 0.27206435799598694 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3550333976745605, + "mean_weights": [ + 0.2380669265985489, + 0.2208915501832962, + 0.24992331862449646, + 0.29111820459365845 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3811836242675781, + "mean_weights": [ + 0.23554804921150208, + 0.23798415064811707, + 0.256327748298645, + 0.27014005184173584 + ], + "sources": 4 + }, + { + "entropy_mean": 1.34543776512146, + "mean_weights": [ + 0.24747341871261597, + 0.21370822191238403, + 0.22782433032989502, + 0.310994029045105 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3800816535949707, + "mean_weights": [ + 0.2494419515132904, + 0.2304031252861023, + 0.23784586787223816, + 0.28230905532836914 + ], + "sources": 4 + }, + { + "entropy_mean": 1.344259262084961, + "mean_weights": [ + 0.24441248178482056, + 0.21649102866649628, + 0.22434546053409576, + 0.3147510290145874 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3820087909698486, + "mean_weights": [ + 0.24779994785785675, + 0.23670683801174164, + 0.2449905276298523, + 0.2705026865005493 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3418524265289307, + "mean_weights": [ + 0.2504882216453552, + 0.21673384308815002, + 0.2161506712436676, + 0.31662726402282715 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6047652959823608, + "mean_weights": [ + 0.1885252594947815, + 0.1973411589860916, + 0.18819493055343628, + 0.20556962490081787, + 0.22036904096603394 + ], + "sources": 5 + }, + { + "entropy_mean": 1.566558837890625, + "mean_weights": [ + 0.19768469035625458, + 0.17173728346824646, + 0.16179370880126953, + 0.21233251690864563, + 0.256451815366745 + ], + "sources": 5 + }, + { + "entropy_mean": 1.603715181350708, + "mean_weights": [ + 0.21396374702453613, + 0.1842249035835266, + 0.18344125151634216, + 0.20039400458335876, + 0.21797610819339752 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5695998668670654, + "mean_weights": [ + 0.2019365280866623, + 0.17275680601596832, + 0.16237829625606537, + 0.20093435049057007, + 0.26199400424957275 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6002347469329834, + "mean_weights": [ + 0.1983470320701599, + 0.18346169590950012, + 0.1812906265258789, + 0.20134364068508148, + 0.23555700480937958 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5658026933670044, + "mean_weights": [ + 0.20218420028686523, + 0.17067359387874603, + 0.16152088344097137, + 0.19121479988098145, + 0.2744065225124359 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6049638986587524, + "mean_weights": [ + 0.19033335149288177, + 0.20086616277694702, + 0.18734194338321686, + 0.19826535880565643, + 0.2231931835412979 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5751742124557495, + "mean_weights": [ + 0.20153957605361938, + 0.17496424913406372, + 0.16635659337043762, + 0.19005471467971802, + 0.26708483695983887 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7848142385482788, + "mean_weights": [ + 0.15849140286445618, + 0.16804572939872742, + 0.15160910785198212, + 0.1621769666671753, + 0.17825418710708618, + 0.18142260611057281 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7645106315612793, + "mean_weights": [ + 0.17290407419204712, + 0.147682324051857, + 0.1349579095840454, + 0.1486993283033371, + 0.18605175614356995, + 0.20970463752746582 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7873966693878174, + "mean_weights": [ + 0.15307126939296722, + 0.17107230424880981, + 0.1623389571905136, + 0.16307476162910461, + 0.1695808470249176, + 0.18086187541484833 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7751150131225586, + "mean_weights": [ + 0.16732725501060486, + 0.15450119972229004, + 0.14628556370735168, + 0.15193745493888855, + 0.17122982442378998, + 0.20871871709823608 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7818505764007568, + "mean_weights": [ + 0.14438942074775696, + 0.1711202710866928, + 0.16690462827682495, + 0.16099679470062256, + 0.16599348187446594, + 0.1905953735113144 + ], + "sources": 6 + }, + { + "entropy_mean": 1.781779408454895, + "mean_weights": [ + 0.17270509898662567, + 0.15913589298725128, + 0.15103164315223694, + 0.1538439691066742, + 0.16335612535476685, + 0.19992728531360626 + ], + "sources": 6 + }, + { + "entropy_mean": 1.772856593132019, + "mean_weights": [ + 0.13313180208206177, + 0.1816387176513672, + 0.16722044348716736, + 0.16252493858337402, + 0.16317199170589447, + 0.1923121213912964 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0707478523254395, + "mean_weights": [ + 0.1182265356183052, + 0.13817650079727173, + 0.12337739020586014, + 0.1258469521999359, + 0.12591652572155, + 0.1342563033103943, + 0.12123153358697891, + 0.11296825110912323 + ], + "sources": 8 + }, + { + "entropy_mean": 2.119823455810547, + "mean_weights": [ + 0.08189164102077484, + 0.16924551129341125, + 0.16028879582881927, + 0.1318211406469345, + 0.1230408176779747, + 0.10848765075206757, + 0.08808310329914093, + 0.07047848403453827, + 0.06666283309459686 + ], + "sources": 9 + }, + { + "entropy_mean": 2.178135871887207, + "mean_weights": [ + 0.10360588133335114, + 0.1296660602092743, + 0.11658625304698944, + 0.11651459336280823, + 0.11510737240314484, + 0.12214130163192749, + 0.1078977957367897, + 0.09206552058458328, + 0.09641523659229279 + ], + "sources": 9 + }, + { + "entropy_mean": 2.122868061065674, + "mean_weights": [ + 0.08270423114299774, + 0.16826891899108887, + 0.16412454843521118, + 0.12992645800113678, + 0.1192590743303299, + 0.10505397617816925, + 0.09205174446105957, + 0.0734248161315918, + 0.06518621742725372 + ], + "sources": 9 + }, + { + "entropy_mean": 2.190673589706421, + "mean_weights": [ + 0.10805831849575043, + 0.12065745890140533, + 0.11422564834356308, + 0.11013759672641754, + 0.10818037390708923, + 0.11351148039102554, + 0.11545215547084808, + 0.10499414801597595, + 0.104782834649086 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1139302253723145, + "mean_weights": [ + 0.0817553848028183, + 0.16819076240062714, + 0.16291680932044983, + 0.1334947645664215, + 0.12584711611270905, + 0.1096724271774292, + 0.08897806704044342, + 0.07051681727170944, + 0.05862784385681152 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1830451488494873, + "mean_weights": [ + 0.11010013520717621, + 0.11996117234230042, + 0.10971016436815262, + 0.11325322091579437, + 0.1122342050075531, + 0.11922186613082886, + 0.11539435386657715, + 0.09814368933439255, + 0.10198120772838593 + ], + "sources": 9 + }, + { + "entropy_mean": 2.110328197479248, + "mean_weights": [ + 0.08115816116333008, + 0.16709910333156586, + 0.1675354242324829, + 0.13573497533798218, + 0.12604784965515137, + 0.1080116555094719, + 0.08738094568252563, + 0.07092888653278351, + 0.05610300973057747 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.04383290186524391, + 0.017135759815573692, + 0.02382885105907917, + 0.026672890409827232, + 0.029747558757662773, + 0.03157307952642441, + 0.0348200798034668, + 0.034763723611831665, + 0.038858287036418915, + 0.027056103572249413, + 0.027092518284916878, + 0.02733360230922699, + 0.028194589540362358, + 0.028629163280129433, + 0.02996184304356575, + 0.03026747517287731, + 0.03154146671295166, + 0.023016080260276794, + 0.024618785828351974, + 0.023278485983610153, + 0.024936042726039886, + 0.0241098590195179, + 0.026010071858763695, + 0.02477046102285385, + 0.026733892038464546, + 0.020374096930027008, + 0.02160041593015194, + 0.020751694217324257, + 0.02222299575805664, + 0.021070493385195732, + 0.022810718044638634, + 0.02177662029862404, + 0.02329069934785366, + 0.01841963827610016, + 0.019086819142103195, + 0.01876281574368477, + 0.01948503963649273, + 0.019551793113350868, + 0.0200536847114563, + 0.020647648721933365, + 0.02071440778672695, + 0.017692919820547104, + 0.018439944833517075, + 0.018615789711475372, + 0.019659213721752167, + 0.019477810710668564, + 0.02059851959347725, + 0.020550785586237907, + 0.022030647844076157, + 0.019320301711559296, + 0.020623933523893356, + 0.020647358149290085, + 0.02210700511932373, + 0.022093426436185837, + 0.023713376373052597, + 0.023746825754642487, + 0.025118902325630188, + 0.021498553454875946, + 0.024261726066470146, + 0.022932186722755432, + 0.02768244594335556, + 0.02391223795711994, + 0.030223043635487556, + 0.02508344128727913 + ], + "loss_nats": 2.137303352355957, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.166712760925293, + "mean_weights": [ + 0.1262352615594864, + 0.08841424435377121, + 0.08745025098323822, + 0.09050662815570831, + 0.09084117412567139, + 0.10568755865097046, + 0.12388661503791809, + 0.1410667449235916, + 0.14591151475906372 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.010896719060838223, + 0.02133006602525711, + 0.02870996855199337, + 0.03659679740667343, + 0.041860803961753845, + 0.04965122416615486, + 0.05183820798993111, + 0.05993568152189255, + 0.020123755559325218, + 0.022419068962335587, + 0.02411990612745285, + 0.02709372714161873, + 0.031289126724004745, + 0.034259479492902756, + 0.03504790738224983, + 0.03876027837395668, + 0.016638703644275665, + 0.019337469711899757, + 0.022248776629567146, + 0.025743016973137856, + 0.029299866408109665, + 0.03297785297036171, + 0.035038210451602936, + 0.039158884435892105, + 0.018125703558325768, + 0.02106749825179577, + 0.029360570013523102, + 0.03277015686035156, + 0.03298123925924301, + 0.037133507430553436, + 0.037826623767614365, + 0.04253266379237175, + 0.02475162222981453, + 0.027558336034417152, + 0.029253283515572548, + 0.033867884427309036, + 0.03681080788373947, + 0.04354614391922951, + 0.042156122624874115, + 0.05029674991965294, + 0.026277849450707436, + 0.02994200587272644, + 0.04157790169119835, + 0.048489294946193695, + 0.04411447048187256, + 0.05383354052901268, + 0.054334916174411774, + 0.06795433908700943, + 0.01471907738596201, + 0.024078167974948883, + 0.03143885359168053, + 0.045469481498003006, + 0.048646729439496994, + 0.06551749259233475, + 0.06716754287481308, + 0.0873798280954361, + 0.006824173498898745, + 0.029591713100671768, + 0.03683311119675636, + 0.06267120689153671, + 0.06499899923801422, + 0.09506357461214066, + 0.09879624843597412, + 0.13159257173538208 + ] + }, + { + "activation_grad_rms_by_block": [ + 8.288036042358726e-05, + 8.706557855475694e-05, + 8.52782977744937e-05, + 8.00883790361695e-05, + 9.299098019255325e-05, + 9.884793689707294e-05, + 9.033123933477327e-05, + 8.419330697506666e-05, + 8.485348371323198e-05, + 7.92632854427211e-05, + 7.343928882619366e-05, + 6.963616760913283e-05, + 6.179724005050957e-05, + 6.035621845512651e-05, + 5.778954437118955e-05, + 5.78681101615075e-05, + 6.598483014386147e-05, + 6.211901927599683e-05, + 5.685548603651114e-05, + 5.228720328886993e-05, + 5.324521771399304e-05, + 5.0081849622074515e-05, + 4.6785877202637494e-05, + 4.4081076339352876e-05, + 4.48548307758756e-05, + 4.188361344859004e-05, + 3.935548011213541e-05, + 3.7429999792948365e-05, + 3.500904858810827e-05, + 3.258328069932759e-05, + 3.138736792607233e-05, + 3.09205825033132e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 8.770950989855919e-05, + "first_to_last_ratio": 2.3913367404273598, + "imbalance_abs_log_ratio": 0.871852515208471, + "last_quartile_mean": 3.667802548079635e-05, + "mean": 6.161075566524232e-05, + "normalized": [ + 1.3452255134462532, + 1.4131555053117932, + 1.384146272086765, + 1.2999090527524777, + 1.5093302977456267, + 1.6043941651057845, + 1.4661602241268012, + 1.3665358599483024, + 1.377251143846983, + 1.2865170145516889, + 1.1919881201461127, + 1.1302599173997476, + 1.0030268154197053, + 0.9796376915593723, + 0.9379781784398968, + 0.9392533744583459, + 1.0709953064426831, + 1.008249592222438, + 0.922817540908464, + 0.8486700532122792, + 0.8642195204242773, + 0.8128751072976721, + 0.7593784023173688, + 0.7154769628027983, + 0.7280357186266493, + 0.6798100915392385, + 0.6387761306803413, + 0.6075237901044684, + 0.5682294951603493, + 0.5288570209456046, + 0.5094462417668332, + 0.5018698792028781 + ], + "population_cv": 0.3247310699062578 + }, + "activation_output_rms_by_block": [ + 0.08424261212348938, + 0.14948369562625885, + 0.1897522509098053, + 0.20237113535404205, + 0.08847226202487946, + 0.11665411293506622, + 0.12323978543281555, + 0.12915553152561188, + 0.044974539428949356, + 0.06980320811271667, + 0.09742555767297745, + 0.12249503284692764, + 0.06076505780220032, + 0.08680541813373566, + 0.10820935666561127, + 0.12568308413028717, + 0.03896655514836311, + 0.06282654404640198, + 0.08634737133979797, + 0.11384473741054535, + 0.05049705132842064, + 0.07969057559967041, + 0.11123643070459366, + 0.14465975761413574, + 0.05405839532613754, + 0.09522009640932083, + 0.12943461537361145, + 0.17285297811031342, + 0.06338083744049072, + 0.1303563416004181, + 0.20190104842185974, + 0.2989218831062317 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.13542142324149609, + "first_to_last_ratio": 0.9452461604254253, + "imbalance_abs_log_ratio": 0.056309898186628085, + "last_quartile_mean": 0.14326577447354794, + "mean": 0.1135539956158027, + "normalized": [ + 0.7418727246596841, + 1.3164107067797093, + 1.6710310357709552, + 1.7821577678230034, + 0.7791206425198414, + 1.027300821097942, + 1.0852967767843558, + 1.1373931038287304, + 0.39606302873846644, + 0.6147137996701415, + 0.8579667949563481, + 1.0787382001191392, + 0.5351203845595369, + 0.7644417764693382, + 0.9529330613052628, + 1.1068134014017603, + 0.34315441686615866, + 0.5532746224005062, + 0.7604080408755028, + 1.002560383658593, + 0.4446963847865978, + 0.7017857466618312, + 0.9795906352864036, + 1.273929260081485, + 0.4760589447599722, + 0.8385446579219231, + 1.1398508231409052, + 1.5222095635907187, + 0.5581559424375759, + 1.1479678975054675, + 1.7780180021579288, + 2.632420651384215 + ], + "population_cv": 0.4799254188953803 + }, + "bits_per_byte": 2.0568944461408507, + "branch_output_rms_by_sublayer": [ + 0.04428068920969963, + 0.06905768811702728, + 0.08325625211000443, + 0.017566822469234467, + 0.05574536696076393, + 0.008080621249973774, + 0.028922758996486664, + 0.024165507405996323, + 0.08704349398612976, + 0.007874648086726665, + 0.040835801512002945, + 0.005915080197155476, + 0.036536529660224915, + 0.008644296787679195, + 0.02981100045144558, + 0.015145987272262573, + 0.03913259133696556, + 0.018604939803481102, + 0.02816401980817318, + 0.021070342510938644, + 0.02895236760377884, + 0.02423849329352379, + 0.025148779153823853, + 0.0235623586922884, + 0.049705442041158676, + 0.02636810764670372, + 0.025095606222748756, + 0.02554391883313656, + 0.021526463329792023, + 0.026616260409355164, + 0.023187240585684776, + 0.02496083825826645, + 0.025456786155700684, + 0.02935253642499447, + 0.023355288431048393, + 0.030162764713168144, + 0.024019557982683182, + 0.034848861396312714, + 0.032193440943956375, + 0.042801689356565475, + 0.04221584275364876, + 0.02864120528101921, + 0.04209953919053078, + 0.031113291159272194, + 0.04749658703804016, + 0.03535406291484833, + 0.04878281056880951, + 0.04421490430831909, + 0.03319414332509041, + 0.040683943778276443, + 0.03629270941019058, + 0.046074315905570984, + 0.019516972824931145, + 0.0515168122947216, + 0.012430308386683464, + 0.06739771366119385, + 0.012108913622796535, + 0.06115702539682388, + 0.016566216945648193, + 0.07422584295272827, + 0.00978229008615017, + 0.08158782869577408, + 0.016065498813986778, + 0.10936928540468216 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0003099006677467289, + 0.00019251975954636302, + 0.00016234234548656966, + 0.00015942387928980208, + 0.0001968584249845078, + 0.00018475217710340694, + 0.00015059984644194012, + 0.00016548501600483054, + 0.00023038955568986693, + 0.000174508204188817, + 0.00019883164995662905, + 0.0001816611221872913, + 0.00017861928642125925, + 0.00015891385400111546, + 0.00016629428678998295, + 0.0001735024350399286, + 0.0002145104699341165, + 0.0002208840832563157, + 0.00022878616342617605, + 0.0002424553362605159, + 0.00019381362196277442, + 0.00021058458056910789, + 0.00020820187746302355, + 0.00022492230132131494, + 0.0001978195038098712, + 0.00019023825229132478, + 0.00017365959259255193, + 0.00017398786947606917, + 0.00017082183175405147, + 0.00016498730541231256, + 0.00015893275079449973, + 0.00016573001864886162 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00019023526457551864, + "first_to_last_ratio": 1.0900351320714095, + "imbalance_abs_log_ratio": 0.0862099269798103, + "last_quartile_mean": 0.0001745221405974428, + "mean": 0.00019140431468287272, + "normalized": [ + 1.619089247074635, + 1.0058276892312403, + 0.8481645032691439, + 0.8329168522347197, + 1.028495231733264, + 0.9652456236919876, + 0.7868153165275334, + 0.8645835193371348, + 1.2036800532505587, + 0.9117255505862031, + 1.0388044297019232, + 0.949096275537339, + 0.933204074930096, + 0.8302522033759326, + 0.8688115890464998, + 0.9064708668004441, + 1.1207190929291593, + 1.1540183073839605, + 1.1953030620299194, + 1.2667182381036017, + 1.0125875285721408, + 1.1002081166143716, + 1.0877595826822493, + 1.1751161497794673, + 1.033516429018998, + 0.9939078573360276, + 0.9072919431324156, + 0.9090070397020051, + 0.8924659406820413, + 0.8619832091333518, + 0.8303509304783785, + 0.8658635460932559 + ], + "population_cv": 0.16810108492691056 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.42615652084350586, + "mean_weights": [ + 0.8330706357955933, + 0.16692936420440674 + ], + "sources": 2 + }, + { + "entropy_mean": 0.632439136505127, + "mean_weights": [ + 0.574067234992981, + 0.42593270540237427 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5968352556228638, + "mean_weights": [ + 0.6922775506973267, + 0.30772244930267334 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6816465854644775, + "mean_weights": [ + 0.511958122253418, + 0.48804187774658203 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6602560877799988, + "mean_weights": [ + 0.43693915009498596, + 0.5630608797073364 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6903396248817444, + "mean_weights": [ + 0.4911279082298279, + 0.5088720917701721 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5757448673248291, + "mean_weights": [ + 0.7224498987197876, + 0.2775501012802124 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6877007484436035, + "mean_weights": [ + 0.5311651825904846, + 0.4688348174095154 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0505623817443848, + "mean_weights": [ + 0.2723766565322876, + 0.2802855968475342, + 0.44733771681785583 + ], + "sources": 3 + }, + { + "entropy_mean": 1.09505295753479, + "mean_weights": [ + 0.33145591616630554, + 0.316012978553772, + 0.3525311350822449 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0268163681030273, + "mean_weights": [ + 0.26195770502090454, + 0.2496340572834015, + 0.48840826749801636 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0833425521850586, + "mean_weights": [ + 0.3501453399658203, + 0.26048338413238525, + 0.38937127590179443 + ], + "sources": 3 + }, + { + "entropy_mean": 1.02703058719635, + "mean_weights": [ + 0.31399691104888916, + 0.20795190334320068, + 0.47805115580558777 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0824486017227173, + "mean_weights": [ + 0.33737611770629883, + 0.26119208335876465, + 0.4014317989349365 + ], + "sources": 3 + }, + { + "entropy_mean": 1.038745403289795, + "mean_weights": [ + 0.28605639934539795, + 0.2355905920267105, + 0.47835299372673035 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0876357555389404, + "mean_weights": [ + 0.38722527027130127, + 0.27462828159332275, + 0.33814650774002075 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3472213745117188, + "mean_weights": [ + 0.2523924708366394, + 0.1665031909942627, + 0.24406279623508453, + 0.3370415270328522 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3759691715240479, + "mean_weights": [ + 0.284850150346756, + 0.21094805002212524, + 0.2276441603899002, + 0.27655768394470215 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3601387739181519, + "mean_weights": [ + 0.2508823275566101, + 0.19099071621894836, + 0.23052498698234558, + 0.32760196924209595 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3788524866104126, + "mean_weights": [ + 0.2756692171096802, + 0.22255206108093262, + 0.22525227069854736, + 0.27652645111083984 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3459397554397583, + "mean_weights": [ + 0.2820647954940796, + 0.18945345282554626, + 0.19558577239513397, + 0.33289599418640137 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3762011528015137, + "mean_weights": [ + 0.27166587114334106, + 0.21332880854606628, + 0.23148927092552185, + 0.2835160493850708 + ], + "sources": 4 + }, + { + "entropy_mean": 1.372815489768982, + "mean_weights": [ + 0.2576539218425751, + 0.21029910445213318, + 0.22968199849128723, + 0.30236494541168213 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3816332817077637, + "mean_weights": [ + 0.27266350388526917, + 0.23081988096237183, + 0.23810699582099915, + 0.25840961933135986 + ], + "sources": 4 + }, + { + "entropy_mean": 1.584712266921997, + "mean_weights": [ + 0.17129909992218018, + 0.1638004034757614, + 0.18086029589176178, + 0.2658728361129761, + 0.21816737949848175 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5913515090942383, + "mean_weights": [ + 0.19420018792152405, + 0.16208960115909576, + 0.16682003438472748, + 0.22261866927146912, + 0.2542715072631836 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5923999547958374, + "mean_weights": [ + 0.20701661705970764, + 0.15537989139556885, + 0.17685483396053314, + 0.24083195626735687, + 0.2199167013168335 + ], + "sources": 5 + }, + { + "entropy_mean": 1.576154112815857, + "mean_weights": [ + 0.2061004340648651, + 0.18280217051506042, + 0.13626328110694885, + 0.1920376718044281, + 0.2827964425086975 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5948741436004639, + "mean_weights": [ + 0.1869676411151886, + 0.16341543197631836, + 0.18965134024620056, + 0.2247822880744934, + 0.23518332839012146 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5539796352386475, + "mean_weights": [ + 0.20197400450706482, + 0.1733989119529724, + 0.1211395114660263, + 0.19084404408931732, + 0.31264352798461914 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5945011377334595, + "mean_weights": [ + 0.1782800406217575, + 0.16835741698741913, + 0.19944429397583008, + 0.2215910106897354, + 0.23232722282409668 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5696406364440918, + "mean_weights": [ + 0.20859265327453613, + 0.18552830815315247, + 0.12435021251440048, + 0.19001242518424988, + 0.29151642322540283 + ], + "sources": 5 + }, + { + "entropy_mean": 1.772887110710144, + "mean_weights": [ + 0.13830670714378357, + 0.13491110503673553, + 0.16653329133987427, + 0.2025756984949112, + 0.1927829086780548, + 0.16489030420780182 + ], + "sources": 6 + }, + { + "entropy_mean": 1.761691927909851, + "mean_weights": [ + 0.16438786685466766, + 0.15245124697685242, + 0.10788099467754364, + 0.14674654603004456, + 0.21106326580047607, + 0.21747006475925446 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7550532817840576, + "mean_weights": [ + 0.12169716507196426, + 0.128048375248909, + 0.16509106755256653, + 0.18748882412910461, + 0.1642613261938095, + 0.2334132194519043 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7598530054092407, + "mean_weights": [ + 0.16286706924438477, + 0.17924869060516357, + 0.1153748631477356, + 0.129547119140625, + 0.1784125715494156, + 0.23454967141151428 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7269670963287354, + "mean_weights": [ + 0.11688845604658127, + 0.12103935331106186, + 0.16403436660766602, + 0.1808641254901886, + 0.14800679683685303, + 0.26916688680648804 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7642854452133179, + "mean_weights": [ + 0.16674157977104187, + 0.17305120825767517, + 0.11254861950874329, + 0.1384996920824051, + 0.18324747681617737, + 0.22591140866279602 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7023617029190063, + "mean_weights": [ + 0.1129877045750618, + 0.12577298283576965, + 0.16877974569797516, + 0.18294131755828857, + 0.13385023176670074, + 0.27566802501678467 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.049060106277466, + "mean_weights": [ + 0.10294582694768906, + 0.1363876760005951, + 0.13859772682189941, + 0.14186465740203857, + 0.11174861341714859, + 0.12047471106052399, + 0.1174238920211792, + 0.13055691123008728 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1108832359313965, + "mean_weights": [ + 0.08000192046165466, + 0.13340924680233002, + 0.12515409290790558, + 0.1369815170764923, + 0.08619116991758347, + 0.11378533393144608, + 0.126826673746109, + 0.1294378638267517, + 0.06821217387914658 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1406984329223633, + "mean_weights": [ + 0.09945221245288849, + 0.12643511593341827, + 0.11737721413373947, + 0.12090113759040833, + 0.11926636099815369, + 0.11251920461654663, + 0.0943274050951004, + 0.1052723303437233, + 0.10444900393486023 + ], + "sources": 9 + }, + { + "entropy_mean": 2.148448944091797, + "mean_weights": [ + 0.0900847539305687, + 0.12191049754619598, + 0.11876705288887024, + 0.12454263120889664, + 0.08499738574028015, + 0.09320119023323059, + 0.12477506697177887, + 0.1425188183784485, + 0.09920259565114975 + ], + "sources": 9 + }, + { + "entropy_mean": 2.161719560623169, + "mean_weights": [ + 0.09516263008117676, + 0.10011610388755798, + 0.1287383884191513, + 0.12565559148788452, + 0.09757968783378601, + 0.10652302205562592, + 0.10644830763339996, + 0.12631137669086456, + 0.11346490681171417 + ], + "sources": 9 + }, + { + "entropy_mean": 2.119661808013916, + "mean_weights": [ + 0.08648113906383514, + 0.12600350379943848, + 0.11755797266960144, + 0.12744447588920593, + 0.09305394440889359, + 0.10885848850011826, + 0.12268061190843582, + 0.13150592148303986, + 0.08641393482685089 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1674675941467285, + "mean_weights": [ + 0.1073547974228859, + 0.10432218760251999, + 0.1259293109178543, + 0.12246961891651154, + 0.10322894155979156, + 0.10405035316944122, + 0.09358370304107666, + 0.11424382030963898, + 0.12481726706027985 + ], + "sources": 9 + }, + { + "entropy_mean": 2.145291328430176, + "mean_weights": [ + 0.08652392029762268, + 0.12970152497291565, + 0.12882786989212036, + 0.1353725790977478, + 0.09321916848421097, + 0.10797271132469177, + 0.12089314311742783, + 0.11848876625299454, + 0.0790003091096878 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05579930171370506, + 0.0479312464594841, + 0.05494503676891327, + 0.06528861820697784, + 0.0840456411242485, + 0.11281505227088928, + 0.10675167292356491, + 0.07452012598514557, + 0.10566704720258713, + 0.0815395936369896, + 0.08493325859308243, + 0.08932148665189743, + 0.08518004417419434, + 0.0852142944931984, + 0.0869225263595581, + 0.08833882957696915, + 0.08420372009277344, + 0.053619373589754105, + 0.0599745474755764, + 0.05425184220075607, + 0.059648532420396805, + 0.050102993845939636, + 0.05703767016530037, + 0.054093118757009506, + 0.059165339916944504, + 0.0417165607213974, + 0.042090870440006256, + 0.04156746342778206, + 0.04672161117196083, + 0.04423058032989502, + 0.05085216090083122, + 0.046465035527944565, + 0.05271224305033684, + 0.040062375366687775, + 0.04212570935487747, + 0.03875921666622162, + 0.044668592512607574, + 0.039916280657052994, + 0.04601491615176201, + 0.04390547052025795, + 0.0448390431702137, + 0.038541000336408615, + 0.03928724303841591, + 0.03931433707475662, + 0.04000399634242058, + 0.0401446633040905, + 0.04094685614109039, + 0.04002338647842407, + 0.04136248677968979, + 0.036048565059900284, + 0.03709477186203003, + 0.03701528534293175, + 0.03859051316976547, + 0.038552187383174896, + 0.04068531468510628, + 0.04065614938735962, + 0.04580239951610565, + 0.04586269333958626, + 0.04387181997299194, + 0.04672257974743843, + 0.04973924160003662, + 0.05108017101883888, + 0.057408981025218964, + 0.05135168135166168 + ], + "loss_nats": 1.425730586051941, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.974043846130371, + "mean_weights": [ + 0.11710391938686371, + 0.09109088778495789, + 0.08311391621828079, + 0.06965755671262741, + 0.06475193798542023, + 0.057108279317617416, + 0.07776540517807007, + 0.16933506727218628, + 0.2700730562210083 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.04428068920969963, + 0.08424261212348938, + 0.14145584404468536, + 0.14948369562625885, + 0.18736635148525238, + 0.1897522509098053, + 0.1932709813117981, + 0.20237113535404205, + 0.08704349398612976, + 0.08847226202487946, + 0.11572060734033585, + 0.11665411293506622, + 0.12246368825435638, + 0.12323978543281555, + 0.12688620388507843, + 0.12915553152561188, + 0.03913259133696556, + 0.044974539428949356, + 0.05960690602660179, + 0.06980320811271667, + 0.08649054914712906, + 0.09742555767297745, + 0.10991228371858597, + 0.12249503284692764, + 0.049705442041158676, + 0.06076505780220032, + 0.07420338690280914, + 0.08680541813373566, + 0.09472409635782242, + 0.10820935666561127, + 0.11319252103567123, + 0.12568308413028717, + 0.025456786155700684, + 0.03896655514836311, + 0.046594005078077316, + 0.06282654404640198, + 0.06837199628353119, + 0.08634737133979797, + 0.09425382316112518, + 0.11384473741054535, + 0.04221584275364876, + 0.05049705132842064, + 0.06848613172769547, + 0.07969057559967041, + 0.0981777086853981, + 0.11123643070459366, + 0.12620587646961212, + 0.14465975761413574, + 0.03319414332509041, + 0.05405839532613754, + 0.07025149464607239, + 0.09522009640932083, + 0.09903644025325775, + 0.12943461537361145, + 0.1303461343050003, + 0.17285297811031342, + 0.012108913622796535, + 0.06338083744049072, + 0.06732851266860962, + 0.1303563416004181, + 0.13211798667907715, + 0.20190104842185974, + 0.2041558176279068, + 0.2989218831062317 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.862539496272802e-05, + 8.138753764797002e-05, + 8.071825868682936e-05, + 7.701994036324322e-05, + 9.188598050968722e-05, + 9.633715671952814e-05, + 8.899183012545109e-05, + 8.30965000204742e-05, + 8.557196269975975e-05, + 8.086617890512571e-05, + 7.539002399425954e-05, + 7.190550240920857e-05, + 6.578989996341988e-05, + 6.378156831488013e-05, + 6.0346741520334035e-05, + 5.965661330264993e-05, + 7.128302968339995e-05, + 6.638480408582836e-05, + 5.957340181339532e-05, + 5.343910015653819e-05, + 5.4122610890772194e-05, + 5.026680082664825e-05, + 4.669557529268786e-05, + 4.3520500184968114e-05, + 4.683015504269861e-05, + 4.395980795379728e-05, + 4.15727736253757e-05, + 3.961606853408739e-05, + 3.7589103158097714e-05, + 3.3854801586130634e-05, + 3.1459927413379773e-05, + 3.07957197946962e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 8.475782487948891e-05, + "first_to_last_ratio": 2.218222465765737, + "imbalance_abs_log_ratio": 0.796706184063345, + "last_quartile_mean": 3.820979463853291e-05, + "mean": 6.226047719337657e-05, + "normalized": [ + 1.26284600611915, + 1.3072103092815395, + 1.2964606492835613, + 1.2370599108006322, + 1.4758316134374614, + 1.5473244193151916, + 1.4293470615242612, + 1.3346589002583844, + 1.374418677100392, + 1.2988364778182018, + 1.2108809214568583, + 1.1549140907782514, + 1.0566880134741206, + 1.0244310867836615, + 0.9692624316530917, + 0.9581779002008091, + 1.144916211644186, + 1.0662430980032795, + 0.9568413943947878, + 0.8583149787072814, + 0.8692932230935405, + 0.8073629225571654, + 0.7500034917441246, + 0.699006852289239, + 0.7521650516305474, + 0.7060628176244341, + 0.6677233374915141, + 0.6362956135244953, + 0.6037394002193183, + 0.5437607148589634, + 0.5052953146450757, + 0.49462710828648004 + ], + "population_cv": 0.3033546543779208 + }, + "activation_output_rms_by_block": [ + 0.1145050972700119, + 0.20400165021419525, + 0.2456863969564438, + 0.2577275037765503, + 0.10259918123483658, + 0.12039783596992493, + 0.12886880338191986, + 0.13730759918689728, + 0.05411752685904503, + 0.07760744541883469, + 0.11189539730548859, + 0.15223495662212372, + 0.06856004893779755, + 0.09863153845071793, + 0.12596677243709564, + 0.15035380423069, + 0.0526665560901165, + 0.0829758569598198, + 0.11263782531023026, + 0.1492708921432495, + 0.06997588276863098, + 0.11031109094619751, + 0.15111951529979706, + 0.1924203783273697, + 0.07272393256425858, + 0.12375207245349884, + 0.16444069147109985, + 0.21152915060520172, + 0.07600132375955582, + 0.1524338275194168, + 0.23548460006713867, + 0.36000555753707886 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.16388675849884748, + "first_to_last_ratio": 0.9389294976328925, + "imbalance_abs_log_ratio": 0.06301488498578389, + "last_quartile_mean": 0.17454639449715614, + "mean": 0.13963158475235105, + "normalized": [ + 0.8200515483162123, + 1.4609993188576225, + 1.7595331127422948, + 1.845767948803652, + 0.7347849085635266, + 0.8622535952983784, + 0.9229201517012077, + 0.9833563045955859, + 0.3875736779398962, + 0.5558015083512684, + 0.8013616511188708, + 1.0902616115983061, + 0.4910067379052873, + 0.706369827612067, + 0.9021366667184136, + 1.0767893560569104, + 0.3771822556015911, + 0.5942484797188602, + 0.8066787003098431, + 1.06903386084169, + 0.5011465199133806, + 0.7900153188252069, + 1.0822731516498998, + 1.3780576842169645, + 0.5208272375712193, + 0.8862756422408589, + 1.177675464779333, + 1.5149090442564794, + 0.5442989413487707, + 1.0916858659863502, + 1.6864708689283403, + 2.5782530376317117 + ], + "population_cv": 0.47984863275066025 + }, + "bits_per_byte": 1.911507192407467, + "branch_output_rms_by_sublayer": [ + 0.06405020505189896, + 0.09185103327035904, + 0.11919604241847992, + 0.013598980382084846, + 0.05750352889299393, + 0.01212528720498085, + 0.030115468427538872, + 0.027524013072252274, + 0.10046301037073135, + 0.007296515628695488, + 0.03326759114861488, + 0.004597592167556286, + 0.031032785773277283, + 0.009118573740124702, + 0.030526647344231606, + 0.01685192435979843, + 0.04817948490381241, + 0.022456368431448936, + 0.02888304367661476, + 0.02516156993806362, + 0.038138486444950104, + 0.030369389802217484, + 0.038701534271240234, + 0.032453518360853195, + 0.05024706944823265, + 0.03674135357141495, + 0.022620366886258125, + 0.03459346666932106, + 0.026417911052703857, + 0.036538660526275635, + 0.02896822616457939, + 0.03523407503962517, + 0.033122286200523376, + 0.03982815146446228, + 0.030214382335543633, + 0.0424463264644146, + 0.03139881789684296, + 0.050580643117427826, + 0.04130503162741661, + 0.06169170141220093, + 0.059053678065538406, + 0.039659008383750916, + 0.05973989889025688, + 0.04409486800432205, + 0.06291545927524567, + 0.05282061919569969, + 0.06124354153871536, + 0.06524951010942459, + 0.04323358088731766, + 0.055994197726249695, + 0.04760046675801277, + 0.06301165372133255, + 0.027900392189621925, + 0.06689292937517166, + 0.01992829516530037, + 0.08155854046344757, + 0.01713724248111248, + 0.07299749553203583, + 0.020643312484025955, + 0.09036710113286972, + 0.016673220321536064, + 0.09852731227874756, + 0.021567966789007187, + 0.14499031007289886 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0002966618437338653, + 0.00016634094453633522, + 0.00014678877446021245, + 0.00014750843291343408, + 0.00017692919762533904, + 0.0001238570113154357, + 0.00013555657170375067, + 0.00015406953821101864, + 0.000261349158605723, + 0.0001874697778729113, + 0.00021615986598471135, + 0.0002230412810554774, + 0.00021009942027959264, + 0.00019544973908339048, + 0.00020978657549958288, + 0.00022637997547432655, + 0.00027974087803096754, + 0.0002937229975156036, + 0.00030290679309222284, + 0.00031471324570394205, + 0.0002492733465455421, + 0.00026877074127662633, + 0.00026686141946371123, + 0.00028132011174265663, + 0.0002545019737336784, + 0.0002442004776846219, + 0.00022397266343454397, + 0.00022201151271312948, + 0.0002179110765659919, + 0.0002035538002201437, + 0.00019567059416389953, + 0.00019552235378590295 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0001684640393124239, + "first_to_last_ratio": 0.7669027621386625, + "imbalance_abs_log_ratio": 0.26539526252441736, + "last_quartile_mean": 0.000219668056537739, + "mean": 0.00022162819043838408, + "normalized": [ + 1.3385564496423648, + 0.7505405526585323, + 0.6623199610566775, + 0.6655671041741579, + 0.7983154005605978, + 0.5588504380712787, + 0.6116395727259136, + 0.6951712140331365, + 1.1792234466597873, + 0.8458751456757062, + 0.9753265843895748, + 1.0063759516075017, + 0.9479814813450069, + 0.8818812205107472, + 0.9465699064934913, + 1.0214403457725454, + 1.2622080136901161, + 1.325296195103226, + 1.3667340444961824, + 1.4200054834244427, + 1.1247366413653221, + 1.2127100832479547, + 1.2040951060235392, + 1.2693336131391995, + 1.148328528199727, + 1.1018475456646084, + 1.010578406075159, + 1.0017295736340543, + 0.9832281540311291, + 0.9184472418310643, + 0.8828777321912884, + 0.8822088625059683 + ], + "population_cv": 0.22951974950717094 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.2742295563220978, + "mean_weights": [ + 0.913676381111145, + 0.08632358908653259 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5765758156776428, + "mean_weights": [ + 0.6629496216773987, + 0.3370503783226013 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6225830316543579, + "mean_weights": [ + 0.6586993932723999, + 0.3413006365299225 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6831302642822266, + "mean_weights": [ + 0.5191006064414978, + 0.4808993339538574 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5954421162605286, + "mean_weights": [ + 0.32184141874313354, + 0.6781585216522217 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6894387006759644, + "mean_weights": [ + 0.4925844669342041, + 0.5074155330657959 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5799020528793335, + "mean_weights": [ + 0.7165489196777344, + 0.2834510803222656 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6812429428100586, + "mean_weights": [ + 0.5621760487556458, + 0.43782392144203186 + ], + "sources": 2 + }, + { + "entropy_mean": 0.9412254095077515, + "mean_weights": [ + 0.18928885459899902, + 0.22454679012298584, + 0.5861644148826599 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0920023918151855, + "mean_weights": [ + 0.3360068202018738, + 0.2963274121284485, + 0.36766576766967773 + ], + "sources": 3 + }, + { + "entropy_mean": 0.8953218460083008, + "mean_weights": [ + 0.1661311388015747, + 0.20686066150665283, + 0.6270081996917725 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0742058753967285, + "mean_weights": [ + 0.3712955713272095, + 0.2407427877187729, + 0.38796162605285645 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0043683052062988, + "mean_weights": [ + 0.2846021354198456, + 0.20251725614070892, + 0.5128806233406067 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0807768106460571, + "mean_weights": [ + 0.3476581275463104, + 0.2575277090072632, + 0.394814133644104 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0317838191986084, + "mean_weights": [ + 0.27654269337654114, + 0.23612691462039948, + 0.4873303771018982 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0801410675048828, + "mean_weights": [ + 0.41122618317604065, + 0.2597043514251709, + 0.32906943559646606 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3381030559539795, + "mean_weights": [ + 0.26073575019836426, + 0.15431058406829834, + 0.24623697996139526, + 0.33871668577194214 + ], + "sources": 4 + }, + { + "entropy_mean": 1.363797664642334, + "mean_weights": [ + 0.30910205841064453, + 0.19983994960784912, + 0.20934709906578064, + 0.2817108631134033 + ], + "sources": 4 + }, + { + "entropy_mean": 1.360697865486145, + "mean_weights": [ + 0.2628037631511688, + 0.18781600892543793, + 0.24645090103149414, + 0.3029293119907379 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3685510158538818, + "mean_weights": [ + 0.29796913266181946, + 0.2099475860595703, + 0.20821179449558258, + 0.28387147188186646 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3320616483688354, + "mean_weights": [ + 0.30047816038131714, + 0.17871183156967163, + 0.1961410641670227, + 0.3246689438819885 + ], + "sources": 4 + }, + { + "entropy_mean": 1.370679259300232, + "mean_weights": [ + 0.28218066692352295, + 0.20532113313674927, + 0.22602522373199463, + 0.28647300601005554 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3708806037902832, + "mean_weights": [ + 0.2708446979522705, + 0.20370902121067047, + 0.23316249251365662, + 0.2922837734222412 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3792238235473633, + "mean_weights": [ + 0.27852123975753784, + 0.2307901829481125, + 0.21968649327754974, + 0.2710020840167999 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5753511190414429, + "mean_weights": [ + 0.17311285436153412, + 0.14763447642326355, + 0.18770167231559753, + 0.26439449191093445, + 0.22715651988983154 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5760574340820312, + "mean_weights": [ + 0.18803037703037262, + 0.15533128380775452, + 0.15620622038841248, + 0.20726050436496735, + 0.29317164421081543 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5830810070037842, + "mean_weights": [ + 0.21359023451805115, + 0.14659720659255981, + 0.1659994125366211, + 0.2256123423576355, + 0.24820077419281006 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5701727867126465, + "mean_weights": [ + 0.200425386428833, + 0.18137116730213165, + 0.13592247664928436, + 0.17893560230731964, + 0.30334535241127014 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5793086290359497, + "mean_weights": [ + 0.18724045157432556, + 0.1444673240184784, + 0.17262053489685059, + 0.23074325919151306, + 0.2649284601211548 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5522117614746094, + "mean_weights": [ + 0.20369622111320496, + 0.1713925004005432, + 0.12165851145982742, + 0.17937912046909332, + 0.3238736689090729 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5786654949188232, + "mean_weights": [ + 0.1807911992073059, + 0.14386388659477234, + 0.18020474910736084, + 0.22963666915893555, + 0.26550352573394775 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5799694061279297, + "mean_weights": [ + 0.21369889378547668, + 0.19715666770935059, + 0.13811513781547546, + 0.17161031067371368, + 0.2794190049171448 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7675673961639404, + "mean_weights": [ + 0.13910025358200073, + 0.12387993186712265, + 0.1596122682094574, + 0.1938106119632721, + 0.19675728678703308, + 0.18683964014053345 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7553372383117676, + "mean_weights": [ + 0.15784114599227905, + 0.1533873826265335, + 0.1110420823097229, + 0.1374262571334839, + 0.1936206966638565, + 0.24668243527412415 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7378818988800049, + "mean_weights": [ + 0.1249084323644638, + 0.11051411926746368, + 0.15597975254058838, + 0.18690134584903717, + 0.16355964541435242, + 0.25813671946525574 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7580437660217285, + "mean_weights": [ + 0.16046461462974548, + 0.17262649536132812, + 0.11904540657997131, + 0.1273362636566162, + 0.17616048455238342, + 0.24436676502227783 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7062923908233643, + "mean_weights": [ + 0.12019055336713791, + 0.11017680168151855, + 0.15525341033935547, + 0.17838828265666962, + 0.14404630661010742, + 0.29194462299346924 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7705397605895996, + "mean_weights": [ + 0.16944682598114014, + 0.18135881423950195, + 0.11809040606021881, + 0.13652509450912476, + 0.17867791652679443, + 0.2159009575843811 + ], + "sources": 6 + }, + { + "entropy_mean": 1.681544303894043, + "mean_weights": [ + 0.11871509253978729, + 0.11930644512176514, + 0.1618763506412506, + 0.1804599016904831, + 0.12120576947927475, + 0.2984364628791809 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0414016246795654, + "mean_weights": [ + 0.10252167284488678, + 0.13682352006435394, + 0.12705300748348236, + 0.1565057933330536, + 0.11857448518276215, + 0.11903785169124603, + 0.11327946186065674, + 0.1262042224407196 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0913989543914795, + "mean_weights": [ + 0.08215485513210297, + 0.12418252974748611, + 0.12314213812351227, + 0.12776142358779907, + 0.07834012806415558, + 0.11542008817195892, + 0.12361127138137817, + 0.14818212389945984, + 0.07720542699098587 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1283392906188965, + "mean_weights": [ + 0.1007528007030487, + 0.12220022082328796, + 0.11355189979076385, + 0.12810677289962769, + 0.12657296657562256, + 0.11733594536781311, + 0.0922354906797409, + 0.0985100269317627, + 0.10073386877775192 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1272106170654297, + "mean_weights": [ + 0.09455966204404831, + 0.11855152249336243, + 0.1109207421541214, + 0.11731322109699249, + 0.07864771038293839, + 0.09181208163499832, + 0.10917074978351593, + 0.17144736647605896, + 0.10757693648338318 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1414804458618164, + "mean_weights": [ + 0.08938497304916382, + 0.10802341997623444, + 0.14467450976371765, + 0.15281793475151062, + 0.09899453818798065, + 0.10280055552721024, + 0.09853018075227737, + 0.10232935100793839, + 0.10244454443454742 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0827794075012207, + "mean_weights": [ + 0.08792929351329803, + 0.1262941211462021, + 0.10903117060661316, + 0.13112589716911316, + 0.08125819265842438, + 0.10460717231035233, + 0.10944748669862747, + 0.148950457572937, + 0.10135620832443237 + ], + "sources": 9 + }, + { + "entropy_mean": 2.150519847869873, + "mean_weights": [ + 0.10761173069477081, + 0.10341285169124603, + 0.12667715549468994, + 0.122761070728302, + 0.10801324248313904, + 0.10069069266319275, + 0.0908002033829689, + 0.10846143960952759, + 0.13157162070274353 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1415891647338867, + "mean_weights": [ + 0.08916807174682617, + 0.12812913954257965, + 0.12285787612199783, + 0.13691258430480957, + 0.08769381046295166, + 0.10658255964517593, + 0.11311973631381989, + 0.1256583333015442, + 0.08987787365913391 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05920173227787018, + 0.05499904230237007, + 0.06358218193054199, + 0.08561401814222336, + 0.10901069641113281, + 0.16725511848926544, + 0.13591907918453217, + 0.09046964347362518, + 0.12571845948696136, + 0.0983843132853508, + 0.10263820737600327, + 0.10806161910295486, + 0.09654191136360168, + 0.09967107325792313, + 0.10075473040342331, + 0.10373975336551666, + 0.09667752683162689, + 0.0636601597070694, + 0.07084812968969345, + 0.06919264048337936, + 0.0705157145857811, + 0.0605684369802475, + 0.06775520741939545, + 0.06408503651618958, + 0.06888782978057861, + 0.047689590603113174, + 0.04722585156559944, + 0.04665428772568703, + 0.052786171436309814, + 0.05025629326701164, + 0.057373952120542526, + 0.05325891822576523, + 0.060849934816360474, + 0.04571268707513809, + 0.04833834990859032, + 0.04494747519493103, + 0.05216481536626816, + 0.04816780984401703, + 0.05495777726173401, + 0.054556217044591904, + 0.053651586174964905, + 0.04636868089437485, + 0.04721657931804657, + 0.047502607107162476, + 0.04846424236893654, + 0.04893815144896507, + 0.05001019313931465, + 0.04957285150885582, + 0.05134204775094986, + 0.044955432415008545, + 0.046151302754879, + 0.04623876512050629, + 0.04794751852750778, + 0.048040587455034256, + 0.05025666579604149, + 0.050303831696510315, + 0.0556449256837368, + 0.05836831033229828, + 0.05197414755821228, + 0.059859104454517365, + 0.05537183955311775, + 0.06453308463096619, + 0.06757453083992004, + 0.06288585066795349 + ], + "loss_nats": 1.3249558210372925, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.9470667839050293, + "mean_weights": [ + 0.10390135645866394, + 0.0945686548948288, + 0.08364264667034149, + 0.06801718473434448, + 0.06987068057060242, + 0.05586811900138855, + 0.06852905452251434, + 0.16156184673309326, + 0.2940404415130615 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.06405020505189896, + 0.1145050972700119, + 0.19869989156723022, + 0.20400165021419525, + 0.24153423309326172, + 0.2456863969564438, + 0.24799224734306335, + 0.2577275037765503, + 0.10046301037073135, + 0.10259918123483658, + 0.11955195665359497, + 0.12039783596992493, + 0.1281825453042984, + 0.12886880338191986, + 0.1352226734161377, + 0.13730759918689728, + 0.04817948490381241, + 0.05411752685904503, + 0.06602110713720322, + 0.07760744541883469, + 0.09840337932109833, + 0.11189539730548859, + 0.13363297283649445, + 0.15223495662212372, + 0.05024706944823265, + 0.06856004893779755, + 0.0795469731092453, + 0.09863153845071793, + 0.10624490678310394, + 0.12596677243709564, + 0.1323748230934143, + 0.15035380423069, + 0.033122286200523376, + 0.0526665560901165, + 0.06185898557305336, + 0.0829758569598198, + 0.08869928866624832, + 0.11263782531023026, + 0.12269332259893417, + 0.1492708921432495, + 0.059053678065538406, + 0.06997588276863098, + 0.09593892097473145, + 0.11031109094619751, + 0.13435089588165283, + 0.15111951529979706, + 0.16890938580036163, + 0.1924203783273697, + 0.04323358088731766, + 0.07272393256425858, + 0.09255246073007584, + 0.12375207245349884, + 0.13024720549583435, + 0.16444069147109985, + 0.1665765792131424, + 0.21152915060520172, + 0.01713724248111248, + 0.07600132375955582, + 0.08019699156284332, + 0.1524338275194168, + 0.15484170615673065, + 0.23548460006713867, + 0.23777693510055542, + 0.36000555753707886 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00013671374472323805, + 0.00014061683032196015, + 0.00014011880557518452, + 0.0001346576027572155, + 0.00016767386114224792, + 0.00017389525601174682, + 0.000161304313223809, + 0.0001517484342912212, + 0.0001599940878804773, + 0.00015165377408266068, + 0.00014070542238187045, + 0.0001329357037320733, + 0.00012773896742146462, + 0.0001227871689479798, + 0.00011468595039332286, + 0.0001126774150179699, + 0.00013911975838709623, + 0.0001255871611647308, + 0.00010900205961661413, + 9.423705341760069e-05, + 9.548493835609406e-05, + 8.807553967926651e-05, + 8.132647053571418e-05, + 7.46887453715317e-05, + 7.960261427797377e-05, + 7.460140477633104e-05, + 7.036523311398923e-05, + 6.68716529617086e-05, + 6.92761386744678e-05, + 5.7790286518866196e-05, + 5.073001011624001e-05, + 4.773967884830199e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.0001508411060058279, + "first_to_last_ratio": 2.3342021076852864, + "imbalance_abs_log_ratio": 0.8476701229540182, + "last_quartile_mean": 6.462212741098483e-05, + "mean": 0.00011232519011628028, + "normalized": [ + 1.2171245344139678, + 1.2518726224846988, + 1.2474388463543393, + 1.1988192730216298, + 1.4927538601863934, + 1.5481412124184124, + 1.4360475424686572, + 1.3509742038640624, + 1.4243829697937715, + 1.3501314702932352, + 1.2526613330118617, + 1.1834896837873747, + 1.1372245823864426, + 1.0931400945849206, + 1.0210171936909138, + 1.0031357605655953, + 1.2385446064509475, + 1.1180676483584988, + 0.9704150912522292, + 0.8389663380052635, + 0.8500759113538735, + 0.7841120908683951, + 0.7240270009917109, + 0.6649331756680142, + 0.7086799870587199, + 0.664155606583909, + 0.6264421457123408, + 0.5953397710031234, + 0.6167462401154395, + 0.5144908854286474, + 0.4516352036771426, + 0.4250131141454677 + ], + "population_cv": 0.3243318492260269 + }, + "activation_output_rms_by_block": [ + 0.09945333749055862, + 0.19213460385799408, + 0.22797581553459167, + 0.23613907396793365, + 0.09485343843698502, + 0.10560820996761322, + 0.11283529549837112, + 0.11850933730602264, + 0.04548373818397522, + 0.06377377361059189, + 0.09456658363342285, + 0.13270698487758636, + 0.057267654687166214, + 0.08288402110338211, + 0.10530658811330795, + 0.12915223836898804, + 0.04864993691444397, + 0.07540855556726456, + 0.10244929790496826, + 0.13460960984230042, + 0.06746271997690201, + 0.10431721806526184, + 0.13895606994628906, + 0.17493325471878052, + 0.06875944882631302, + 0.11196905374526978, + 0.14755180478096008, + 0.18564240634441376, + 0.06505608558654785, + 0.12338310480117798, + 0.18688006699085236, + 0.2793145477771759 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.14843863900750875, + "first_to_last_ratio": 1.016218807478792, + "imbalance_abs_log_ratio": 0.016088687661014327, + "last_quartile_mean": 0.14606956485658884, + "mean": 0.12231230863835663, + "normalized": [ + 0.8131098055275401, + 1.5708525658368735, + 1.863882859154041, + 1.930623962516569, + 0.7755019874364417, + 0.8634307629648887, + 0.9225179113575038, + 0.968907697232841, + 0.3718655848321688, + 0.5214011109801969, + 0.7731567222153403, + 1.0849847113095044, + 0.46820843564069003, + 0.677642520414254, + 0.8609647654077903, + 1.0559218430816737, + 0.39775176722637345, + 0.6165246687496237, + 0.8376041548515142, + 1.1005401773610821, + 0.5515611693371797, + 0.8528758816391795, + 1.1360759313042108, + 1.4302179123771537, + 0.5621629547490231, + 0.9154356989232459, + 1.2063528717884773, + 1.5177737090492385, + 0.5318850270327293, + 1.0087546067500648, + 1.5278925651170936, + 2.283617657835493 + ], + "population_cv": 0.45990152454102057 + }, + "bits_per_byte": 1.8677059629551234, + "branch_output_rms_by_sublayer": [ + 0.05879996344447136, + 0.07695004343986511, + 0.12070102244615555, + 0.009311930276453495, + 0.04793894290924072, + 0.011815963312983513, + 0.02440839447081089, + 0.020539281889796257, + 0.09285115450620651, + 0.005813785828649998, + 0.02481892891228199, + 0.003573456546291709, + 0.0236979890614748, + 0.007113786414265633, + 0.025421446189284325, + 0.014093861915171146, + 0.040811773389577866, + 0.01866593025624752, + 0.024119239300489426, + 0.021141231060028076, + 0.034150104969739914, + 0.026980604976415634, + 0.03615512698888779, + 0.030364125967025757, + 0.038965776562690735, + 0.03407087177038193, + 0.018716182559728622, + 0.032044559717178345, + 0.02321203052997589, + 0.034267883747816086, + 0.026723654940724373, + 0.034647371619939804, + 0.029996519908308983, + 0.03736970201134682, + 0.026377608999609947, + 0.04041645675897598, + 0.028714295476675034, + 0.049527641385793686, + 0.03495945781469345, + 0.06098909676074982, + 0.055284686386585236, + 0.039899833500385284, + 0.053873173892498016, + 0.044678881764411926, + 0.054217252880334854, + 0.053162794560194016, + 0.05406007170677185, + 0.06517288833856583, + 0.0389670766890049, + 0.05485546216368675, + 0.03813140466809273, + 0.061326317489147186, + 0.02552611567080021, + 0.0645928829908371, + 0.018847614526748657, + 0.07410307973623276, + 0.01462650392204523, + 0.06278921663761139, + 0.01683272421360016, + 0.0749676302075386, + 0.014945002272725105, + 0.08143441379070282, + 0.01824469305574894, + 0.11297120153903961 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0004491753187892123, + 0.0002466429983270344, + 0.00019878457170446542, + 0.000222356272929301, + 0.00026454874257958515, + 0.00016588889790009986, + 0.00020513542072779866, + 0.0002706446514079606, + 0.00040316318199514406, + 0.0003056668184317147, + 0.000385208138285157, + 0.00040132433616738185, + 0.00040606153778702936, + 0.00037454006028040564, + 0.00039615315334540034, + 0.00041923164923306965, + 0.0005247739099336019, + 0.0005532165561034508, + 0.0005655399368817451, + 0.0005611189391440631, + 0.00044910888639062175, + 0.00047779179340376807, + 0.0004676468860570374, + 0.0004908054047586765, + 0.0004269503319197823, + 0.0004023161076721272, + 0.00037079550322997846, + 0.00036509022881260985, + 0.00037132974775746396, + 0.0003369698469635127, + 0.00030996802925634375, + 0.0002892548326377254 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0002528971092956822, + "first_to_last_ratio": 0.7042833373713037, + "imbalance_abs_log_ratio": 0.35057453593559307, + "last_quartile_mean": 0.00035908432853119294, + "mean": 0.00037741258408791463, + "normalized": [ + 1.1901439902294866, + 0.653510266286143, + 0.5267036131952624, + 0.5891596685008925, + 0.700953687643762, + 0.43954257196007446, + 0.5435309509446944, + 0.7171055307072552, + 1.0682293039312942, + 0.8099009712948856, + 1.0206552577362562, + 1.0633570609132026, + 1.0759088459340858, + 0.9923889029443707, + 1.049655390539708, + 1.1108046390297723, + 1.3904515431084852, + 1.4658137524491879, + 1.4984660308783133, + 1.4867520660449303, + 1.1899679696053964, + 1.2659667789256097, + 1.2390866276682062, + 1.3004479062212406, + 1.1312562164602553, + 1.0659848787087913, + 0.9824672490083298, + 0.967350438764293, + 0.9838827940908464, + 0.8928421074720148, + 0.8212975463058213, + 0.7664154424971327 + ], + "population_cv": 0.28157056603291497 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.2886376678943634, + "mean_weights": [ + 0.9084135293960571, + 0.09158647060394287 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5567276477813721, + "mean_weights": [ + 0.6939359307289124, + 0.30606409907341003 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6638832688331604, + "mean_weights": [ + 0.5756981372833252, + 0.4243018925189972 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6865397691726685, + "mean_weights": [ + 0.490195095539093, + 0.509804904460907 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5604678988456726, + "mean_weights": [ + 0.2793388366699219, + 0.7206612229347229 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6861096620559692, + "mean_weights": [ + 0.4567631483078003, + 0.5432368516921997 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6263248920440674, + "mean_weights": [ + 0.6577913761138916, + 0.342208594083786 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6871790289878845, + "mean_weights": [ + 0.5335646867752075, + 0.4664352536201477 + ], + "sources": 2 + }, + { + "entropy_mean": 0.9065086245536804, + "mean_weights": [ + 0.16351722180843353, + 0.2177291363477707, + 0.6187536120414734 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0887153148651123, + "mean_weights": [ + 0.31462299823760986, + 0.2936253845691681, + 0.39175161719322205 + ], + "sources": 3 + }, + { + "entropy_mean": 0.8672108054161072, + "mean_weights": [ + 0.14541032910346985, + 0.206082284450531, + 0.6485074162483215 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0703508853912354, + "mean_weights": [ + 0.34077826142311096, + 0.23920544981956482, + 0.4200162887573242 + ], + "sources": 3 + }, + { + "entropy_mean": 0.9692573547363281, + "mean_weights": [ + 0.24959897994995117, + 0.19082629680633545, + 0.5595747232437134 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0768523216247559, + "mean_weights": [ + 0.3241707980632782, + 0.25768136978149414, + 0.41814783215522766 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0059022903442383, + "mean_weights": [ + 0.24969778954982758, + 0.22190222144126892, + 0.5284000039100647 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0855510234832764, + "mean_weights": [ + 0.38152068853378296, + 0.26308494806289673, + 0.3553943932056427 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3414515256881714, + "mean_weights": [ + 0.2443043291568756, + 0.15456175804138184, + 0.26582303643226624, + 0.3353108763694763 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3666372299194336, + "mean_weights": [ + 0.290629506111145, + 0.19363881647586823, + 0.22157421708106995, + 0.2941575050354004 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3559900522232056, + "mean_weights": [ + 0.24515581130981445, + 0.18063119053840637, + 0.2550903558731079, + 0.31912264227867126 + ], + "sources": 4 + }, + { + "entropy_mean": 1.370351791381836, + "mean_weights": [ + 0.28074610233306885, + 0.20544829964637756, + 0.21817776560783386, + 0.29562780261039734 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3270303010940552, + "mean_weights": [ + 0.27338331937789917, + 0.17288148403167725, + 0.2010507583618164, + 0.3526844382286072 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3688936233520508, + "mean_weights": [ + 0.26944196224212646, + 0.19457344710826874, + 0.23748193681240082, + 0.2985026240348816 + ], + "sources": 4 + }, + { + "entropy_mean": 1.362016201019287, + "mean_weights": [ + 0.2565782070159912, + 0.18700271844863892, + 0.23452776670455933, + 0.32189127802848816 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3803584575653076, + "mean_weights": [ + 0.2717689573764801, + 0.22202849388122559, + 0.23147040605545044, + 0.2747321128845215 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5673255920410156, + "mean_weights": [ + 0.16662028431892395, + 0.1404871791601181, + 0.18319538235664368, + 0.27637210488319397, + 0.2333250343799591 + ], + "sources": 5 + }, + { + "entropy_mean": 1.573498010635376, + "mean_weights": [ + 0.1865186095237732, + 0.15515196323394775, + 0.15584367513656616, + 0.20077599585056305, + 0.30170977115631104 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5704931020736694, + "mean_weights": [ + 0.1989477574825287, + 0.1317538470029831, + 0.16452556848526, + 0.22815071046352386, + 0.27662211656570435 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5648319721221924, + "mean_weights": [ + 0.19760456681251526, + 0.1755809187889099, + 0.13326793909072876, + 0.18024374544620514, + 0.3133028447628021 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5685514211654663, + "mean_weights": [ + 0.18044300377368927, + 0.13280656933784485, + 0.17437370121479034, + 0.22617360949516296, + 0.2862030863761902 + ], + "sources": 5 + }, + { + "entropy_mean": 1.559435486793518, + "mean_weights": [ + 0.203350231051445, + 0.1763553023338318, + 0.1268777847290039, + 0.17579129338264465, + 0.31762537360191345 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5666444301605225, + "mean_weights": [ + 0.17397071421146393, + 0.13124798238277435, + 0.1824997365474701, + 0.22334709763526917, + 0.28893446922302246 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5829676389694214, + "mean_weights": [ + 0.208939790725708, + 0.19224119186401367, + 0.1409887820482254, + 0.17907415330410004, + 0.2787560522556305 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7642302513122559, + "mean_weights": [ + 0.13727891445159912, + 0.11752122640609741, + 0.15556874871253967, + 0.1911260187625885, + 0.20210129022598267, + 0.19640380144119263 + ], + "sources": 6 + }, + { + "entropy_mean": 1.760313630104065, + "mean_weights": [ + 0.1569042205810547, + 0.15488529205322266, + 0.11598058044910431, + 0.13790756464004517, + 0.19184449315071106, + 0.24247784912586212 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7252838611602783, + "mean_weights": [ + 0.12333965301513672, + 0.10839565098285675, + 0.14657631516456604, + 0.17611858248710632, + 0.1619301438331604, + 0.2836396396160126 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7644871473312378, + "mean_weights": [ + 0.15994563698768616, + 0.1735057234764099, + 0.12058398127555847, + 0.1332891285419464, + 0.1778017282485962, + 0.23487380146980286 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6954247951507568, + "mean_weights": [ + 0.11809982359409332, + 0.10797758400440216, + 0.14872059226036072, + 0.17263229191303253, + 0.1425720602273941, + 0.30999767780303955 + ], + "sources": 6 + }, + { + "entropy_mean": 1.775092363357544, + "mean_weights": [ + 0.16631971299648285, + 0.17688938975334167, + 0.12578684091567993, + 0.13950610160827637, + 0.17812088131904602, + 0.21337708830833435 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6726689338684082, + "mean_weights": [ + 0.11664745211601257, + 0.11694379150867462, + 0.15338079631328583, + 0.17386116087436676, + 0.1211126297712326, + 0.3180541694164276 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.042613983154297, + "mean_weights": [ + 0.10510344803333282, + 0.14694339036941528, + 0.12587970495224, + 0.15129750967025757, + 0.12525907158851624, + 0.11773985624313354, + 0.10783850401639938, + 0.11993850767612457 + ], + "sources": 8 + }, + { + "entropy_mean": 2.080890655517578, + "mean_weights": [ + 0.0839042067527771, + 0.12582458555698395, + 0.11565825343132019, + 0.12251108884811401, + 0.07689991593360901, + 0.11749555170536041, + 0.12039513885974884, + 0.1562080681324005, + 0.08110320568084717 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1295504570007324, + "mean_weights": [ + 0.10200542956590652, + 0.13464105129241943, + 0.11013421416282654, + 0.1195402592420578, + 0.12953504920005798, + 0.11238446831703186, + 0.09050984680652618, + 0.09491261094808578, + 0.10633707046508789 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1028950214385986, + "mean_weights": [ + 0.09345091879367828, + 0.11156691610813141, + 0.10446309298276901, + 0.10856644064188004, + 0.07178381830453873, + 0.08904972672462463, + 0.10463681071996689, + 0.18328651785850525, + 0.13319575786590576 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1256871223449707, + "mean_weights": [ + 0.08671481907367706, + 0.1175852119922638, + 0.15113359689712524, + 0.16495013236999512, + 0.10911397635936737, + 0.09323187172412872, + 0.08714233338832855, + 0.09217001497745514, + 0.0979580506682396 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0627269744873047, + "mean_weights": [ + 0.08607552945613861, + 0.11850239336490631, + 0.10225851088762283, + 0.12150567770004272, + 0.07895441353321075, + 0.10178560763597488, + 0.10786771774291992, + 0.15558519959449768, + 0.12746495008468628 + ], + "sources": 9 + }, + { + "entropy_mean": 2.14670991897583, + "mean_weights": [ + 0.1085396558046341, + 0.11281973123550415, + 0.11845046281814575, + 0.12011014670133591, + 0.11137059330940247, + 0.09822011739015579, + 0.08586028963327408, + 0.10546768456697464, + 0.13916131854057312 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1382153034210205, + "mean_weights": [ + 0.0883704274892807, + 0.12608887255191803, + 0.11896911263465881, + 0.1286507546901703, + 0.08017632365226746, + 0.10515399277210236, + 0.11371965706348419, + 0.1289934366941452, + 0.10987742990255356 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.062110062688589096, + 0.057186078280210495, + 0.059676021337509155, + 0.09277907758951187, + 0.10812339931726456, + 0.16207344830036163, + 0.13376617431640625, + 0.09478043764829636, + 0.12260771542787552, + 0.0912390947341919, + 0.0949578508734703, + 0.09839925169944763, + 0.08921178430318832, + 0.09110469371080399, + 0.092745341360569, + 0.09349297732114792, + 0.08939606696367264, + 0.05843309685587883, + 0.06346471607685089, + 0.06137843802571297, + 0.06323812156915665, + 0.05275778844952583, + 0.05894050747156143, + 0.053545031696558, + 0.059849075973033905, + 0.04022706672549248, + 0.04067765548825264, + 0.03861979767680168, + 0.04519735276699066, + 0.04193759337067604, + 0.048788923770189285, + 0.04480575770139694, + 0.05173776298761368, + 0.03802125155925751, + 0.04117017984390259, + 0.037297505885362625, + 0.04483562707901001, + 0.040994949638843536, + 0.04667338728904724, + 0.047261666506528854, + 0.04569203034043312, + 0.03907719627022743, + 0.03974020853638649, + 0.039902664721012115, + 0.040775116533041, + 0.041109226644039154, + 0.04212002083659172, + 0.041933007538318634, + 0.0435471385717392, + 0.03817557170987129, + 0.039146605879068375, + 0.039246756583452225, + 0.040678851306438446, + 0.040773481130599976, + 0.04264426976442337, + 0.042663030326366425, + 0.04642742872238159, + 0.0507759191095829, + 0.04353255033493042, + 0.052450813353061676, + 0.045101623982191086, + 0.056091971695423126, + 0.055095601826906204, + 0.054030075669288635 + ], + "loss_nats": 1.2945951223373413, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.832993984222412, + "mean_weights": [ + 0.09787635505199432, + 0.08212883025407791, + 0.0798412412405014, + 0.05888679623603821, + 0.0595969557762146, + 0.04959839954972267, + 0.05849500000476837, + 0.14884504675865173, + 0.3647313714027405 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.05879996344447136, + 0.09945333749055862, + 0.18900763988494873, + 0.19213460385799408, + 0.2235446721315384, + 0.22797581553459167, + 0.23000991344451904, + 0.23613907396793365, + 0.09285115450620651, + 0.09485343843698502, + 0.1049794852733612, + 0.10560820996761322, + 0.11230000853538513, + 0.11283529549837112, + 0.11697530746459961, + 0.11850933730602264, + 0.040811773389577866, + 0.04548373818397522, + 0.05428897589445114, + 0.06377377361059189, + 0.08257241547107697, + 0.09456658363342285, + 0.11547749489545822, + 0.13270698487758636, + 0.038965776562690735, + 0.057267654687166214, + 0.0650634914636612, + 0.08288402110338211, + 0.08710475265979767, + 0.10530658811330795, + 0.11187241971492767, + 0.12915223836898804, + 0.029996519908308983, + 0.04864993691444397, + 0.05619852989912033, + 0.07540855556726456, + 0.0804043561220169, + 0.10244929790496826, + 0.10977394133806229, + 0.13460960984230042, + 0.055284686386585236, + 0.06746271997690201, + 0.08945158869028091, + 0.10431721806526184, + 0.12253712117671967, + 0.13895606994628906, + 0.15319955348968506, + 0.17493325471878052, + 0.0389670766890049, + 0.06875944882631302, + 0.08242810517549515, + 0.11196905374526978, + 0.1173071339726448, + 0.14755180478096008, + 0.14938129484653473, + 0.18564240634441376, + 0.01462650392204523, + 0.06505608558654785, + 0.06788776069879532, + 0.12338310480117798, + 0.12473592162132263, + 0.18688006699085236, + 0.1871739625930786, + 0.2793145477771759 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 8.061194200882085, + "cross_entropy_nats": 5.587594032287598, + "step": 0 + }, + { + "bits_per_byte": 5.192524616386341, + "cross_entropy_nats": 3.5991837978363037, + "step": 100 + }, + { + "bits_per_byte": 3.098694918172217, + "cross_entropy_nats": 2.1478516459465027, + "step": 500 + }, + { + "bits_per_byte": 1.9555047400557704, + "cross_entropy_nats": 1.3554525971412659, + "step": 2000 + }, + { + "bits_per_byte": 1.7930155745162515, + "cross_entropy_nats": 1.2428236901760101, + "step": 4000 + }, + { + "bits_per_byte": 1.7075378752716488, + "cross_entropy_nats": 1.183575063943863, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.40.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.40.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.48.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.48.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "40": true, + "41": true, + "42": true, + "43": true, + "44": true, + "45": true, + "46": true, + "47": true, + "48": true, + "49": true, + "50": true, + "51": true, + "52": true, + "53": true, + "54": true, + "55": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_groups_6_7_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "3a5b536ed501868c8dc4f704dce30eed5ab9ac377aabda951fae1fce10e92c39", + "final_model_state": "b12b63ba2865ac9979106ce317fe1ea59a6a5b3ef23418c7b1d505f59a0efe70", + "final_optimizer_state": "f9d60ba0b0884ebc9d557e3b07ec1b43f38e5c58ee70a15484c21cb89aec76e4", + "final_public_parameters": "68347658e28d0239390ee09f0f556e20c4ebe03123fb457d7e823e689b891539", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "3f93c11463780b386fff00c53b0424f687e1136b1b80f7b626f058adef96c764" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "3eff5c07fa0cfbd5ceed5922cbd09525ad514f801cc0a440c03fd2fa15df4c84", + "1": "e83e4001c5811b19c69a54c7ea7a802af996a34aa57072cde43a76bc8369e2d0", + "7999": "d04c4ca879f0209183a48ac54a0c4ffe3a0f62a78888479d4641f6c304426442" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "e6cb61cfa6e21312670a4709513ff079e515409ae8d8545267b993260344345a", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "formal", + "schema_version": 2, + "seed": 2026073003, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 231.2336079587732, + "measured_steps": 7980, + "median_ms": 233.27342700213194, + "p95_ms": 241.92315847612917, + "peak_allocated_bytes": 10566180864, + "peak_reserved_bytes": 11068768256, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 8.082576459898554, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.602415084838867, + "step": 1, + "unclipped_grad_norm": 19.08204460144043 + }, + { + "bits_per_byte": 7.337211906226384, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.08576774597168, + "step": 10, + "unclipped_grad_norm": 11.73788833618164 + }, + { + "bits_per_byte": 6.794616961713275, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.709669589996338, + "step": 20, + "unclipped_grad_norm": 3.3706064224243164 + }, + { + "bits_per_byte": 6.669787826133427, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.623144626617432, + "step": 30, + "unclipped_grad_norm": 3.0378971099853516 + }, + { + "bits_per_byte": 6.489052634202909, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.497868537902832, + "step": 40, + "unclipped_grad_norm": 3.0709593296051025 + }, + { + "bits_per_byte": 6.170847581854126, + "learning_rate": 3.75e-05, + "loss_nats": 4.277305603027344, + "step": 50, + "unclipped_grad_norm": 2.8265340328216553 + }, + { + "bits_per_byte": 5.946219665701005, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.121605396270752, + "step": 60, + "unclipped_grad_norm": 2.599360942840576 + }, + { + "bits_per_byte": 5.764473215783413, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.9956283569335938, + "step": 70, + "unclipped_grad_norm": 2.5301287174224854 + }, + { + "bits_per_byte": 5.596098760935706, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.878920078277588, + "step": 80, + "unclipped_grad_norm": 2.5978147983551025 + }, + { + "bits_per_byte": 5.405643797509025, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.7469067573547363, + "step": 90, + "unclipped_grad_norm": 2.3179943561553955 + }, + { + "bits_per_byte": 5.178624634578307, + "learning_rate": 7.5e-05, + "loss_nats": 3.5895490646362305, + "step": 100, + "unclipped_grad_norm": 2.2174019813537598 + }, + { + "bits_per_byte": 4.996275557452091, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.4631543159484863, + "step": 110, + "unclipped_grad_norm": 2.0354487895965576 + }, + { + "bits_per_byte": 4.79984660466504, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.327000141143799, + "step": 120, + "unclipped_grad_norm": 1.8100807666778564 + }, + { + "bits_per_byte": 4.6339273741410505, + "learning_rate": 9.75e-05, + "loss_nats": 3.21199369430542, + "step": 130, + "unclipped_grad_norm": 1.5889536142349243 + }, + { + "bits_per_byte": 4.478435795704175, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.104215145111084, + "step": 140, + "unclipped_grad_norm": 1.4092257022857666 + }, + { + "bits_per_byte": 4.276398144707185, + "learning_rate": 0.0001125, + "loss_nats": 2.9641733169555664, + "step": 150, + "unclipped_grad_norm": 1.2035799026489258 + }, + { + "bits_per_byte": 4.130917332411732, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.8633337020874023, + "step": 160, + "unclipped_grad_norm": 1.054665446281433 + }, + { + "bits_per_byte": 4.024317669892739, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.7894444465637207, + "step": 170, + "unclipped_grad_norm": 0.9591813087463379 + }, + { + "bits_per_byte": 3.9197824871147033, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.7169861793518066, + "step": 180, + "unclipped_grad_norm": 0.8343167304992676 + }, + { + "bits_per_byte": 3.835099949656365, + "learning_rate": 0.0001425, + "loss_nats": 2.6582887172698975, + "step": 190, + "unclipped_grad_norm": 0.901328980922699 + }, + { + "bits_per_byte": 3.76554913375154, + "learning_rate": 0.00015, + "loss_nats": 2.610079765319824, + "step": 200, + "unclipped_grad_norm": 0.7249732613563538 + }, + { + "bits_per_byte": 3.685954187071025, + "learning_rate": 0.0001575, + "loss_nats": 2.5549087524414062, + "step": 210, + "unclipped_grad_norm": 0.9448117017745972 + }, + { + "bits_per_byte": 3.6501370802023367, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.5300822257995605, + "step": 220, + "unclipped_grad_norm": 0.7459669709205627 + }, + { + "bits_per_byte": 3.6185218534595283, + "learning_rate": 0.0001725, + "loss_nats": 2.5081682205200195, + "step": 230, + "unclipped_grad_norm": 1.0888783931732178 + }, + { + "bits_per_byte": 3.5838047476257584, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.4841041564941406, + "step": 240, + "unclipped_grad_norm": 0.8678395748138428 + }, + { + "bits_per_byte": 3.5107024899806163, + "learning_rate": 0.0001875, + "loss_nats": 2.4334335327148438, + "step": 250, + "unclipped_grad_norm": 1.1660263538360596 + }, + { + "bits_per_byte": 3.522232206894459, + "learning_rate": 0.000195, + "loss_nats": 2.441425323486328, + "step": 260, + "unclipped_grad_norm": 0.9872084856033325 + }, + { + "bits_per_byte": 3.507011054362497, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.430874824523926, + "step": 270, + "unclipped_grad_norm": 1.1323814392089844 + }, + { + "bits_per_byte": 3.451860689779717, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.3926475048065186, + "step": 280, + "unclipped_grad_norm": 1.0510966777801514 + }, + { + "bits_per_byte": 3.4240211701539502, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.3733506202697754, + "step": 290, + "unclipped_grad_norm": 0.8413814306259155 + }, + { + "bits_per_byte": 3.4370908197236187, + "learning_rate": 0.000225, + "loss_nats": 2.3824098110198975, + "step": 300, + "unclipped_grad_norm": 1.2061761617660522 + }, + { + "bits_per_byte": 3.332725899769937, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.3100695610046387, + "step": 310, + "unclipped_grad_norm": 1.2932002544403076 + }, + { + "bits_per_byte": 3.402488598300625, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.3584253787994385, + "step": 320, + "unclipped_grad_norm": 0.9365999698638916 + }, + { + "bits_per_byte": 3.391900658382907, + "learning_rate": 0.0002475, + "loss_nats": 2.351086378097534, + "step": 330, + "unclipped_grad_norm": 0.7356032729148865 + }, + { + "bits_per_byte": 3.3248573495276617, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.3046154975891113, + "step": 340, + "unclipped_grad_norm": 0.9641121625900269 + }, + { + "bits_per_byte": 3.320587708237342, + "learning_rate": 0.0002625, + "loss_nats": 2.3016560077667236, + "step": 350, + "unclipped_grad_norm": 1.5531033277511597 + }, + { + "bits_per_byte": 3.309825377911649, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.294196128845215, + "step": 360, + "unclipped_grad_norm": 1.0697664022445679 + }, + { + "bits_per_byte": 3.3262854934604267, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.305605411529541, + "step": 370, + "unclipped_grad_norm": 1.2775403261184692 + }, + { + "bits_per_byte": 3.2005634239572998, + "learning_rate": 0.000285, + "loss_nats": 2.218461513519287, + "step": 380, + "unclipped_grad_norm": 0.9361549615859985 + }, + { + "bits_per_byte": 3.278491514785652, + "learning_rate": 0.0002925, + "loss_nats": 2.272477149963379, + "step": 390, + "unclipped_grad_norm": 0.892299234867096 + }, + { + "bits_per_byte": 3.2752616806024486, + "learning_rate": 0.0003, + "loss_nats": 2.2702383995056152, + "step": 400, + "unclipped_grad_norm": 1.1024504899978638 + }, + { + "bits_per_byte": 3.2820928314947193, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.2749733924865723, + "step": 410, + "unclipped_grad_norm": 1.0088258981704712 + }, + { + "bits_per_byte": 3.2126902646349382, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.226867198944092, + "step": 420, + "unclipped_grad_norm": 1.1113554239273071 + }, + { + "bits_per_byte": 3.24002656302428, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.2458152770996094, + "step": 430, + "unclipped_grad_norm": 0.9754257798194885 + }, + { + "bits_per_byte": 3.1737437614511497, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.19987154006958, + "step": 440, + "unclipped_grad_norm": 1.3948684930801392 + }, + { + "bits_per_byte": 3.2455141854480103, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.2496190071105957, + "step": 450, + "unclipped_grad_norm": 1.1475800275802612 + }, + { + "bits_per_byte": 3.1845112512563687, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.2073349952697754, + "step": 460, + "unclipped_grad_norm": 1.3374797105789185 + }, + { + "bits_per_byte": 3.143523314006839, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.178924322128296, + "step": 470, + "unclipped_grad_norm": 1.0365763902664185 + }, + { + "bits_per_byte": 3.1390906331635007, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.175851821899414, + "step": 480, + "unclipped_grad_norm": 1.4345242977142334 + }, + { + "bits_per_byte": 3.1028483851818796, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.150730609893799, + "step": 490, + "unclipped_grad_norm": 1.000265121459961 + }, + { + "bits_per_byte": 3.0543190086920156, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.1170926094055176, + "step": 500, + "unclipped_grad_norm": 1.398908257484436 + }, + { + "bits_per_byte": 3.1120473932112294, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.157106876373291, + "step": 510, + "unclipped_grad_norm": 1.0043644905090332 + }, + { + "bits_per_byte": 3.1350621115496904, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.1730594635009766, + "step": 520, + "unclipped_grad_norm": 1.119065284729004 + }, + { + "bits_per_byte": 2.98878261182363, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.0716662406921387, + "step": 530, + "unclipped_grad_norm": 1.2179008722305298 + }, + { + "bits_per_byte": 2.9955112610559866, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.0763301849365234, + "step": 540, + "unclipped_grad_norm": 1.248818278312683 + }, + { + "bits_per_byte": 2.9851953976919336, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.0691797733306885, + "step": 550, + "unclipped_grad_norm": 0.8645681142807007 + }, + { + "bits_per_byte": 3.032782997185673, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.1021649837493896, + "step": 560, + "unclipped_grad_norm": 1.7811505794525146 + }, + { + "bits_per_byte": 2.9658122650090917, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.0557444095611572, + "step": 570, + "unclipped_grad_norm": 0.8682382702827454 + }, + { + "bits_per_byte": 2.9836107495468838, + "learning_rate": 0.000299626473885547, + "loss_nats": 2.0680813789367676, + "step": 580, + "unclipped_grad_norm": 1.7613459825515747 + }, + { + "bits_per_byte": 2.927977457680481, + "learning_rate": 0.00029958384005397226, + "loss_nats": 2.0295193195343018, + "step": 590, + "unclipped_grad_norm": 1.6035332679748535 + }, + { + "bits_per_byte": 2.8844314504820177, + "learning_rate": 0.0002995389065559004, + "loss_nats": 1.999335527420044, + "step": 600, + "unclipped_grad_norm": 0.9793181419372559 + }, + { + "bits_per_byte": 2.8835866717009795, + "learning_rate": 0.000299491674159121, + "loss_nats": 1.9987499713897705, + "step": 610, + "unclipped_grad_norm": 0.7642365097999573 + }, + { + "bits_per_byte": 2.870783079292196, + "learning_rate": 0.00029944214367070535, + "loss_nats": 1.9898751974105835, + "step": 620, + "unclipped_grad_norm": 0.7781364321708679 + }, + { + "bits_per_byte": 2.804356328240294, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.9438316822052002, + "step": 630, + "unclipped_grad_norm": 1.376232385635376 + }, + { + "bits_per_byte": 2.8092413234554012, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.9472177028656006, + "step": 640, + "unclipped_grad_norm": 1.5085498094558716 + }, + { + "bits_per_byte": 2.783354494847531, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.929274320602417, + "step": 650, + "unclipped_grad_norm": 1.492533564567566 + }, + { + "bits_per_byte": 2.7947794743269982, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.937193512916565, + "step": 660, + "unclipped_grad_norm": 0.9243024587631226 + }, + { + "bits_per_byte": 2.7785888555921225, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.9259710311889648, + "step": 670, + "unclipped_grad_norm": 0.9211107492446899 + }, + { + "bits_per_byte": 2.7593677305663937, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.9126479625701904, + "step": 680, + "unclipped_grad_norm": 0.8353617191314697 + }, + { + "bits_per_byte": 2.7291723925891085, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.8917181491851807, + "step": 690, + "unclipped_grad_norm": 0.9634787440299988 + }, + { + "bits_per_byte": 2.721908189381965, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.8866829872131348, + "step": 700, + "unclipped_grad_norm": 0.8833006620407104 + }, + { + "bits_per_byte": 2.7028569832152884, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.8734776973724365, + "step": 710, + "unclipped_grad_norm": 1.7387349605560303 + }, + { + "bits_per_byte": 2.7220855034950047, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.8868058919906616, + "step": 720, + "unclipped_grad_norm": 1.2254352569580078 + }, + { + "bits_per_byte": 2.6828746629768405, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.8596270084381104, + "step": 730, + "unclipped_grad_norm": 1.2014808654785156 + }, + { + "bits_per_byte": 2.6359949440740524, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.8271324634552002, + "step": 740, + "unclipped_grad_norm": 0.948524534702301 + }, + { + "bits_per_byte": 2.63603828370207, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.827162504196167, + "step": 750, + "unclipped_grad_norm": 0.9068747758865356 + }, + { + "bits_per_byte": 2.5483751149216283, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.7663990259170532, + "step": 760, + "unclipped_grad_norm": 1.1133238077163696 + }, + { + "bits_per_byte": 2.5792877925356623, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.7878260612487793, + "step": 770, + "unclipped_grad_norm": 1.0377305746078491 + }, + { + "bits_per_byte": 2.5256859597412507, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.7506721019744873, + "step": 780, + "unclipped_grad_norm": 1.1458407640457153 + }, + { + "bits_per_byte": 2.534462062432129, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.7567552328109741, + "step": 790, + "unclipped_grad_norm": 1.4523664712905884 + }, + { + "bits_per_byte": 2.457085863930395, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.7031221389770508, + "step": 800, + "unclipped_grad_norm": 1.1469895839691162 + }, + { + "bits_per_byte": 2.4910782348734295, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7266838550567627, + "step": 810, + "unclipped_grad_norm": 0.9928609728813171 + }, + { + "bits_per_byte": 2.4769797850862996, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.7169115543365479, + "step": 820, + "unclipped_grad_norm": 1.2352039813995361 + }, + { + "bits_per_byte": 2.478607600876716, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.718039870262146, + "step": 830, + "unclipped_grad_norm": 0.957145631313324 + }, + { + "bits_per_byte": 2.479086572559369, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.718371868133545, + "step": 840, + "unclipped_grad_norm": 1.386124849319458 + }, + { + "bits_per_byte": 2.3933224362097545, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.6589246988296509, + "step": 850, + "unclipped_grad_norm": 1.0637670755386353 + }, + { + "bits_per_byte": 2.4636267081080097, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.707655906677246, + "step": 860, + "unclipped_grad_norm": 1.1763083934783936 + }, + { + "bits_per_byte": 2.4441959362308787, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.6941875219345093, + "step": 870, + "unclipped_grad_norm": 0.9347332715988159 + }, + { + "bits_per_byte": 2.3974445163856344, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.661781907081604, + "step": 880, + "unclipped_grad_norm": 1.0671870708465576 + }, + { + "bits_per_byte": 2.3500964887245472, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.628962755203247, + "step": 890, + "unclipped_grad_norm": 1.0013151168823242 + }, + { + "bits_per_byte": 2.4059361597719855, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.6676678657531738, + "step": 900, + "unclipped_grad_norm": 1.0294618606567383 + }, + { + "bits_per_byte": 2.3770595847615166, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.6476521492004395, + "step": 910, + "unclipped_grad_norm": 1.1526020765304565 + }, + { + "bits_per_byte": 2.366948724717292, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.640643835067749, + "step": 920, + "unclipped_grad_norm": 1.0233529806137085 + }, + { + "bits_per_byte": 2.3069259196529175, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.5990391969680786, + "step": 930, + "unclipped_grad_norm": 1.0709304809570312 + }, + { + "bits_per_byte": 2.3644150763047813, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.638887643814087, + "step": 940, + "unclipped_grad_norm": 1.021758794784546 + }, + { + "bits_per_byte": 2.378594185955165, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.648715853691101, + "step": 950, + "unclipped_grad_norm": 0.9562984704971313 + }, + { + "bits_per_byte": 2.3127041927566063, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.6030443906784058, + "step": 960, + "unclipped_grad_norm": 0.8433545827865601 + }, + { + "bits_per_byte": 2.2620458430317116, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5679306983947754, + "step": 970, + "unclipped_grad_norm": 1.0910544395446777 + }, + { + "bits_per_byte": 2.408267384604431, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.6692837476730347, + "step": 980, + "unclipped_grad_norm": 0.9450616836547852 + }, + { + "bits_per_byte": 2.2508487405646376, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.5601694583892822, + "step": 990, + "unclipped_grad_norm": 1.0212528705596924 + }, + { + "bits_per_byte": 2.2349097324479743, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.549121379852295, + "step": 1000, + "unclipped_grad_norm": 0.7464981079101562 + }, + { + "bits_per_byte": 2.236099508426643, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.5499460697174072, + "step": 1010, + "unclipped_grad_norm": 0.8849756121635437 + }, + { + "bits_per_byte": 2.217692033322146, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.5371869802474976, + "step": 1020, + "unclipped_grad_norm": 0.9932225346565247 + }, + { + "bits_per_byte": 2.216398035857054, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.5362900495529175, + "step": 1030, + "unclipped_grad_norm": 1.178478717803955 + }, + { + "bits_per_byte": 2.3132077579583323, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.6033934354782104, + "step": 1040, + "unclipped_grad_norm": 1.1624207496643066 + }, + { + "bits_per_byte": 2.220996679958475, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5394775867462158, + "step": 1050, + "unclipped_grad_norm": 0.9422160387039185 + }, + { + "bits_per_byte": 2.2397568914798915, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.5524811744689941, + "step": 1060, + "unclipped_grad_norm": 0.8465353846549988 + }, + { + "bits_per_byte": 2.2895797494868946, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.5870157480239868, + "step": 1070, + "unclipped_grad_norm": 0.8776537775993347 + }, + { + "bits_per_byte": 2.2454596641998514, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.5564340353012085, + "step": 1080, + "unclipped_grad_norm": 0.8563132882118225 + }, + { + "bits_per_byte": 2.1916003733770983, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.519101619720459, + "step": 1090, + "unclipped_grad_norm": 0.9418866634368896 + }, + { + "bits_per_byte": 2.192813882961586, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5199427604675293, + "step": 1100, + "unclipped_grad_norm": 0.8768544793128967 + }, + { + "bits_per_byte": 2.1347227870317025, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.4796770811080933, + "step": 1110, + "unclipped_grad_norm": 0.9323295950889587 + }, + { + "bits_per_byte": 2.2005445031178694, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.525301218032837, + "step": 1120, + "unclipped_grad_norm": 0.9501368999481201 + }, + { + "bits_per_byte": 2.222731124992426, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.5406798124313354, + "step": 1130, + "unclipped_grad_norm": 0.9085060358047485 + }, + { + "bits_per_byte": 2.1360071534683462, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.4805673360824585, + "step": 1140, + "unclipped_grad_norm": 0.834496796131134 + }, + { + "bits_per_byte": 2.194778612765043, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.5213046073913574, + "step": 1150, + "unclipped_grad_norm": 0.9277384877204895 + }, + { + "bits_per_byte": 2.104810188532436, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.458943247795105, + "step": 1160, + "unclipped_grad_norm": 0.9865785837173462 + }, + { + "bits_per_byte": 2.1590363183494796, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.4965299367904663, + "step": 1170, + "unclipped_grad_norm": 0.8092359304428101 + }, + { + "bits_per_byte": 2.1600855845023936, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.4972572326660156, + "step": 1180, + "unclipped_grad_norm": 0.8338137865066528 + }, + { + "bits_per_byte": 2.141384191047572, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.4842944145202637, + "step": 1190, + "unclipped_grad_norm": 0.8842478394508362 + }, + { + "bits_per_byte": 2.1433492648163304, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.485656499862671, + "step": 1200, + "unclipped_grad_norm": 0.7912976741790771 + }, + { + "bits_per_byte": 2.159898811343557, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.4971277713775635, + "step": 1210, + "unclipped_grad_norm": 0.8705281615257263 + }, + { + "bits_per_byte": 2.123618211230793, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4719799757003784, + "step": 1220, + "unclipped_grad_norm": 0.8929391503334045 + }, + { + "bits_per_byte": 2.125121167616683, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.4730217456817627, + "step": 1230, + "unclipped_grad_norm": 0.8618266582489014 + }, + { + "bits_per_byte": 2.0995275694285342, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.4552816152572632, + "step": 1240, + "unclipped_grad_norm": 0.8652570247650146 + }, + { + "bits_per_byte": 2.027210412584581, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.4051551818847656, + "step": 1250, + "unclipped_grad_norm": 0.8439058661460876 + }, + { + "bits_per_byte": 2.085459388587956, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.4455302953720093, + "step": 1260, + "unclipped_grad_norm": 0.8378538489341736 + }, + { + "bits_per_byte": 2.1032788550091537, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.4578818082809448, + "step": 1270, + "unclipped_grad_norm": 0.8435846567153931 + }, + { + "bits_per_byte": 2.153981232292669, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.4930260181427002, + "step": 1280, + "unclipped_grad_norm": 1.1229397058486938 + }, + { + "bits_per_byte": 2.0853529313270722, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4454565048217773, + "step": 1290, + "unclipped_grad_norm": 0.879615843296051 + }, + { + "bits_per_byte": 2.048468156161827, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.4198899269104004, + "step": 1300, + "unclipped_grad_norm": 0.8570435643196106 + }, + { + "bits_per_byte": 2.0600948712907567, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.4279489517211914, + "step": 1310, + "unclipped_grad_norm": 0.8467429280281067 + }, + { + "bits_per_byte": 2.0514076836303747, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.4219274520874023, + "step": 1320, + "unclipped_grad_norm": 0.8539228439331055 + }, + { + "bits_per_byte": 2.059377359671357, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.4274516105651855, + "step": 1330, + "unclipped_grad_norm": 0.8059477806091309 + }, + { + "bits_per_byte": 2.033356556578465, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.409415364265442, + "step": 1340, + "unclipped_grad_norm": 0.730267345905304 + }, + { + "bits_per_byte": 2.0603968728256716, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.4281582832336426, + "step": 1350, + "unclipped_grad_norm": 0.7600128054618835 + }, + { + "bits_per_byte": 2.01741324889553, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.3983643054962158, + "step": 1360, + "unclipped_grad_norm": 0.8321276903152466 + }, + { + "bits_per_byte": 2.054138424160774, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4238202571868896, + "step": 1370, + "unclipped_grad_norm": 0.7984517216682434 + }, + { + "bits_per_byte": 2.0171165788227916, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.3981586694717407, + "step": 1380, + "unclipped_grad_norm": 0.7809591293334961 + }, + { + "bits_per_byte": 2.0647244722693325, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.4311579465866089, + "step": 1390, + "unclipped_grad_norm": 0.7597355842590332 + }, + { + "bits_per_byte": 2.1746487314121548, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.5073516368865967, + "step": 1400, + "unclipped_grad_norm": 0.820720911026001 + }, + { + "bits_per_byte": 2.0509493498658253, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.4216097593307495, + "step": 1410, + "unclipped_grad_norm": 0.7983555793762207 + }, + { + "bits_per_byte": 2.060398936617482, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.4281597137451172, + "step": 1420, + "unclipped_grad_norm": 0.8105078339576721 + }, + { + "bits_per_byte": 1.9950183560307786, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.3828413486480713, + "step": 1430, + "unclipped_grad_norm": 0.743954062461853 + }, + { + "bits_per_byte": 2.033311841089241, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.4093843698501587, + "step": 1440, + "unclipped_grad_norm": 0.8468954563140869 + }, + { + "bits_per_byte": 1.9890330158154512, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.378692626953125, + "step": 1450, + "unclipped_grad_norm": 0.6986736059188843 + }, + { + "bits_per_byte": 2.1185717243065256, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.4684820175170898, + "step": 1460, + "unclipped_grad_norm": 0.7915589809417725 + }, + { + "bits_per_byte": 2.0957740480734532, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.4526798725128174, + "step": 1470, + "unclipped_grad_norm": 0.6974943280220032 + }, + { + "bits_per_byte": 2.046802848153523, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.4187356233596802, + "step": 1480, + "unclipped_grad_norm": 0.744508683681488 + }, + { + "bits_per_byte": 2.0498985358690534, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.4208813905715942, + "step": 1490, + "unclipped_grad_norm": 0.867777943611145 + }, + { + "bits_per_byte": 1.926238022401819, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.3351664543151855, + "step": 1500, + "unclipped_grad_norm": 0.7736293077468872 + }, + { + "bits_per_byte": 1.9521104044670166, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.3530998229980469, + "step": 1510, + "unclipped_grad_norm": 0.831000566482544 + }, + { + "bits_per_byte": 2.0155499888560824, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.3970727920532227, + "step": 1520, + "unclipped_grad_norm": 0.8072499632835388 + }, + { + "bits_per_byte": 2.040432782748215, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.4143202304840088, + "step": 1530, + "unclipped_grad_norm": 0.7439060807228088 + }, + { + "bits_per_byte": 2.0119130717382867, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3945518732070923, + "step": 1540, + "unclipped_grad_norm": 0.7004512548446655 + }, + { + "bits_per_byte": 2.0702468351885375, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.4349857568740845, + "step": 1550, + "unclipped_grad_norm": 0.7680191993713379 + }, + { + "bits_per_byte": 1.952848210039218, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.3536112308502197, + "step": 1560, + "unclipped_grad_norm": 0.7375255823135376 + }, + { + "bits_per_byte": 1.9443120231462934, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.3476943969726562, + "step": 1570, + "unclipped_grad_norm": 0.7867926955223083 + }, + { + "bits_per_byte": 1.9780873519839386, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.371105670928955, + "step": 1580, + "unclipped_grad_norm": 0.8975521326065063 + }, + { + "bits_per_byte": 1.9704100464494227, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3657841682434082, + "step": 1590, + "unclipped_grad_norm": 0.7647879719734192 + }, + { + "bits_per_byte": 1.9633377758806423, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.360882043838501, + "step": 1600, + "unclipped_grad_norm": 0.7478978037834167 + }, + { + "bits_per_byte": 1.9358802297224424, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.34184992313385, + "step": 1610, + "unclipped_grad_norm": 0.8023220896720886 + }, + { + "bits_per_byte": 1.9376086553636136, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.3430479764938354, + "step": 1620, + "unclipped_grad_norm": 0.7497729659080505 + }, + { + "bits_per_byte": 1.979127159091055, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.371826410293579, + "step": 1630, + "unclipped_grad_norm": 0.7113741636276245 + }, + { + "bits_per_byte": 1.9299442485279126, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.3377354145050049, + "step": 1640, + "unclipped_grad_norm": 0.6879138946533203 + }, + { + "bits_per_byte": 1.9337292427081012, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.3403589725494385, + "step": 1650, + "unclipped_grad_norm": 0.7300919890403748 + }, + { + "bits_per_byte": 1.9459011428402657, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.3487958908081055, + "step": 1660, + "unclipped_grad_norm": 0.6742871403694153 + }, + { + "bits_per_byte": 1.8997525221862956, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.3168081045150757, + "step": 1670, + "unclipped_grad_norm": 0.9737929105758667 + }, + { + "bits_per_byte": 1.917500443824734, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.329110026359558, + "step": 1680, + "unclipped_grad_norm": 0.7087656259536743 + }, + { + "bits_per_byte": 1.957551763557665, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.356871485710144, + "step": 1690, + "unclipped_grad_norm": 0.6480438709259033 + }, + { + "bits_per_byte": 1.8908369415655675, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.3106282949447632, + "step": 1700, + "unclipped_grad_norm": 0.7340534925460815 + }, + { + "bits_per_byte": 1.94583028598811, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3487467765808105, + "step": 1710, + "unclipped_grad_norm": 0.7663553357124329 + }, + { + "bits_per_byte": 1.8756331592813427, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.3000898361206055, + "step": 1720, + "unclipped_grad_norm": 0.7853311896324158 + }, + { + "bits_per_byte": 1.9546891123340535, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.3548872470855713, + "step": 1730, + "unclipped_grad_norm": 0.7390387654304504 + }, + { + "bits_per_byte": 1.9446525487950017, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.3479304313659668, + "step": 1740, + "unclipped_grad_norm": 0.6868768930435181 + }, + { + "bits_per_byte": 1.9357163302561702, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.3417363166809082, + "step": 1750, + "unclipped_grad_norm": 0.7297267317771912 + }, + { + "bits_per_byte": 1.9540273230935332, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.3544285297393799, + "step": 1760, + "unclipped_grad_norm": 0.7383672595024109 + }, + { + "bits_per_byte": 1.9463651520322935, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.3491175174713135, + "step": 1770, + "unclipped_grad_norm": 0.6786460876464844 + }, + { + "bits_per_byte": 2.0182373897584647, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.3989355564117432, + "step": 1780, + "unclipped_grad_norm": 0.746569037437439 + }, + { + "bits_per_byte": 1.8664809305330274, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.293745994567871, + "step": 1790, + "unclipped_grad_norm": 0.7174710035324097 + }, + { + "bits_per_byte": 2.004611204330604, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.3894906044006348, + "step": 1800, + "unclipped_grad_norm": 0.8113299012184143 + }, + { + "bits_per_byte": 1.884513139493342, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.306244969367981, + "step": 1810, + "unclipped_grad_norm": 0.696925699710846 + }, + { + "bits_per_byte": 1.9341337459029304, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.340639352798462, + "step": 1820, + "unclipped_grad_norm": 0.7401153445243835 + }, + { + "bits_per_byte": 1.9692436601112713, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.3649756908416748, + "step": 1830, + "unclipped_grad_norm": 0.7103039026260376 + }, + { + "bits_per_byte": 1.9415372550572727, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.345771074295044, + "step": 1840, + "unclipped_grad_norm": 0.7066033482551575 + }, + { + "bits_per_byte": 1.896660790071735, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.3146650791168213, + "step": 1850, + "unclipped_grad_norm": 0.7395609617233276 + }, + { + "bits_per_byte": 1.908950670302391, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.3231837749481201, + "step": 1860, + "unclipped_grad_norm": 0.7295752167701721 + }, + { + "bits_per_byte": 2.0038617039381443, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.3889710903167725, + "step": 1870, + "unclipped_grad_norm": 0.7074335217475891 + }, + { + "bits_per_byte": 1.8811056472317973, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.3038830757141113, + "step": 1880, + "unclipped_grad_norm": 0.6886325478553772 + }, + { + "bits_per_byte": 1.9055569366529155, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3208314180374146, + "step": 1890, + "unclipped_grad_norm": 0.6827114224433899 + }, + { + "bits_per_byte": 1.9049398629016197, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.3204036951065063, + "step": 1900, + "unclipped_grad_norm": 0.7431093454360962 + }, + { + "bits_per_byte": 1.9423875372831385, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.346360445022583, + "step": 1910, + "unclipped_grad_norm": 0.7958953976631165 + }, + { + "bits_per_byte": 1.859783410160477, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.289103627204895, + "step": 1920, + "unclipped_grad_norm": 0.6755863428115845 + }, + { + "bits_per_byte": 1.9238182265041792, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.333489179611206, + "step": 1930, + "unclipped_grad_norm": 0.7016053199768066 + }, + { + "bits_per_byte": 1.9398882854008002, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.344628095626831, + "step": 1940, + "unclipped_grad_norm": 0.6630347967147827 + }, + { + "bits_per_byte": 1.8742882549515956, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.2991576194763184, + "step": 1950, + "unclipped_grad_norm": 0.7005661725997925 + }, + { + "bits_per_byte": 1.8633924660888332, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.2916052341461182, + "step": 1960, + "unclipped_grad_norm": 0.7143921256065369 + }, + { + "bits_per_byte": 1.8762397420909358, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.300510287284851, + "step": 1970, + "unclipped_grad_norm": 0.7290456295013428 + }, + { + "bits_per_byte": 1.9021226150978359, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.318450927734375, + "step": 1980, + "unclipped_grad_norm": 0.737045407295227 + }, + { + "bits_per_byte": 1.865246783030436, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.2928905487060547, + "step": 1990, + "unclipped_grad_norm": 0.694668173789978 + }, + { + "bits_per_byte": 1.9535923789695013, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.354127049446106, + "step": 2000, + "unclipped_grad_norm": 0.736270546913147 + }, + { + "bits_per_byte": 1.9491767244085987, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.3510663509368896, + "step": 2010, + "unclipped_grad_norm": 0.7217370271682739 + }, + { + "bits_per_byte": 1.8518727241687405, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.2836203575134277, + "step": 2020, + "unclipped_grad_norm": 0.6572960019111633 + }, + { + "bits_per_byte": 1.8784137747804925, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.3020172119140625, + "step": 2030, + "unclipped_grad_norm": 0.773311972618103 + }, + { + "bits_per_byte": 1.8785644315826484, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.302121639251709, + "step": 2040, + "unclipped_grad_norm": 0.6841026544570923 + }, + { + "bits_per_byte": 1.8429299702891762, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.2774217128753662, + "step": 2050, + "unclipped_grad_norm": 0.7023804783821106 + }, + { + "bits_per_byte": 1.8325336190445194, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.2702155113220215, + "step": 2060, + "unclipped_grad_norm": 0.730611264705658 + }, + { + "bits_per_byte": 1.9403249493513408, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.3449307680130005, + "step": 2070, + "unclipped_grad_norm": 0.6915774345397949 + }, + { + "bits_per_byte": 1.928234912960987, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.3365505933761597, + "step": 2080, + "unclipped_grad_norm": 0.7128904461860657 + }, + { + "bits_per_byte": 1.8437104275587917, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.2779626846313477, + "step": 2090, + "unclipped_grad_norm": 0.6963062286376953 + }, + { + "bits_per_byte": 1.7766848329165852, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.2315040826797485, + "step": 2100, + "unclipped_grad_norm": 0.6231761574745178 + }, + { + "bits_per_byte": 1.8246372076128043, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.264742136001587, + "step": 2110, + "unclipped_grad_norm": 0.6504113078117371 + }, + { + "bits_per_byte": 1.8525613427027952, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.284097671508789, + "step": 2120, + "unclipped_grad_norm": 0.8626440763473511 + }, + { + "bits_per_byte": 1.8924867711352944, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.3117718696594238, + "step": 2130, + "unclipped_grad_norm": 0.8109397292137146 + }, + { + "bits_per_byte": 1.9155744101177214, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.327775001525879, + "step": 2140, + "unclipped_grad_norm": 0.7570427060127258 + }, + { + "bits_per_byte": 1.8170833856216082, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2595062255859375, + "step": 2150, + "unclipped_grad_norm": 0.762603759765625 + }, + { + "bits_per_byte": 1.8961307395417757, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.3142976760864258, + "step": 2160, + "unclipped_grad_norm": 0.692477285861969 + }, + { + "bits_per_byte": 1.8193545885089024, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.2610805034637451, + "step": 2170, + "unclipped_grad_norm": 0.6577823758125305 + }, + { + "bits_per_byte": 1.8899816718428268, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.3100354671478271, + "step": 2180, + "unclipped_grad_norm": 0.6571688055992126 + }, + { + "bits_per_byte": 1.8651791938486468, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.2928436994552612, + "step": 2190, + "unclipped_grad_norm": 0.6501729488372803 + }, + { + "bits_per_byte": 1.8402315624971388, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.2755513191223145, + "step": 2200, + "unclipped_grad_norm": 0.6561394929885864 + }, + { + "bits_per_byte": 1.8285296909497823, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.2674401998519897, + "step": 2210, + "unclipped_grad_norm": 0.6960810422897339 + }, + { + "bits_per_byte": 1.8238025758081673, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.264163613319397, + "step": 2220, + "unclipped_grad_norm": 0.667670488357544 + }, + { + "bits_per_byte": 1.7979141993564396, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.2462191581726074, + "step": 2230, + "unclipped_grad_norm": 0.6137583255767822 + }, + { + "bits_per_byte": 1.8770060967831805, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.3010414838790894, + "step": 2240, + "unclipped_grad_norm": 0.7784464955329895 + }, + { + "bits_per_byte": 1.8525673620955754, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2841018438339233, + "step": 2250, + "unclipped_grad_norm": 0.7028475403785706 + }, + { + "bits_per_byte": 1.9045362196200446, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.3201239109039307, + "step": 2260, + "unclipped_grad_norm": 0.6728191375732422 + }, + { + "bits_per_byte": 1.7940107091298063, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.2435134649276733, + "step": 2270, + "unclipped_grad_norm": 0.6802076101303101 + }, + { + "bits_per_byte": 1.861413289742704, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.2902333736419678, + "step": 2280, + "unclipped_grad_norm": 0.6694319248199463 + }, + { + "bits_per_byte": 1.80705352940594, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.2525540590286255, + "step": 2290, + "unclipped_grad_norm": 0.6245759129524231 + }, + { + "bits_per_byte": 1.808776967550236, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.2537486553192139, + "step": 2300, + "unclipped_grad_norm": 0.6814858913421631 + }, + { + "bits_per_byte": 1.8076102372467828, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.2529399394989014, + "step": 2310, + "unclipped_grad_norm": 0.6598613858222961 + }, + { + "bits_per_byte": 1.8621746569380737, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.290761113166809, + "step": 2320, + "unclipped_grad_norm": 0.7034543752670288 + }, + { + "bits_per_byte": 1.792050622857923, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.242154836654663, + "step": 2330, + "unclipped_grad_norm": 0.69230055809021 + }, + { + "bits_per_byte": 1.8136939515384038, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.2571568489074707, + "step": 2340, + "unclipped_grad_norm": 0.6537413001060486 + }, + { + "bits_per_byte": 1.8714209601964107, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2971701622009277, + "step": 2350, + "unclipped_grad_norm": 0.6890630125999451 + }, + { + "bits_per_byte": 1.8040257748375002, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.250455379486084, + "step": 2360, + "unclipped_grad_norm": 0.6813631653785706 + }, + { + "bits_per_byte": 1.7316644184522751, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.2002983093261719, + "step": 2370, + "unclipped_grad_norm": 0.628761887550354 + }, + { + "bits_per_byte": 1.847932085689521, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.2808889150619507, + "step": 2380, + "unclipped_grad_norm": 0.6671104431152344 + }, + { + "bits_per_byte": 1.8377914726466973, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.273859977722168, + "step": 2390, + "unclipped_grad_norm": 0.6854297518730164 + }, + { + "bits_per_byte": 1.867578351828183, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.2945066690444946, + "step": 2400, + "unclipped_grad_norm": 0.7025692462921143 + }, + { + "bits_per_byte": 1.793177453186376, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2429358959197998, + "step": 2410, + "unclipped_grad_norm": 0.6886538863182068 + }, + { + "bits_per_byte": 1.8211504313492117, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.2623252868652344, + "step": 2420, + "unclipped_grad_norm": 0.6598705649375916 + }, + { + "bits_per_byte": 1.8333581038727558, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.270787000656128, + "step": 2430, + "unclipped_grad_norm": 0.6405289173126221 + }, + { + "bits_per_byte": 1.767887404377, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.2254061698913574, + "step": 2440, + "unclipped_grad_norm": 0.615987241268158 + }, + { + "bits_per_byte": 1.8959554892205466, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.3141762018203735, + "step": 2450, + "unclipped_grad_norm": 0.7571413516998291 + }, + { + "bits_per_byte": 1.7592143192939886, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.2193944454193115, + "step": 2460, + "unclipped_grad_norm": 0.6548334956169128 + }, + { + "bits_per_byte": 1.7730030283269143, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.2289520502090454, + "step": 2470, + "unclipped_grad_norm": 0.7108430862426758 + }, + { + "bits_per_byte": 1.8755082998768162, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.3000032901763916, + "step": 2480, + "unclipped_grad_norm": 0.7384935617446899 + }, + { + "bits_per_byte": 1.8498280224326327, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.2822030782699585, + "step": 2490, + "unclipped_grad_norm": 0.6788212060928345 + }, + { + "bits_per_byte": 1.819267221322264, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2610199451446533, + "step": 2500, + "unclipped_grad_norm": 0.6574063301086426 + }, + { + "bits_per_byte": 1.7604587857556317, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.220257043838501, + "step": 2510, + "unclipped_grad_norm": 0.6887367963790894 + }, + { + "bits_per_byte": 1.7853797597962562, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.2375309467315674, + "step": 2520, + "unclipped_grad_norm": 0.6420572400093079 + }, + { + "bits_per_byte": 1.8162004266920784, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.2588942050933838, + "step": 2530, + "unclipped_grad_norm": 0.7083710432052612 + }, + { + "bits_per_byte": 1.8125803638740672, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.2563849687576294, + "step": 2540, + "unclipped_grad_norm": 0.691429615020752 + }, + { + "bits_per_byte": 1.758893055702177, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.2191717624664307, + "step": 2550, + "unclipped_grad_norm": 0.6528177857398987 + }, + { + "bits_per_byte": 1.7318646062578795, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.200437068939209, + "step": 2560, + "unclipped_grad_norm": 0.6268978714942932 + }, + { + "bits_per_byte": 1.7715970701561108, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2279775142669678, + "step": 2570, + "unclipped_grad_norm": 0.6713714003562927 + }, + { + "bits_per_byte": 1.7430132096174102, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.2081646919250488, + "step": 2580, + "unclipped_grad_norm": 0.6750173568725586 + }, + { + "bits_per_byte": 1.7519779772762851, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.2143785953521729, + "step": 2590, + "unclipped_grad_norm": 0.7115974426269531 + }, + { + "bits_per_byte": 1.77463669152746, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.2300844192504883, + "step": 2600, + "unclipped_grad_norm": 0.6391100287437439 + }, + { + "bits_per_byte": 1.7960966867021213, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.2449593544006348, + "step": 2610, + "unclipped_grad_norm": 0.751370370388031 + }, + { + "bits_per_byte": 1.7205777288470552, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.1926136016845703, + "step": 2620, + "unclipped_grad_norm": 0.6383137702941895 + }, + { + "bits_per_byte": 1.729081239036316, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.1985077857971191, + "step": 2630, + "unclipped_grad_norm": 0.6266732215881348 + }, + { + "bits_per_byte": 1.7018415948967591, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.179626703262329, + "step": 2640, + "unclipped_grad_norm": 0.6449143886566162 + }, + { + "bits_per_byte": 1.7776119913873867, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2321467399597168, + "step": 2650, + "unclipped_grad_norm": 0.6444293260574341 + }, + { + "bits_per_byte": 1.6937511870348707, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.1740188598632812, + "step": 2660, + "unclipped_grad_norm": 0.6434645056724548 + }, + { + "bits_per_byte": 1.7669996299332462, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.2247908115386963, + "step": 2670, + "unclipped_grad_norm": 0.6679447889328003 + }, + { + "bits_per_byte": 1.8612896342167335, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.2901476621627808, + "step": 2680, + "unclipped_grad_norm": 0.7220403552055359 + }, + { + "bits_per_byte": 1.7907739956405682, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.2412699460983276, + "step": 2690, + "unclipped_grad_norm": 0.6872449517250061 + }, + { + "bits_per_byte": 1.705604919262939, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.1822352409362793, + "step": 2700, + "unclipped_grad_norm": 0.6397694945335388 + }, + { + "bits_per_byte": 1.7650247531533885, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.2234219312667847, + "step": 2710, + "unclipped_grad_norm": 0.6128670573234558 + }, + { + "bits_per_byte": 1.827526172181998, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.266744613647461, + "step": 2720, + "unclipped_grad_norm": 0.7149207592010498 + }, + { + "bits_per_byte": 1.7201419248097687, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.1923115253448486, + "step": 2730, + "unclipped_grad_norm": 0.7096569538116455 + }, + { + "bits_per_byte": 1.7837400772029302, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.2363944053649902, + "step": 2740, + "unclipped_grad_norm": 0.6388362646102905 + }, + { + "bits_per_byte": 1.7385579990468092, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.2050765752792358, + "step": 2750, + "unclipped_grad_norm": 0.6091862320899963 + }, + { + "bits_per_byte": 1.8668833699360465, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.29402494430542, + "step": 2760, + "unclipped_grad_norm": 0.7161790132522583 + }, + { + "bits_per_byte": 1.7938732949917668, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.2434182167053223, + "step": 2770, + "unclipped_grad_norm": 0.6475892066955566 + }, + { + "bits_per_byte": 1.8032145326733804, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.249893069267273, + "step": 2780, + "unclipped_grad_norm": 0.6787914037704468 + }, + { + "bits_per_byte": 1.7589897099519618, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2192387580871582, + "step": 2790, + "unclipped_grad_norm": 0.6403056383132935 + }, + { + "bits_per_byte": 1.7714065133789547, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.2278454303741455, + "step": 2800, + "unclipped_grad_norm": 0.685519814491272 + }, + { + "bits_per_byte": 1.719603619112568, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.1919384002685547, + "step": 2810, + "unclipped_grad_norm": 0.6321927309036255 + }, + { + "bits_per_byte": 1.7901433352598544, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.240832805633545, + "step": 2820, + "unclipped_grad_norm": 0.6286884546279907 + }, + { + "bits_per_byte": 1.729210226024463, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.1985971927642822, + "step": 2830, + "unclipped_grad_norm": 0.6264799237251282 + }, + { + "bits_per_byte": 1.7584152878980799, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.2188405990600586, + "step": 2840, + "unclipped_grad_norm": 0.6248931884765625 + }, + { + "bits_per_byte": 1.7070361588834193, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.183227300643921, + "step": 2850, + "unclipped_grad_norm": 0.6134223341941833 + }, + { + "bits_per_byte": 1.744675937885952, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.2093172073364258, + "step": 2860, + "unclipped_grad_norm": 0.6612468361854553 + }, + { + "bits_per_byte": 1.7373642674671705, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.2042491436004639, + "step": 2870, + "unclipped_grad_norm": 0.699167788028717 + }, + { + "bits_per_byte": 1.7492565238090318, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2124922275543213, + "step": 2880, + "unclipped_grad_norm": 0.6605263948440552 + }, + { + "bits_per_byte": 1.7049021981515136, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.1817481517791748, + "step": 2890, + "unclipped_grad_norm": 0.607163667678833 + }, + { + "bits_per_byte": 1.7306813656199433, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.1996169090270996, + "step": 2900, + "unclipped_grad_norm": 0.6622748374938965 + }, + { + "bits_per_byte": 1.700074129193842, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.1784015893936157, + "step": 2910, + "unclipped_grad_norm": 0.6293155550956726 + }, + { + "bits_per_byte": 1.7467608835623618, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.21076238155365, + "step": 2920, + "unclipped_grad_norm": 0.623350977897644 + }, + { + "bits_per_byte": 1.711529377619861, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.1863417625427246, + "step": 2930, + "unclipped_grad_norm": 0.6433477997779846 + }, + { + "bits_per_byte": 1.721818927638332, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.1934739351272583, + "step": 2940, + "unclipped_grad_norm": 0.7278959155082703 + }, + { + "bits_per_byte": 1.7763260771068854, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2312554121017456, + "step": 2950, + "unclipped_grad_norm": 0.6551075577735901 + }, + { + "bits_per_byte": 1.7927103483066327, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.2426121234893799, + "step": 2960, + "unclipped_grad_norm": 0.7276303768157959 + }, + { + "bits_per_byte": 1.7394328747917482, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.2056829929351807, + "step": 2970, + "unclipped_grad_norm": 0.6340150833129883 + }, + { + "bits_per_byte": 1.7752078459109755, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.2304803133010864, + "step": 2980, + "unclipped_grad_norm": 0.6536198854446411 + }, + { + "bits_per_byte": 1.7205220064681757, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.1925749778747559, + "step": 2990, + "unclipped_grad_norm": 0.677970290184021 + }, + { + "bits_per_byte": 1.795457771154166, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.2445164918899536, + "step": 3000, + "unclipped_grad_norm": 0.6789783835411072 + }, + { + "bits_per_byte": 1.679561586442784, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.1641833782196045, + "step": 3010, + "unclipped_grad_norm": 0.6741669774055481 + }, + { + "bits_per_byte": 1.7512315725715404, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.2138612270355225, + "step": 3020, + "unclipped_grad_norm": 0.6495442390441895 + }, + { + "bits_per_byte": 1.6812504560742567, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.1653540134429932, + "step": 3030, + "unclipped_grad_norm": 0.6156594157218933 + }, + { + "bits_per_byte": 1.6871945204533771, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.1694741249084473, + "step": 3040, + "unclipped_grad_norm": 0.6149004697799683 + }, + { + "bits_per_byte": 1.8287646192508609, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.2676030397415161, + "step": 3050, + "unclipped_grad_norm": 0.7062676548957825 + }, + { + "bits_per_byte": 1.7642879794770923, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.2229112386703491, + "step": 3060, + "unclipped_grad_norm": 0.6970083117485046 + }, + { + "bits_per_byte": 1.685780651080634, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.1684941053390503, + "step": 3070, + "unclipped_grad_norm": 0.6777254343032837 + }, + { + "bits_per_byte": 1.7774056122063513, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.2320036888122559, + "step": 3080, + "unclipped_grad_norm": 0.6711161136627197 + }, + { + "bits_per_byte": 1.7511767101059152, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.2138231992721558, + "step": 3090, + "unclipped_grad_norm": 0.6319589614868164 + }, + { + "bits_per_byte": 1.696530082707512, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.1759450435638428, + "step": 3100, + "unclipped_grad_norm": 0.6250280737876892 + }, + { + "bits_per_byte": 1.8466953584471666, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.280031681060791, + "step": 3110, + "unclipped_grad_norm": 0.7225695848464966 + }, + { + "bits_per_byte": 1.7298647919936467, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.1990509033203125, + "step": 3120, + "unclipped_grad_norm": 0.6621631383895874 + }, + { + "bits_per_byte": 1.683061433387842, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.166609287261963, + "step": 3130, + "unclipped_grad_norm": 0.6796402931213379 + }, + { + "bits_per_byte": 1.7110302839670573, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.1859958171844482, + "step": 3140, + "unclipped_grad_norm": 0.6529068946838379 + }, + { + "bits_per_byte": 1.6967134162133317, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.176072120666504, + "step": 3150, + "unclipped_grad_norm": 0.6252351403236389 + }, + { + "bits_per_byte": 1.7020041185018244, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.1797393560409546, + "step": 3160, + "unclipped_grad_norm": 0.612695574760437 + }, + { + "bits_per_byte": 1.7644205781009075, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.2230031490325928, + "step": 3170, + "unclipped_grad_norm": 0.6548941135406494 + }, + { + "bits_per_byte": 1.7526794945091544, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.2148648500442505, + "step": 3180, + "unclipped_grad_norm": 0.6844728589057922 + }, + { + "bits_per_byte": 1.7344025542366621, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.2021962404251099, + "step": 3190, + "unclipped_grad_norm": 0.6939358115196228 + }, + { + "bits_per_byte": 1.7150146060395959, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.1887575387954712, + "step": 3200, + "unclipped_grad_norm": 0.6091709733009338 + }, + { + "bits_per_byte": 1.6992030370672218, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.177797794342041, + "step": 3210, + "unclipped_grad_norm": 0.6184134483337402 + }, + { + "bits_per_byte": 1.670590627408478, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.1579651832580566, + "step": 3220, + "unclipped_grad_norm": 0.6558188796043396 + }, + { + "bits_per_byte": 1.721338752077123, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.1931411027908325, + "step": 3230, + "unclipped_grad_norm": 0.6356065273284912 + }, + { + "bits_per_byte": 1.7374605777516536, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.2043159008026123, + "step": 3240, + "unclipped_grad_norm": 0.6890019774436951 + }, + { + "bits_per_byte": 1.712212492709088, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.1868152618408203, + "step": 3250, + "unclipped_grad_norm": 0.5957981944084167 + }, + { + "bits_per_byte": 1.710690790214254, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.185760498046875, + "step": 3260, + "unclipped_grad_norm": 0.6608059406280518 + }, + { + "bits_per_byte": 1.7523938313260712, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.2146668434143066, + "step": 3270, + "unclipped_grad_norm": 0.6652039289474487 + }, + { + "bits_per_byte": 1.7481873076686179, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.211751103401184, + "step": 3280, + "unclipped_grad_norm": 0.6477850079536438 + }, + { + "bits_per_byte": 1.6876074507980987, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.1697603464126587, + "step": 3290, + "unclipped_grad_norm": 0.665995717048645 + }, + { + "bits_per_byte": 1.7257557824992327, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.1962027549743652, + "step": 3300, + "unclipped_grad_norm": 0.661629855632782 + }, + { + "bits_per_byte": 1.6905997769404606, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.1718344688415527, + "step": 3310, + "unclipped_grad_norm": 0.6472657322883606 + }, + { + "bits_per_byte": 1.6963439974792784, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.1758160591125488, + "step": 3320, + "unclipped_grad_norm": 0.633185088634491 + }, + { + "bits_per_byte": 1.696641527465271, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.1760222911834717, + "step": 3330, + "unclipped_grad_norm": 0.6425480246543884 + }, + { + "bits_per_byte": 1.6414856594376632, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.1377911567687988, + "step": 3340, + "unclipped_grad_norm": 0.6628016829490662 + }, + { + "bits_per_byte": 1.690492803731624, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.1717603206634521, + "step": 3350, + "unclipped_grad_norm": 0.6525082588195801 + }, + { + "bits_per_byte": 1.7070523252526004, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.1832385063171387, + "step": 3360, + "unclipped_grad_norm": 0.6502514481544495 + }, + { + "bits_per_byte": 1.655162751730179, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.1472713947296143, + "step": 3370, + "unclipped_grad_norm": 0.613136887550354 + }, + { + "bits_per_byte": 1.6913382704432656, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.1723463535308838, + "step": 3380, + "unclipped_grad_norm": 0.6373270750045776 + }, + { + "bits_per_byte": 1.707975528122432, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.1838784217834473, + "step": 3390, + "unclipped_grad_norm": 0.6745176911354065 + }, + { + "bits_per_byte": 1.695567323827982, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.1752777099609375, + "step": 3400, + "unclipped_grad_norm": 0.7079716324806213 + }, + { + "bits_per_byte": 1.6844483014843998, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.1675705909729004, + "step": 3410, + "unclipped_grad_norm": 0.6367164850234985 + }, + { + "bits_per_byte": 1.6666838695114785, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.155257225036621, + "step": 3420, + "unclipped_grad_norm": 0.625562846660614 + }, + { + "bits_per_byte": 1.6982894652258387, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1771645545959473, + "step": 3430, + "unclipped_grad_norm": 0.6201092004776001 + }, + { + "bits_per_byte": 1.7075823757825597, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.1836059093475342, + "step": 3440, + "unclipped_grad_norm": 0.64792799949646 + }, + { + "bits_per_byte": 1.7785049253106664, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.2327656745910645, + "step": 3450, + "unclipped_grad_norm": 0.7012131214141846 + }, + { + "bits_per_byte": 1.7760039536018193, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.231032133102417, + "step": 3460, + "unclipped_grad_norm": 0.6784840822219849 + }, + { + "bits_per_byte": 1.7340295238669405, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.2019376754760742, + "step": 3470, + "unclipped_grad_norm": 0.6913369297981262 + }, + { + "bits_per_byte": 1.7042603588984935, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.1813032627105713, + "step": 3480, + "unclipped_grad_norm": 0.6801669597625732 + }, + { + "bits_per_byte": 1.626472949878547, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.127385139465332, + "step": 3490, + "unclipped_grad_norm": 0.6522605419158936 + }, + { + "bits_per_byte": 1.6376270566829052, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1351165771484375, + "step": 3500, + "unclipped_grad_norm": 0.6360974907875061 + }, + { + "bits_per_byte": 1.6932747950919809, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.1736886501312256, + "step": 3510, + "unclipped_grad_norm": 0.7047001719474792 + }, + { + "bits_per_byte": 1.6593868176180204, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.150199294090271, + "step": 3520, + "unclipped_grad_norm": 0.6179392337799072 + }, + { + "bits_per_byte": 1.666585323452534, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.1551889181137085, + "step": 3530, + "unclipped_grad_norm": 0.6735309958457947 + }, + { + "bits_per_byte": 1.6865141570865638, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.1690025329589844, + "step": 3540, + "unclipped_grad_norm": 0.7295958399772644 + }, + { + "bits_per_byte": 1.7265307363240203, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.196739912033081, + "step": 3550, + "unclipped_grad_norm": 0.6688145399093628 + }, + { + "bits_per_byte": 1.6528715988553846, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.1456832885742188, + "step": 3560, + "unclipped_grad_norm": 0.6610836982727051 + }, + { + "bits_per_byte": 1.709095135179549, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.1846544742584229, + "step": 3570, + "unclipped_grad_norm": 0.6575421690940857 + }, + { + "bits_per_byte": 1.7747852845378056, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.2301874160766602, + "step": 3580, + "unclipped_grad_norm": 0.7230638265609741 + }, + { + "bits_per_byte": 1.7046134392807148, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.181547999382019, + "step": 3590, + "unclipped_grad_norm": 0.6479980945587158 + }, + { + "bits_per_byte": 1.6810839768682215, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.165238618850708, + "step": 3600, + "unclipped_grad_norm": 0.6374408602714539 + }, + { + "bits_per_byte": 1.6682251780285111, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.1563255786895752, + "step": 3610, + "unclipped_grad_norm": 0.6649436354637146 + }, + { + "bits_per_byte": 1.664760243561578, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.1539238691329956, + "step": 3620, + "unclipped_grad_norm": 0.641756534576416 + }, + { + "bits_per_byte": 1.6883313257755803, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.170262098312378, + "step": 3630, + "unclipped_grad_norm": 0.6772762537002563 + }, + { + "bits_per_byte": 1.6317875577555094, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.1310689449310303, + "step": 3640, + "unclipped_grad_norm": 0.6365849375724792 + }, + { + "bits_per_byte": 1.7063038567560456, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.1827197074890137, + "step": 3650, + "unclipped_grad_norm": 0.6643222570419312 + }, + { + "bits_per_byte": 1.6214374698439349, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.1238948106765747, + "step": 3660, + "unclipped_grad_norm": 0.6767879128456116 + }, + { + "bits_per_byte": 1.6360575430111313, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.134028673171997, + "step": 3670, + "unclipped_grad_norm": 0.6129776239395142 + }, + { + "bits_per_byte": 1.6959006262053542, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.175508737564087, + "step": 3680, + "unclipped_grad_norm": 0.6803691983222961 + }, + { + "bits_per_byte": 1.6526397662420216, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.1455225944519043, + "step": 3690, + "unclipped_grad_norm": 0.6290789246559143 + }, + { + "bits_per_byte": 1.5991261605475027, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.1084297895431519, + "step": 3700, + "unclipped_grad_norm": 0.6236196756362915 + }, + { + "bits_per_byte": 1.6705627662190383, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.1579458713531494, + "step": 3710, + "unclipped_grad_norm": 0.6582556962966919 + }, + { + "bits_per_byte": 1.6625043471302106, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.152360200881958, + "step": 3720, + "unclipped_grad_norm": 0.637691080570221 + }, + { + "bits_per_byte": 1.7223126898289591, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.1938161849975586, + "step": 3730, + "unclipped_grad_norm": 0.6756547093391418 + }, + { + "bits_per_byte": 1.6450293619586913, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.1402474641799927, + "step": 3740, + "unclipped_grad_norm": 0.6311995983123779 + }, + { + "bits_per_byte": 1.6403206489607185, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1369836330413818, + "step": 3750, + "unclipped_grad_norm": 0.6675049662590027 + }, + { + "bits_per_byte": 1.6243938516122665, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.1259440183639526, + "step": 3760, + "unclipped_grad_norm": 0.6680413484573364 + }, + { + "bits_per_byte": 1.6824844315941974, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.16620934009552, + "step": 3770, + "unclipped_grad_norm": 0.7039961814880371 + }, + { + "bits_per_byte": 1.6268635224786565, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.1276558637619019, + "step": 3780, + "unclipped_grad_norm": 0.6569859385490417 + }, + { + "bits_per_byte": 1.678115900279631, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.1631813049316406, + "step": 3790, + "unclipped_grad_norm": 0.6454007029533386 + }, + { + "bits_per_byte": 1.6168016774899279, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.1206815242767334, + "step": 3800, + "unclipped_grad_norm": 0.6339223384857178 + }, + { + "bits_per_byte": 1.6107516717978758, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.116487979888916, + "step": 3810, + "unclipped_grad_norm": 0.6380460262298584 + }, + { + "bits_per_byte": 1.71881662650222, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1913928985595703, + "step": 3820, + "unclipped_grad_norm": 0.6510602831840515 + }, + { + "bits_per_byte": 1.6377271505857074, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.135185956954956, + "step": 3830, + "unclipped_grad_norm": 0.6695832014083862 + }, + { + "bits_per_byte": 1.6462093349262612, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.1410653591156006, + "step": 3840, + "unclipped_grad_norm": 0.6693991422653198 + }, + { + "bits_per_byte": 1.6724212107442618, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1592340469360352, + "step": 3850, + "unclipped_grad_norm": 0.6494592428207397 + }, + { + "bits_per_byte": 1.6067009644221042, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.1136802434921265, + "step": 3860, + "unclipped_grad_norm": 0.6467571258544922 + }, + { + "bits_per_byte": 1.6829503325953847, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.166532278060913, + "step": 3870, + "unclipped_grad_norm": 0.7071244716644287 + }, + { + "bits_per_byte": 1.636628181446694, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1344242095947266, + "step": 3880, + "unclipped_grad_norm": 0.6827104091644287 + }, + { + "bits_per_byte": 1.6781810817043081, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.1632264852523804, + "step": 3890, + "unclipped_grad_norm": 0.6936315298080444 + }, + { + "bits_per_byte": 1.6639765186215962, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.1533806324005127, + "step": 3900, + "unclipped_grad_norm": 0.6743345856666565 + }, + { + "bits_per_byte": 1.6938478412846556, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1740858554840088, + "step": 3910, + "unclipped_grad_norm": 0.6842510104179382 + }, + { + "bits_per_byte": 1.6475088358361805, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.1419661045074463, + "step": 3920, + "unclipped_grad_norm": 0.6473428010940552 + }, + { + "bits_per_byte": 1.6155436243988663, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.1198095083236694, + "step": 3930, + "unclipped_grad_norm": 0.6767602562904358 + }, + { + "bits_per_byte": 1.6222624706201236, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.1244666576385498, + "step": 3940, + "unclipped_grad_norm": 0.6448774933815002 + }, + { + "bits_per_byte": 1.6972190452068685, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1764225959777832, + "step": 3950, + "unclipped_grad_norm": 0.6874628663063049 + }, + { + "bits_per_byte": 1.714906600934854, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.1886826753616333, + "step": 3960, + "unclipped_grad_norm": 0.6846359372138977 + }, + { + "bits_per_byte": 1.6900645669309757, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1714634895324707, + "step": 3970, + "unclipped_grad_norm": 0.6600151062011719 + }, + { + "bits_per_byte": 1.6791825366802824, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.1639206409454346, + "step": 3980, + "unclipped_grad_norm": 0.6519694924354553 + }, + { + "bits_per_byte": 1.7028239597984873, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1803076267242432, + "step": 3990, + "unclipped_grad_norm": 0.7655619978904724 + }, + { + "bits_per_byte": 1.6840544612139239, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.167297601699829, + "step": 4000, + "unclipped_grad_norm": 0.676017701625824 + }, + { + "bits_per_byte": 1.7132636506711616, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.1875438690185547, + "step": 4010, + "unclipped_grad_norm": 0.7318490743637085 + }, + { + "bits_per_byte": 1.590128716184964, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.1021932363510132, + "step": 4020, + "unclipped_grad_norm": 0.6530770063400269 + }, + { + "bits_per_byte": 1.621146303216024, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.1236929893493652, + "step": 4030, + "unclipped_grad_norm": 0.6695507764816284 + }, + { + "bits_per_byte": 1.6341364968009937, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1326971054077148, + "step": 4040, + "unclipped_grad_norm": 0.6895481944084167 + }, + { + "bits_per_byte": 1.6197804170028716, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.122746229171753, + "step": 4050, + "unclipped_grad_norm": 0.6896605491638184 + }, + { + "bits_per_byte": 1.640764020234643, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1372909545898438, + "step": 4060, + "unclipped_grad_norm": 0.6854106783866882 + }, + { + "bits_per_byte": 1.60420050866121, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.1119470596313477, + "step": 4070, + "unclipped_grad_norm": 0.6506727337837219 + }, + { + "bits_per_byte": 1.6444716222219433, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.1398608684539795, + "step": 4080, + "unclipped_grad_norm": 0.6733965873718262 + }, + { + "bits_per_byte": 1.6019886397884633, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.110413908958435, + "step": 4090, + "unclipped_grad_norm": 0.638359546661377 + }, + { + "bits_per_byte": 1.5641801458054343, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.0842070579528809, + "step": 4100, + "unclipped_grad_norm": 0.6245489120483398 + }, + { + "bits_per_byte": 1.6799829439373979, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.164475440979004, + "step": 4110, + "unclipped_grad_norm": 0.7018517255783081 + }, + { + "bits_per_byte": 1.614404583302202, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.1190199851989746, + "step": 4120, + "unclipped_grad_norm": 0.6595736742019653 + }, + { + "bits_per_byte": 1.5879756653788124, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.100700855255127, + "step": 4130, + "unclipped_grad_norm": 0.6557380557060242 + }, + { + "bits_per_byte": 1.6227443660078413, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.124800682067871, + "step": 4140, + "unclipped_grad_norm": 0.6592831015586853 + }, + { + "bits_per_byte": 1.6529744444806005, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1457545757293701, + "step": 4150, + "unclipped_grad_norm": 0.6795181632041931 + }, + { + "bits_per_byte": 1.6335146075354738, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.1322660446166992, + "step": 4160, + "unclipped_grad_norm": 0.6809347867965698 + }, + { + "bits_per_byte": 1.5837517714736218, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.0977730751037598, + "step": 4170, + "unclipped_grad_norm": 0.655536949634552 + }, + { + "bits_per_byte": 1.570588907307186, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.0886492729187012, + "step": 4180, + "unclipped_grad_norm": 0.6697047352790833 + }, + { + "bits_per_byte": 1.71080361083322, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.1858386993408203, + "step": 4190, + "unclipped_grad_norm": 0.6862300634384155 + }, + { + "bits_per_byte": 1.6084414288488358, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1148866415023804, + "step": 4200, + "unclipped_grad_norm": 0.6557636857032776 + }, + { + "bits_per_byte": 1.620260248598779, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.1230788230895996, + "step": 4210, + "unclipped_grad_norm": 0.6884835362434387 + }, + { + "bits_per_byte": 1.6521675018827524, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.1451952457427979, + "step": 4220, + "unclipped_grad_norm": 0.6960432529449463 + }, + { + "bits_per_byte": 1.638372085526443, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.1356329917907715, + "step": 4230, + "unclipped_grad_norm": 0.6713078022003174 + }, + { + "bits_per_byte": 1.6761331122978338, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.1618069410324097, + "step": 4240, + "unclipped_grad_norm": 0.6890337467193604 + }, + { + "bits_per_byte": 1.6955432462568614, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1752610206604004, + "step": 4250, + "unclipped_grad_norm": 0.6852550506591797 + }, + { + "bits_per_byte": 1.6623471549873219, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1522512435913086, + "step": 4260, + "unclipped_grad_norm": 0.6862250566482544 + }, + { + "bits_per_byte": 1.650103366107097, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.1437644958496094, + "step": 4270, + "unclipped_grad_norm": 0.6657096743583679 + }, + { + "bits_per_byte": 1.6880485862975618, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1700661182403564, + "step": 4280, + "unclipped_grad_norm": 0.7050718069076538 + }, + { + "bits_per_byte": 1.6334869183286849, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.1322468519210815, + "step": 4290, + "unclipped_grad_norm": 0.6843206286430359 + }, + { + "bits_per_byte": 1.5686836835009277, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.0873286724090576, + "step": 4300, + "unclipped_grad_norm": 0.6503427624702454 + }, + { + "bits_per_byte": 1.7074250116570202, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.1834968328475952, + "step": 4310, + "unclipped_grad_norm": 0.744146466255188 + }, + { + "bits_per_byte": 1.5816374167639144, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.0963075160980225, + "step": 4320, + "unclipped_grad_norm": 0.6721951365470886 + }, + { + "bits_per_byte": 1.5801403077881537, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.095269799232483, + "step": 4330, + "unclipped_grad_norm": 0.6572669148445129 + }, + { + "bits_per_byte": 1.5760342219988042, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.092423677444458, + "step": 4340, + "unclipped_grad_norm": 0.6564091444015503 + }, + { + "bits_per_byte": 1.5089250437882784, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.0459071397781372, + "step": 4350, + "unclipped_grad_norm": 0.6355080604553223 + }, + { + "bits_per_byte": 1.6662786783860457, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.1549763679504395, + "step": 4360, + "unclipped_grad_norm": 0.7146135568618774 + }, + { + "bits_per_byte": 1.6108194329623158, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.116534948348999, + "step": 4370, + "unclipped_grad_norm": 0.6846863031387329 + }, + { + "bits_per_byte": 1.5770229502586144, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.0931090116500854, + "step": 4380, + "unclipped_grad_norm": 0.7024040222167969 + }, + { + "bits_per_byte": 1.6186181582483408, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1219406127929688, + "step": 4390, + "unclipped_grad_norm": 0.682716429233551 + }, + { + "bits_per_byte": 1.5735174278860777, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.0906791687011719, + "step": 4400, + "unclipped_grad_norm": 0.6795985102653503 + }, + { + "bits_per_byte": 1.574473651424875, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.0913419723510742, + "step": 4410, + "unclipped_grad_norm": 0.6700407266616821 + }, + { + "bits_per_byte": 1.5842102772208222, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.0980908870697021, + "step": 4420, + "unclipped_grad_norm": 0.6867445707321167 + }, + { + "bits_per_byte": 1.5812563032096025, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.096043348312378, + "step": 4430, + "unclipped_grad_norm": 0.6580121517181396 + }, + { + "bits_per_byte": 1.6016384831113066, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.1101711988449097, + "step": 4440, + "unclipped_grad_norm": 0.7302095293998718 + }, + { + "bits_per_byte": 1.569942080557291, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.0882009267807007, + "step": 4450, + "unclipped_grad_norm": 0.6726658344268799 + }, + { + "bits_per_byte": 1.5514498180059182, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.0753830671310425, + "step": 4460, + "unclipped_grad_norm": 0.6564345955848694 + }, + { + "bits_per_byte": 1.6380475542642647, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1354080438613892, + "step": 4470, + "unclipped_grad_norm": 0.6903609037399292 + }, + { + "bits_per_byte": 1.5681226760938132, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.086939811706543, + "step": 4480, + "unclipped_grad_norm": 0.6648551225662231 + }, + { + "bits_per_byte": 1.520411249091453, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.0538687705993652, + "step": 4490, + "unclipped_grad_norm": 0.6600596308708191 + }, + { + "bits_per_byte": 1.6095809858934527, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.1156765222549438, + "step": 4500, + "unclipped_grad_norm": 0.6929576396942139 + }, + { + "bits_per_byte": 1.5787192151440743, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.0942847728729248, + "step": 4510, + "unclipped_grad_norm": 0.6750292181968689 + }, + { + "bits_per_byte": 1.5724538871731422, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.0899419784545898, + "step": 4520, + "unclipped_grad_norm": 0.6638123393058777 + }, + { + "bits_per_byte": 1.5872187697323652, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.100176215171814, + "step": 4530, + "unclipped_grad_norm": 0.6703746914863586 + }, + { + "bits_per_byte": 1.5531421272904082, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.0765560865402222, + "step": 4540, + "unclipped_grad_norm": 0.6724599003791809 + }, + { + "bits_per_byte": 1.6136201704316167, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.1184762716293335, + "step": 4550, + "unclipped_grad_norm": 0.6803196668624878 + }, + { + "bits_per_byte": 1.6182676576058823, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.1216976642608643, + "step": 4560, + "unclipped_grad_norm": 0.6897071003913879 + }, + { + "bits_per_byte": 1.5658268796874457, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.0853484869003296, + "step": 4570, + "unclipped_grad_norm": 0.6798649430274963 + }, + { + "bits_per_byte": 1.5325105725449533, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.0622553825378418, + "step": 4580, + "unclipped_grad_norm": 0.6785328388214111 + }, + { + "bits_per_byte": 1.6556398316036725, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.1476020812988281, + "step": 4590, + "unclipped_grad_norm": 0.7550393342971802 + }, + { + "bits_per_byte": 1.6130273462340925, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.118065357208252, + "step": 4600, + "unclipped_grad_norm": 0.6955652236938477 + }, + { + "bits_per_byte": 1.5850409534244894, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.0986666679382324, + "step": 4610, + "unclipped_grad_norm": 0.6908915042877197 + }, + { + "bits_per_byte": 1.533505148214893, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.062944769859314, + "step": 4620, + "unclipped_grad_norm": 0.6388741731643677 + }, + { + "bits_per_byte": 1.6043164249678914, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.1120274066925049, + "step": 4630, + "unclipped_grad_norm": 0.7149990797042847 + }, + { + "bits_per_byte": 1.5765895539784403, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.0928086042404175, + "step": 4640, + "unclipped_grad_norm": 0.6664532423019409 + }, + { + "bits_per_byte": 1.547051189727451, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.0723341703414917, + "step": 4650, + "unclipped_grad_norm": 0.695128321647644 + }, + { + "bits_per_byte": 1.5772393044333999, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.0932589769363403, + "step": 4660, + "unclipped_grad_norm": 0.7180529236793518 + }, + { + "bits_per_byte": 1.6033039631022619, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.1113256216049194, + "step": 4670, + "unclipped_grad_norm": 0.6803218722343445 + }, + { + "bits_per_byte": 1.5572786540089605, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.0794233083724976, + "step": 4680, + "unclipped_grad_norm": 0.7060900926589966 + }, + { + "bits_per_byte": 1.5338437820544417, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.0631794929504395, + "step": 4690, + "unclipped_grad_norm": 0.6773106455802917 + }, + { + "bits_per_byte": 1.654057247250433, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.1465051174163818, + "step": 4700, + "unclipped_grad_norm": 0.7309046983718872 + }, + { + "bits_per_byte": 1.5587661319562731, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.0804543495178223, + "step": 4710, + "unclipped_grad_norm": 0.7153287529945374 + }, + { + "bits_per_byte": 1.539003261580326, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.066755771636963, + "step": 4720, + "unclipped_grad_norm": 0.7165635228157043 + }, + { + "bits_per_byte": 1.639170600974399, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.1361864805221558, + "step": 4730, + "unclipped_grad_norm": 0.6902788281440735 + }, + { + "bits_per_byte": 1.6006057272928753, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.1094553470611572, + "step": 4740, + "unclipped_grad_norm": 0.7176128029823303 + }, + { + "bits_per_byte": 1.6268796888478374, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.1276670694351196, + "step": 4750, + "unclipped_grad_norm": 0.706055223941803 + }, + { + "bits_per_byte": 1.5135297072824794, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.0490988492965698, + "step": 4760, + "unclipped_grad_norm": 0.6362533569335938 + }, + { + "bits_per_byte": 1.621320693623999, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.1238138675689697, + "step": 4770, + "unclipped_grad_norm": 0.703389585018158 + }, + { + "bits_per_byte": 1.566306539300702, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.0856809616088867, + "step": 4780, + "unclipped_grad_norm": 0.6690196394920349 + }, + { + "bits_per_byte": 1.5902358613764516, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.1022675037384033, + "step": 4790, + "unclipped_grad_norm": 0.722443163394928 + }, + { + "bits_per_byte": 1.5643378538962753, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.084316372871399, + "step": 4800, + "unclipped_grad_norm": 0.7233239412307739 + }, + { + "bits_per_byte": 1.5346613875766435, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.0637462139129639, + "step": 4810, + "unclipped_grad_norm": 0.7002009749412537 + }, + { + "bits_per_byte": 1.6033634690994605, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.111366868019104, + "step": 4820, + "unclipped_grad_norm": 0.7495388984680176 + }, + { + "bits_per_byte": 1.5968030188996478, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.1068195104599, + "step": 4830, + "unclipped_grad_norm": 0.7439894080162048 + }, + { + "bits_per_byte": 1.6027377962156215, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.1109331846237183, + "step": 4840, + "unclipped_grad_norm": 0.6802996397018433 + }, + { + "bits_per_byte": 1.5407253238634153, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.0679494142532349, + "step": 4850, + "unclipped_grad_norm": 0.7061647772789001 + }, + { + "bits_per_byte": 1.5594722927207156, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.0809438228607178, + "step": 4860, + "unclipped_grad_norm": 0.7636238932609558 + }, + { + "bits_per_byte": 1.558933987023515, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.0805706977844238, + "step": 4870, + "unclipped_grad_norm": 0.7073072791099548 + }, + { + "bits_per_byte": 1.7145951403541415, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.1884667873382568, + "step": 4880, + "unclipped_grad_norm": 0.7676188945770264 + }, + { + "bits_per_byte": 1.512548546259307, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.0484187602996826, + "step": 4890, + "unclipped_grad_norm": 0.6739963293075562 + }, + { + "bits_per_byte": 1.6008265530165833, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.1096084117889404, + "step": 4900, + "unclipped_grad_norm": 0.7072500586509705 + }, + { + "bits_per_byte": 1.5881724135313995, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.100837230682373, + "step": 4910, + "unclipped_grad_norm": 0.7132666110992432 + }, + { + "bits_per_byte": 1.514549220436794, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.0498055219650269, + "step": 4920, + "unclipped_grad_norm": 0.6661472320556641 + }, + { + "bits_per_byte": 1.5352091523196416, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.064125895500183, + "step": 4930, + "unclipped_grad_norm": 0.6741364598274231 + }, + { + "bits_per_byte": 1.5512379353800554, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.075236201286316, + "step": 4940, + "unclipped_grad_norm": 0.7382766008377075 + }, + { + "bits_per_byte": 1.580290448642357, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.0953738689422607, + "step": 4950, + "unclipped_grad_norm": 0.7024677991867065 + }, + { + "bits_per_byte": 1.5946062845001772, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.1052968502044678, + "step": 4960, + "unclipped_grad_norm": 0.7240102887153625 + }, + { + "bits_per_byte": 1.5462110544779863, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.0717518329620361, + "step": 4970, + "unclipped_grad_norm": 0.718582272529602 + }, + { + "bits_per_byte": 1.6141228757200887, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.1188247203826904, + "step": 4980, + "unclipped_grad_norm": 0.7391600608825684 + }, + { + "bits_per_byte": 1.5603070965080037, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.0815224647521973, + "step": 4990, + "unclipped_grad_norm": 0.7530345916748047 + }, + { + "bits_per_byte": 1.5716490083671042, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.0893840789794922, + "step": 5000, + "unclipped_grad_norm": 0.707313597202301 + }, + { + "bits_per_byte": 1.6092638498852618, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.1154567003250122, + "step": 5010, + "unclipped_grad_norm": 0.705553412437439 + }, + { + "bits_per_byte": 1.5863175806418441, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.0995515584945679, + "step": 5020, + "unclipped_grad_norm": 0.7448808550834656 + }, + { + "bits_per_byte": 1.518654790278191, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.052651286125183, + "step": 5030, + "unclipped_grad_norm": 0.6777296662330627 + }, + { + "bits_per_byte": 1.5762264986024688, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.0925569534301758, + "step": 5040, + "unclipped_grad_norm": 0.7256330847740173 + }, + { + "bits_per_byte": 1.5364531028333321, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.064988136291504, + "step": 5050, + "unclipped_grad_norm": 0.7029790878295898 + }, + { + "bits_per_byte": 1.6146604934866857, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.1191973686218262, + "step": 5060, + "unclipped_grad_norm": 0.7553480267524719 + }, + { + "bits_per_byte": 1.5606498579311734, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.081760048866272, + "step": 5070, + "unclipped_grad_norm": 0.7393523454666138 + }, + { + "bits_per_byte": 1.6278340205774753, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.128328561782837, + "step": 5080, + "unclipped_grad_norm": 0.7396665811538696 + }, + { + "bits_per_byte": 1.5506494107488027, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.0748282670974731, + "step": 5090, + "unclipped_grad_norm": 0.6867292523384094 + }, + { + "bits_per_byte": 1.5403780908913232, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.0677087306976318, + "step": 5100, + "unclipped_grad_norm": 0.7300744652748108 + }, + { + "bits_per_byte": 1.583487090173944, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.0975896120071411, + "step": 5110, + "unclipped_grad_norm": 0.6969560384750366 + }, + { + "bits_per_byte": 1.4966463424299283, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0373961925506592, + "step": 5120, + "unclipped_grad_norm": 0.7119253873825073 + }, + { + "bits_per_byte": 1.621785734711932, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.124136209487915, + "step": 5130, + "unclipped_grad_norm": 0.7355929613113403 + }, + { + "bits_per_byte": 1.5875032290368922, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.100373387336731, + "step": 5140, + "unclipped_grad_norm": 0.7337709665298462 + }, + { + "bits_per_byte": 1.5282780795072195, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.059321641921997, + "step": 5150, + "unclipped_grad_norm": 0.7043288946151733 + }, + { + "bits_per_byte": 1.554402588138582, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.0774297714233398, + "step": 5160, + "unclipped_grad_norm": 0.7431972026824951 + }, + { + "bits_per_byte": 1.6102955738077878, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.1161718368530273, + "step": 5170, + "unclipped_grad_norm": 0.7488743662834167 + }, + { + "bits_per_byte": 1.569338937400715, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.087782859802246, + "step": 5180, + "unclipped_grad_norm": 0.7440537810325623 + }, + { + "bits_per_byte": 1.461319042150897, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.012909173965454, + "step": 5190, + "unclipped_grad_norm": 0.7041038870811462 + }, + { + "bits_per_byte": 1.5457332866738893, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.071420669555664, + "step": 5200, + "unclipped_grad_norm": 0.7203606367111206 + }, + { + "bits_per_byte": 1.5101941037689952, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.0467867851257324, + "step": 5210, + "unclipped_grad_norm": 0.7148077487945557 + }, + { + "bits_per_byte": 1.5310258463200546, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.06122624874115, + "step": 5220, + "unclipped_grad_norm": 0.7114641666412354 + }, + { + "bits_per_byte": 1.5979776604050409, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.1076337099075317, + "step": 5230, + "unclipped_grad_norm": 0.7518506050109863 + }, + { + "bits_per_byte": 1.5711922224464125, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0890674591064453, + "step": 5240, + "unclipped_grad_norm": 0.7655813694000244 + }, + { + "bits_per_byte": 1.5651432486502659, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0848746299743652, + "step": 5250, + "unclipped_grad_norm": 0.721767246723175 + }, + { + "bits_per_byte": 1.5066187564402083, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.0443085432052612, + "step": 5260, + "unclipped_grad_norm": 0.7343600392341614 + }, + { + "bits_per_byte": 1.5590867076174812, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.080676555633545, + "step": 5270, + "unclipped_grad_norm": 0.7202844619750977 + }, + { + "bits_per_byte": 1.4878253522671747, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.0312819480895996, + "step": 5280, + "unclipped_grad_norm": 0.7298920750617981 + }, + { + "bits_per_byte": 1.512711585812325, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.0485317707061768, + "step": 5290, + "unclipped_grad_norm": 0.7234549522399902 + }, + { + "bits_per_byte": 1.4987307721583856, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.0388410091400146, + "step": 5300, + "unclipped_grad_norm": 0.7503591179847717 + }, + { + "bits_per_byte": 1.4903792946324874, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.0330522060394287, + "step": 5310, + "unclipped_grad_norm": 0.7066974639892578 + }, + { + "bits_per_byte": 1.5173601048824958, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0517538785934448, + "step": 5320, + "unclipped_grad_norm": 0.7187825441360474 + }, + { + "bits_per_byte": 1.554223038251081, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.0773053169250488, + "step": 5330, + "unclipped_grad_norm": 0.7340478301048279 + }, + { + "bits_per_byte": 1.6030403136984892, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.111142873764038, + "step": 5340, + "unclipped_grad_norm": 0.7655161023139954 + }, + { + "bits_per_byte": 1.4826178895816997, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.0276724100112915, + "step": 5350, + "unclipped_grad_norm": 0.7006027102470398 + }, + { + "bits_per_byte": 1.506020944745809, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.0438941717147827, + "step": 5360, + "unclipped_grad_norm": 0.727604866027832 + }, + { + "bits_per_byte": 1.5844082292519652, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.098228096961975, + "step": 5370, + "unclipped_grad_norm": 0.7804976105690002 + }, + { + "bits_per_byte": 1.509563615370932, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.0463497638702393, + "step": 5380, + "unclipped_grad_norm": 0.6970203518867493 + }, + { + "bits_per_byte": 1.5013407808678794, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.0406501293182373, + "step": 5390, + "unclipped_grad_norm": 0.7277929782867432 + }, + { + "bits_per_byte": 1.5563463360586334, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.0787770748138428, + "step": 5400, + "unclipped_grad_norm": 0.7462701201438904 + }, + { + "bits_per_byte": 1.4455724826205492, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.0019944906234741, + "step": 5410, + "unclipped_grad_norm": 0.7340894341468811 + }, + { + "bits_per_byte": 1.608882564348299, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.1151924133300781, + "step": 5420, + "unclipped_grad_norm": 0.8010579943656921 + }, + { + "bits_per_byte": 1.539171288630219, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.066872239112854, + "step": 5430, + "unclipped_grad_norm": 0.7386476397514343 + }, + { + "bits_per_byte": 1.5408231819917562, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.0680172443389893, + "step": 5440, + "unclipped_grad_norm": 0.7174329161643982 + }, + { + "bits_per_byte": 1.545910428804278, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.0715434551239014, + "step": 5450, + "unclipped_grad_norm": 0.7474415302276611 + }, + { + "bits_per_byte": 1.5681410782374556, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.086952567100525, + "step": 5460, + "unclipped_grad_norm": 0.7237385511398315 + }, + { + "bits_per_byte": 1.5412906308368013, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0683412551879883, + "step": 5470, + "unclipped_grad_norm": 0.7598943710327148 + }, + { + "bits_per_byte": 1.4869028373279467, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.0306425094604492, + "step": 5480, + "unclipped_grad_norm": 0.7240223288536072 + }, + { + "bits_per_byte": 1.538147991857585, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.0661629438400269, + "step": 5490, + "unclipped_grad_norm": 0.7488465905189514 + }, + { + "bits_per_byte": 1.5652971731227883, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.0849813222885132, + "step": 5500, + "unclipped_grad_norm": 0.7459914684295654 + }, + { + "bits_per_byte": 1.5514190331114137, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.075361728668213, + "step": 5510, + "unclipped_grad_norm": 0.743117094039917 + }, + { + "bits_per_byte": 1.5383910033432544, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.066331386566162, + "step": 5520, + "unclipped_grad_norm": 0.7540643811225891 + }, + { + "bits_per_byte": 1.5552494307114304, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.078016757965088, + "step": 5530, + "unclipped_grad_norm": 0.7664636969566345 + }, + { + "bits_per_byte": 1.5867925967235272, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.0998808145523071, + "step": 5540, + "unclipped_grad_norm": 0.7745412588119507 + }, + { + "bits_per_byte": 1.5213924101146252, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.0545488595962524, + "step": 5550, + "unclipped_grad_norm": 0.7512962818145752 + }, + { + "bits_per_byte": 1.5096733403021825, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.0464258193969727, + "step": 5560, + "unclipped_grad_norm": 0.7278321981430054 + }, + { + "bits_per_byte": 1.5697073242388633, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.0880382061004639, + "step": 5570, + "unclipped_grad_norm": 0.7823853492736816 + }, + { + "bits_per_byte": 1.4552483985407416, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.0087013244628906, + "step": 5580, + "unclipped_grad_norm": 0.7032573223114014 + }, + { + "bits_per_byte": 1.5215007591846688, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.0546239614486694, + "step": 5590, + "unclipped_grad_norm": 0.7657963037490845 + }, + { + "bits_per_byte": 1.512369340337108, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.0482945442199707, + "step": 5600, + "unclipped_grad_norm": 0.7165441513061523 + }, + { + "bits_per_byte": 1.5384162847929312, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.066348910331726, + "step": 5610, + "unclipped_grad_norm": 0.7338334321975708 + }, + { + "bits_per_byte": 1.475370712656992, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.0226490497589111, + "step": 5620, + "unclipped_grad_norm": 0.741429328918457 + }, + { + "bits_per_byte": 1.4807990010661747, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0264116525650024, + "step": 5630, + "unclipped_grad_norm": 0.7429983615875244 + }, + { + "bits_per_byte": 1.4642707803876553, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.0149551630020142, + "step": 5640, + "unclipped_grad_norm": 0.7470905184745789 + }, + { + "bits_per_byte": 1.5717626888993246, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.0894628763198853, + "step": 5650, + "unclipped_grad_norm": 0.7854259014129639 + }, + { + "bits_per_byte": 1.5285147276348068, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.059485673904419, + "step": 5660, + "unclipped_grad_norm": 0.7478188872337341 + }, + { + "bits_per_byte": 1.5381870319193311, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.0661900043487549, + "step": 5670, + "unclipped_grad_norm": 0.7506261467933655 + }, + { + "bits_per_byte": 1.529865479374683, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0604219436645508, + "step": 5680, + "unclipped_grad_norm": 0.7439587116241455 + }, + { + "bits_per_byte": 1.5592215420157578, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.0807700157165527, + "step": 5690, + "unclipped_grad_norm": 0.7450974583625793 + }, + { + "bits_per_byte": 1.5062584527866507, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.0440587997436523, + "step": 5700, + "unclipped_grad_norm": 0.7572526931762695 + }, + { + "bits_per_byte": 1.542002982976675, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.0688350200653076, + "step": 5710, + "unclipped_grad_norm": 0.7398989796638489 + }, + { + "bits_per_byte": 1.5712675508474905, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0891196727752686, + "step": 5720, + "unclipped_grad_norm": 0.8168424367904663 + }, + { + "bits_per_byte": 1.468587544924312, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.0179473161697388, + "step": 5730, + "unclipped_grad_norm": 0.7199760675430298 + }, + { + "bits_per_byte": 1.5355907818219061, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.0643904209136963, + "step": 5740, + "unclipped_grad_norm": 0.7808306813240051 + }, + { + "bits_per_byte": 1.4974770186335957, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.0379719734191895, + "step": 5750, + "unclipped_grad_norm": 0.7880769968032837 + }, + { + "bits_per_byte": 1.499835588707528, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.0396068096160889, + "step": 5760, + "unclipped_grad_norm": 0.7763125896453857 + }, + { + "bits_per_byte": 1.4576090324064845, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.0103375911712646, + "step": 5770, + "unclipped_grad_norm": 0.7382789850234985 + }, + { + "bits_per_byte": 1.538267347817284, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.066245675086975, + "step": 5780, + "unclipped_grad_norm": 0.7327389121055603 + }, + { + "bits_per_byte": 1.491346353078289, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 1.0337225198745728, + "step": 5790, + "unclipped_grad_norm": 0.7199346423149109 + }, + { + "bits_per_byte": 1.4870258049233136, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.030727744102478, + "step": 5800, + "unclipped_grad_norm": 0.7510647177696228 + }, + { + "bits_per_byte": 1.5330280683413995, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.0626140832901, + "step": 5810, + "unclipped_grad_norm": 0.773936927318573 + }, + { + "bits_per_byte": 1.5095316265978715, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.0463275909423828, + "step": 5820, + "unclipped_grad_norm": 0.7972245216369629 + }, + { + "bits_per_byte": 1.4845955180839712, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.029043197631836, + "step": 5830, + "unclipped_grad_norm": 0.7750056982040405 + }, + { + "bits_per_byte": 1.4626711697519803, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.0138463973999023, + "step": 5840, + "unclipped_grad_norm": 0.7466725707054138 + }, + { + "bits_per_byte": 1.474649761384575, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.0221493244171143, + "step": 5850, + "unclipped_grad_norm": 0.7539114952087402 + }, + { + "bits_per_byte": 1.488105683988081, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.0314762592315674, + "step": 5860, + "unclipped_grad_norm": 0.7800592184066772 + }, + { + "bits_per_byte": 1.5465969835465223, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.072019338607788, + "step": 5870, + "unclipped_grad_norm": 0.7841717004776001 + }, + { + "bits_per_byte": 1.543482205756746, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.0698603391647339, + "step": 5880, + "unclipped_grad_norm": 0.7701864242553711 + }, + { + "bits_per_byte": 1.5220736333946927, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.055021047592163, + "step": 5890, + "unclipped_grad_norm": 0.7658758163452148 + }, + { + "bits_per_byte": 1.5033466145248924, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.042040467262268, + "step": 5900, + "unclipped_grad_norm": 0.7883568406105042 + }, + { + "bits_per_byte": 1.4877102958737476, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.0312021970748901, + "step": 5910, + "unclipped_grad_norm": 0.7940378785133362 + }, + { + "bits_per_byte": 1.510579000941626, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.047053575515747, + "step": 5920, + "unclipped_grad_norm": 0.7805882096290588 + }, + { + "bits_per_byte": 1.554354948944293, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.0773967504501343, + "step": 5930, + "unclipped_grad_norm": 0.7843778729438782 + }, + { + "bits_per_byte": 1.5316138550033545, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.061633825302124, + "step": 5940, + "unclipped_grad_norm": 0.8224195241928101 + }, + { + "bits_per_byte": 1.5128973270752568, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.0486605167388916, + "step": 5950, + "unclipped_grad_norm": 0.8192062377929688 + }, + { + "bits_per_byte": 1.5044584823627205, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.0428111553192139, + "step": 5960, + "unclipped_grad_norm": 0.7744107246398926 + }, + { + "bits_per_byte": 1.5207576221502905, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.0541088581085205, + "step": 5970, + "unclipped_grad_norm": 0.8335334658622742 + }, + { + "bits_per_byte": 1.4766832842483768, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.0235588550567627, + "step": 5980, + "unclipped_grad_norm": 0.787668764591217 + }, + { + "bits_per_byte": 1.445810850574645, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0021597146987915, + "step": 5990, + "unclipped_grad_norm": 0.7685462832450867 + }, + { + "bits_per_byte": 1.516961105132494, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.051477313041687, + "step": 6000, + "unclipped_grad_norm": 0.7635723352432251 + }, + { + "bits_per_byte": 1.5010169375363047, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0404256582260132, + "step": 6010, + "unclipped_grad_norm": 0.7984814047813416 + }, + { + "bits_per_byte": 1.4756424452453551, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0228374004364014, + "step": 6020, + "unclipped_grad_norm": 0.7804116606712341 + }, + { + "bits_per_byte": 1.5478184043329501, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.0728659629821777, + "step": 6030, + "unclipped_grad_norm": 0.7833549380302429 + }, + { + "bits_per_byte": 1.4943269844003924, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.0357885360717773, + "step": 6040, + "unclipped_grad_norm": 0.8053407669067383 + }, + { + "bits_per_byte": 1.5075827191982942, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.0449767112731934, + "step": 6050, + "unclipped_grad_norm": 0.8034298419952393 + }, + { + "bits_per_byte": 1.5370013835242828, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.0653681755065918, + "step": 6060, + "unclipped_grad_norm": 0.7789096832275391 + }, + { + "bits_per_byte": 1.5026303067840487, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 1.041543960571289, + "step": 6070, + "unclipped_grad_norm": 0.7537946701049805 + }, + { + "bits_per_byte": 1.4655401843336737, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.0158350467681885, + "step": 6080, + "unclipped_grad_norm": 0.7796688079833984 + }, + { + "bits_per_byte": 1.494266102541987, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.0357463359832764, + "step": 6090, + "unclipped_grad_norm": 0.7650325298309326 + }, + { + "bits_per_byte": 1.4853904218962592, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.0295941829681396, + "step": 6100, + "unclipped_grad_norm": 0.7634284496307373 + }, + { + "bits_per_byte": 1.4513882479421258, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.0060256719589233, + "step": 6110, + "unclipped_grad_norm": 0.761825680732727 + }, + { + "bits_per_byte": 1.463188321583125, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0142048597335815, + "step": 6120, + "unclipped_grad_norm": 0.7645092010498047 + }, + { + "bits_per_byte": 1.498213792309892, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.038482666015625, + "step": 6130, + "unclipped_grad_norm": 0.8006239533424377 + }, + { + "bits_per_byte": 1.523158843921637, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.0557732582092285, + "step": 6140, + "unclipped_grad_norm": 0.7845824360847473 + }, + { + "bits_per_byte": 1.4636738286065105, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.0145413875579834, + "step": 6150, + "unclipped_grad_norm": 0.7672165632247925 + }, + { + "bits_per_byte": 1.5490222828889897, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 1.0737004280090332, + "step": 6160, + "unclipped_grad_norm": 0.8044849038124084 + }, + { + "bits_per_byte": 1.4192805488785718, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 0.9837703108787537, + "step": 6170, + "unclipped_grad_norm": 0.7445047497749329 + }, + { + "bits_per_byte": 1.5037653922797434, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0423307418823242, + "step": 6180, + "unclipped_grad_norm": 0.7923136353492737 + }, + { + "bits_per_byte": 1.5020665476545205, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.0411531925201416, + "step": 6190, + "unclipped_grad_norm": 0.8200557231903076 + }, + { + "bits_per_byte": 1.4516574007907261, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.0062122344970703, + "step": 6200, + "unclipped_grad_norm": 0.7482684254646301 + }, + { + "bits_per_byte": 1.4787684019074376, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.0250041484832764, + "step": 6210, + "unclipped_grad_norm": 0.781791090965271 + }, + { + "bits_per_byte": 1.4619254529778392, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0133295059204102, + "step": 6220, + "unclipped_grad_norm": 0.7642713785171509 + }, + { + "bits_per_byte": 1.444159645143711, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.0010151863098145, + "step": 6230, + "unclipped_grad_norm": 0.7728588581085205 + }, + { + "bits_per_byte": 1.4545422377762989, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.0082118511199951, + "step": 6240, + "unclipped_grad_norm": 0.8221566677093506 + }, + { + "bits_per_byte": 1.4619588176121066, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.013352632522583, + "step": 6250, + "unclipped_grad_norm": 0.7835553884506226 + }, + { + "bits_per_byte": 1.5199867959091236, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 1.053574562072754, + "step": 6260, + "unclipped_grad_norm": 0.8351667523384094 + }, + { + "bits_per_byte": 1.478633739491812, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.024910807609558, + "step": 6270, + "unclipped_grad_norm": 0.8318396210670471 + }, + { + "bits_per_byte": 1.589328824875801, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.1016387939453125, + "step": 6280, + "unclipped_grad_norm": 0.8732567429542542 + }, + { + "bits_per_byte": 1.4688828391358433, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 1.0181519985198975, + "step": 6290, + "unclipped_grad_norm": 0.8071078658103943 + }, + { + "bits_per_byte": 1.490956468408783, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.0334522724151611, + "step": 6300, + "unclipped_grad_norm": 0.8064041137695312 + }, + { + "bits_per_byte": 1.5287245464688592, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.059631109237671, + "step": 6310, + "unclipped_grad_norm": 0.820055365562439 + }, + { + "bits_per_byte": 1.4907531849154632, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 1.033311367034912, + "step": 6320, + "unclipped_grad_norm": 0.7774296402931213 + }, + { + "bits_per_byte": 1.4349515220338656, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 0.9946326017379761, + "step": 6330, + "unclipped_grad_norm": 0.7782560586929321 + }, + { + "bits_per_byte": 1.4942542357390773, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 1.0357381105422974, + "step": 6340, + "unclipped_grad_norm": 0.8143325448036194 + }, + { + "bits_per_byte": 1.4851666724674866, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.029439091682434, + "step": 6350, + "unclipped_grad_norm": 0.7848084568977356 + }, + { + "bits_per_byte": 1.4448221223148348, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.001474380493164, + "step": 6360, + "unclipped_grad_norm": 0.757684588432312 + }, + { + "bits_per_byte": 1.4930216360803437, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 1.034883737564087, + "step": 6370, + "unclipped_grad_norm": 0.8128012418746948 + }, + { + "bits_per_byte": 1.4604040944483068, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.012274980545044, + "step": 6380, + "unclipped_grad_norm": 0.7984938025474548 + }, + { + "bits_per_byte": 1.498449408541574, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0386459827423096, + "step": 6390, + "unclipped_grad_norm": 0.8357493877410889 + }, + { + "bits_per_byte": 1.4830395910416154, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.02796471118927, + "step": 6400, + "unclipped_grad_norm": 0.8292281031608582 + }, + { + "bits_per_byte": 1.458597588683644, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.0110228061676025, + "step": 6410, + "unclipped_grad_norm": 0.8157880902290344 + }, + { + "bits_per_byte": 1.444321480818173, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0011273622512817, + "step": 6420, + "unclipped_grad_norm": 0.7859262228012085 + }, + { + "bits_per_byte": 1.4700296194517966, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.018946886062622, + "step": 6430, + "unclipped_grad_norm": 0.8117997050285339 + }, + { + "bits_per_byte": 1.4824080707476472, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.0275269746780396, + "step": 6440, + "unclipped_grad_norm": 0.8291072249412537 + }, + { + "bits_per_byte": 1.457504294972109, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0102649927139282, + "step": 6450, + "unclipped_grad_norm": 0.8316922187805176 + }, + { + "bits_per_byte": 1.4455943244172087, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.002009630203247, + "step": 6460, + "unclipped_grad_norm": 0.9506665468215942 + }, + { + "bits_per_byte": 1.482229036808099, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.0274028778076172, + "step": 6470, + "unclipped_grad_norm": 0.8135435581207275 + }, + { + "bits_per_byte": 1.494786522043498, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 1.036107063293457, + "step": 6480, + "unclipped_grad_norm": 0.8327356576919556 + }, + { + "bits_per_byte": 1.480783694610248, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0264010429382324, + "step": 6490, + "unclipped_grad_norm": 0.8300126194953918 + }, + { + "bits_per_byte": 1.4794718109494664, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.025491714477539, + "step": 6500, + "unclipped_grad_norm": 0.7848952412605286 + }, + { + "bits_per_byte": 1.4289866477539908, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 0.9904980659484863, + "step": 6510, + "unclipped_grad_norm": 0.7933451533317566 + }, + { + "bits_per_byte": 1.3967986327924837, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 0.9681870341300964, + "step": 6520, + "unclipped_grad_norm": 0.7868184447288513 + }, + { + "bits_per_byte": 1.4999294912348993, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.0396718978881836, + "step": 6530, + "unclipped_grad_norm": 0.8443325757980347 + }, + { + "bits_per_byte": 1.4565786843451654, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.009623408317566, + "step": 6540, + "unclipped_grad_norm": 0.7749677896499634 + }, + { + "bits_per_byte": 1.499482336342656, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 1.0393619537353516, + "step": 6550, + "unclipped_grad_norm": 0.8301845788955688 + }, + { + "bits_per_byte": 1.4687036332136443, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.0180277824401855, + "step": 6560, + "unclipped_grad_norm": 0.8123624920845032 + }, + { + "bits_per_byte": 1.4061458038755519, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 0.9746659994125366, + "step": 6570, + "unclipped_grad_norm": 0.8421014547348022 + }, + { + "bits_per_byte": 1.4795481712464496, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.0255446434020996, + "step": 6580, + "unclipped_grad_norm": 0.7824509143829346 + }, + { + "bits_per_byte": 1.4564835779392382, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.0095574855804443, + "step": 6590, + "unclipped_grad_norm": 0.8128623366355896 + }, + { + "bits_per_byte": 1.4012816185611991, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9712944030761719, + "step": 6600, + "unclipped_grad_norm": 0.801582396030426 + }, + { + "bits_per_byte": 1.504982857465201, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.0431746244430542, + "step": 6610, + "unclipped_grad_norm": 0.7954027652740479 + }, + { + "bits_per_byte": 1.5121832551088743, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 1.0481655597686768, + "step": 6620, + "unclipped_grad_norm": 0.8685457110404968 + }, + { + "bits_per_byte": 1.4914144582080306, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.0337697267532349, + "step": 6630, + "unclipped_grad_norm": 0.8549599051475525 + }, + { + "bits_per_byte": 1.467648519650601, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.0172964334487915, + "step": 6640, + "unclipped_grad_norm": 0.8680722713470459 + }, + { + "bits_per_byte": 1.445384505583156, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.0018641948699951, + "step": 6650, + "unclipped_grad_norm": 0.7804962396621704 + }, + { + "bits_per_byte": 1.5044949426847034, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.0428364276885986, + "step": 6660, + "unclipped_grad_norm": 0.8356144428253174 + }, + { + "bits_per_byte": 1.4747041079022476, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.0221869945526123, + "step": 6670, + "unclipped_grad_norm": 0.8340083360671997 + }, + { + "bits_per_byte": 1.4979609778131235, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.0383074283599854, + "step": 6680, + "unclipped_grad_norm": 0.8310403823852539 + }, + { + "bits_per_byte": 1.47108679680665, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.0196796655654907, + "step": 6690, + "unclipped_grad_norm": 0.8335619568824768 + }, + { + "bits_per_byte": 1.5152930454017757, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.050321102142334, + "step": 6700, + "unclipped_grad_norm": 0.8671045899391174 + }, + { + "bits_per_byte": 1.4392078346687438, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 0.9975828528404236, + "step": 6710, + "unclipped_grad_norm": 0.8319199085235596 + }, + { + "bits_per_byte": 1.4416132700150364, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 0.9992501735687256, + "step": 6720, + "unclipped_grad_norm": 0.8375449776649475 + }, + { + "bits_per_byte": 1.4240010426881287, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 0.9870423078536987, + "step": 6730, + "unclipped_grad_norm": 0.8281634449958801 + }, + { + "bits_per_byte": 1.4551885485782414, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 1.008659839630127, + "step": 6740, + "unclipped_grad_norm": 0.8503311276435852 + }, + { + "bits_per_byte": 1.4162871048489791, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 0.9816954135894775, + "step": 6750, + "unclipped_grad_norm": 0.8190869092941284 + }, + { + "bits_per_byte": 1.3980038012184048, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 0.9690223932266235, + "step": 6760, + "unclipped_grad_norm": 0.8038136959075928 + }, + { + "bits_per_byte": 1.4312844219608432, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 0.9920907616615295, + "step": 6770, + "unclipped_grad_norm": 0.8015828132629395 + }, + { + "bits_per_byte": 1.4362121548646898, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 0.9955064058303833, + "step": 6780, + "unclipped_grad_norm": 0.8221172094345093 + }, + { + "bits_per_byte": 1.5240550455152833, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.0563944578170776, + "step": 6790, + "unclipped_grad_norm": 0.839040994644165 + }, + { + "bits_per_byte": 1.5110246079900116, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0473624467849731, + "step": 6800, + "unclipped_grad_norm": 0.8993076086044312 + }, + { + "bits_per_byte": 1.481978286103141, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0272290706634521, + "step": 6810, + "unclipped_grad_norm": 0.8410735726356506 + }, + { + "bits_per_byte": 1.436286107404561, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 0.9955576658248901, + "step": 6820, + "unclipped_grad_norm": 0.8490327596664429 + }, + { + "bits_per_byte": 1.4430656635015728, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.000256896018982, + "step": 6830, + "unclipped_grad_norm": 0.8410844206809998 + }, + { + "bits_per_byte": 1.4757095184791917, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 1.0228838920593262, + "step": 6840, + "unclipped_grad_norm": 0.8297956585884094 + }, + { + "bits_per_byte": 1.4475170904538548, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.0033423900604248, + "step": 6850, + "unclipped_grad_norm": 0.8365767598152161 + }, + { + "bits_per_byte": 1.445049827344577, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.0016322135925293, + "step": 6860, + "unclipped_grad_norm": 0.8103650212287903 + }, + { + "bits_per_byte": 1.4721945370608576, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 1.0204474925994873, + "step": 6870, + "unclipped_grad_norm": 0.8531231880187988 + }, + { + "bits_per_byte": 1.4995490656111907, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 1.0394082069396973, + "step": 6880, + "unclipped_grad_norm": 0.8313021063804626 + }, + { + "bits_per_byte": 1.4047026114608367, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 0.9736656546592712, + "step": 6890, + "unclipped_grad_norm": 0.842310905456543 + }, + { + "bits_per_byte": 1.424149807681125, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 0.9871454238891602, + "step": 6900, + "unclipped_grad_norm": 0.8237481713294983 + }, + { + "bits_per_byte": 1.5991122299527827, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.1084201335906982, + "step": 6910, + "unclipped_grad_norm": 0.9396551847457886 + }, + { + "bits_per_byte": 1.4095282726613962, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 0.9770105481147766, + "step": 6920, + "unclipped_grad_norm": 0.8222891092300415 + }, + { + "bits_per_byte": 1.4166267705844333, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.9819308519363403, + "step": 6930, + "unclipped_grad_norm": 0.8385740518569946 + }, + { + "bits_per_byte": 1.450679851403222, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 1.0055346488952637, + "step": 6940, + "unclipped_grad_norm": 0.845413088798523 + }, + { + "bits_per_byte": 1.4355700576376935, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 0.9950613379478455, + "step": 6950, + "unclipped_grad_norm": 0.8435958623886108 + }, + { + "bits_per_byte": 1.485354305539578, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 1.029569149017334, + "step": 6960, + "unclipped_grad_norm": 0.8694136142730713 + }, + { + "bits_per_byte": 1.4633228120160995, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 1.0142980813980103, + "step": 6970, + "unclipped_grad_norm": 0.8648980855941772 + }, + { + "bits_per_byte": 1.5053073887273793, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.0433995723724365, + "step": 6980, + "unclipped_grad_norm": 0.9064987301826477 + }, + { + "bits_per_byte": 1.4555743056641266, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.0089272260665894, + "step": 6990, + "unclipped_grad_norm": 0.8786826729774475 + }, + { + "bits_per_byte": 1.410162286703802, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 0.9774500131607056, + "step": 7000, + "unclipped_grad_norm": 0.8255307078361511 + }, + { + "bits_per_byte": 1.4432978400802376, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 1.0004178285598755, + "step": 7010, + "unclipped_grad_norm": 0.873204231262207 + }, + { + "bits_per_byte": 1.422183014085858, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 0.9857821464538574, + "step": 7020, + "unclipped_grad_norm": 0.8475666046142578 + }, + { + "bits_per_byte": 1.4226718747709355, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 0.9861209988594055, + "step": 7030, + "unclipped_grad_norm": 0.8115434050559998 + }, + { + "bits_per_byte": 1.5043391264030217, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.0427284240722656, + "step": 7040, + "unclipped_grad_norm": 0.8999432325363159 + }, + { + "bits_per_byte": 1.4615047834138288, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.013037919998169, + "step": 7050, + "unclipped_grad_norm": 0.8429679870605469 + }, + { + "bits_per_byte": 1.4751295929804822, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 1.022481918334961, + "step": 7060, + "unclipped_grad_norm": 0.9018688797950745 + }, + { + "bits_per_byte": 1.4248705009795657, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 0.9876449704170227, + "step": 7070, + "unclipped_grad_norm": 0.8220717906951904 + }, + { + "bits_per_byte": 1.357736815241292, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 0.9411114454269409, + "step": 7080, + "unclipped_grad_norm": 0.8423992395401001 + }, + { + "bits_per_byte": 1.420799499650992, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 0.9848231673240662, + "step": 7090, + "unclipped_grad_norm": 0.8346288800239563 + }, + { + "bits_per_byte": 1.411367455129723, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 0.9782853722572327, + "step": 7100, + "unclipped_grad_norm": 0.8508378267288208 + }, + { + "bits_per_byte": 1.4701558547175297, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 1.0190343856811523, + "step": 7110, + "unclipped_grad_norm": 0.8868972659111023 + }, + { + "bits_per_byte": 1.4713418470778798, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.0198564529418945, + "step": 7120, + "unclipped_grad_norm": 0.8707273006439209 + }, + { + "bits_per_byte": 1.406841645680943, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 0.9751483201980591, + "step": 7130, + "unclipped_grad_norm": 0.833061695098877 + }, + { + "bits_per_byte": 1.513175251039051, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.0488531589508057, + "step": 7140, + "unclipped_grad_norm": 0.9011214971542358 + }, + { + "bits_per_byte": 1.5265056263074273, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 1.0580930709838867, + "step": 7150, + "unclipped_grad_norm": 0.858849823474884 + }, + { + "bits_per_byte": 1.4644068186644879, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0150494575500488, + "step": 7160, + "unclipped_grad_norm": 0.922670304775238 + }, + { + "bits_per_byte": 1.4511968312517156, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.0058929920196533, + "step": 7170, + "unclipped_grad_norm": 0.8228445053100586 + }, + { + "bits_per_byte": 1.371989189500944, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 0.9509904384613037, + "step": 7180, + "unclipped_grad_norm": 0.7978084683418274 + }, + { + "bits_per_byte": 1.3817090469884317, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 0.9577277302742004, + "step": 7190, + "unclipped_grad_norm": 0.8511900901794434 + }, + { + "bits_per_byte": 1.4680136388183826, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 1.0175495147705078, + "step": 7200, + "unclipped_grad_norm": 0.8525802493095398 + }, + { + "bits_per_byte": 1.4718189269513733, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 1.0201871395111084, + "step": 7210, + "unclipped_grad_norm": 0.8266363739967346 + }, + { + "bits_per_byte": 1.420904495059344, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 0.9848959445953369, + "step": 7220, + "unclipped_grad_norm": 0.8417283892631531 + }, + { + "bits_per_byte": 1.4350502400754608, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 0.9947010278701782, + "step": 7230, + "unclipped_grad_norm": 0.8409048914909363 + }, + { + "bits_per_byte": 1.3901885655978972, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 0.9636052846908569, + "step": 7240, + "unclipped_grad_norm": 0.8238283395767212 + }, + { + "bits_per_byte": 1.4356194166584912, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 0.9950955510139465, + "step": 7250, + "unclipped_grad_norm": 0.8763477206230164 + }, + { + "bits_per_byte": 1.4241162710642068, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 0.9871221780776978, + "step": 7260, + "unclipped_grad_norm": 0.8532221913337708 + }, + { + "bits_per_byte": 1.4981054432398484, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 1.038407564163208, + "step": 7270, + "unclipped_grad_norm": 0.9358603358268738 + }, + { + "bits_per_byte": 1.438327627461628, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 0.9969727396965027, + "step": 7280, + "unclipped_grad_norm": 0.849073588848114 + }, + { + "bits_per_byte": 1.4392007833800584, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 0.997577965259552, + "step": 7290, + "unclipped_grad_norm": 0.8500707745552063 + }, + { + "bits_per_byte": 1.4248026538238003, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 0.9875979423522949, + "step": 7300, + "unclipped_grad_norm": 0.8606101274490356 + }, + { + "bits_per_byte": 1.4801721243037798, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 1.0259771347045898, + "step": 7310, + "unclipped_grad_norm": 0.8684185147285461 + }, + { + "bits_per_byte": 1.4657266135272091, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 1.0159642696380615, + "step": 7320, + "unclipped_grad_norm": 0.8756072521209717 + }, + { + "bits_per_byte": 1.4556759474107865, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 1.0089976787567139, + "step": 7330, + "unclipped_grad_norm": 0.960211992263794 + }, + { + "bits_per_byte": 1.4334342910879538, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.9935809373855591, + "step": 7340, + "unclipped_grad_norm": 0.8651729822158813 + }, + { + "bits_per_byte": 1.4765180089208978, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 1.0234442949295044, + "step": 7350, + "unclipped_grad_norm": 0.8694680333137512 + }, + { + "bits_per_byte": 1.4569269492131627, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 1.0098648071289062, + "step": 7360, + "unclipped_grad_norm": 0.8548897504806519 + }, + { + "bits_per_byte": 1.4681032417794821, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 1.0176116228103638, + "step": 7370, + "unclipped_grad_norm": 0.8606715798377991 + }, + { + "bits_per_byte": 1.5110347549664125, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 1.0473694801330566, + "step": 7380, + "unclipped_grad_norm": 0.9176380634307861 + }, + { + "bits_per_byte": 1.4941954176724823, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 1.035697340965271, + "step": 7390, + "unclipped_grad_norm": 0.9042743444442749 + }, + { + "bits_per_byte": 1.4149094378242426, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.980740487575531, + "step": 7400, + "unclipped_grad_norm": 0.8498074412345886 + }, + { + "bits_per_byte": 1.4308479299929535, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 0.9917882084846497, + "step": 7410, + "unclipped_grad_norm": 0.8460224866867065 + }, + { + "bits_per_byte": 1.5114327228205091, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0476453304290771, + "step": 7420, + "unclipped_grad_norm": 0.9031701683998108 + }, + { + "bits_per_byte": 1.4157659114255394, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 0.9813341498374939, + "step": 7430, + "unclipped_grad_norm": 0.9146517515182495 + }, + { + "bits_per_byte": 1.4371513521210517, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 0.9961574077606201, + "step": 7440, + "unclipped_grad_norm": 0.8787059187889099 + }, + { + "bits_per_byte": 1.3687977074488826, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 0.9487782716751099, + "step": 7450, + "unclipped_grad_norm": 0.8917891383171082 + }, + { + "bits_per_byte": 1.414513017814004, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 0.9804657101631165, + "step": 7460, + "unclipped_grad_norm": 0.8450844287872314 + }, + { + "bits_per_byte": 1.4664716423707467, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 1.0164806842803955, + "step": 7470, + "unclipped_grad_norm": 0.8780871629714966 + }, + { + "bits_per_byte": 1.382436447610256, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 0.9582319259643555, + "step": 7480, + "unclipped_grad_norm": 0.8598330020904541 + }, + { + "bits_per_byte": 1.4063960386325574, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 0.974839448928833, + "step": 7490, + "unclipped_grad_norm": 0.8495810031890869 + }, + { + "bits_per_byte": 1.4715931137307903, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.0200306177139282, + "step": 7500, + "unclipped_grad_norm": 0.8626576662063599 + }, + { + "bits_per_byte": 1.4680590422382105, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 1.0175809860229492, + "step": 7510, + "unclipped_grad_norm": 0.9082968831062317 + }, + { + "bits_per_byte": 1.4349203931740593, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 0.9946110248565674, + "step": 7520, + "unclipped_grad_norm": 0.8796036243438721 + }, + { + "bits_per_byte": 1.4537043383012953, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.0076310634613037, + "step": 7530, + "unclipped_grad_norm": 0.8794750571250916 + }, + { + "bits_per_byte": 1.4869279467949728, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.0306599140167236, + "step": 7540, + "unclipped_grad_norm": 0.9268702864646912 + }, + { + "bits_per_byte": 1.427805470907865, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 0.9896793365478516, + "step": 7550, + "unclipped_grad_norm": 0.8863946795463562 + }, + { + "bits_per_byte": 1.468888342580671, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 1.018155813217163, + "step": 7560, + "unclipped_grad_norm": 0.8562321066856384 + }, + { + "bits_per_byte": 1.479489697145156, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 1.0255041122436523, + "step": 7570, + "unclipped_grad_norm": 0.9035179018974304 + }, + { + "bits_per_byte": 1.4093398656673761, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 0.9768799543380737, + "step": 7580, + "unclipped_grad_norm": 0.8376500606536865 + }, + { + "bits_per_byte": 1.4273161802661603, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 0.9893401861190796, + "step": 7590, + "unclipped_grad_norm": 0.8528894782066345 + }, + { + "bits_per_byte": 1.3896320297397051, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 0.9632195234298706, + "step": 7600, + "unclipped_grad_norm": 0.8752840161323547 + }, + { + "bits_per_byte": 1.356728394967958, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 0.9404124617576599, + "step": 7610, + "unclipped_grad_norm": 0.8305469751358032 + }, + { + "bits_per_byte": 1.4513170471246686, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 1.0059763193130493, + "step": 7620, + "unclipped_grad_norm": 0.9083051681518555 + }, + { + "bits_per_byte": 1.4324859787510962, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 0.9929236173629761, + "step": 7630, + "unclipped_grad_norm": 0.9056708216667175 + }, + { + "bits_per_byte": 1.4319502527936587, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.9925522804260254, + "step": 7640, + "unclipped_grad_norm": 0.8590258955955505 + }, + { + "bits_per_byte": 1.5054876265454835, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 1.0435245037078857, + "step": 7650, + "unclipped_grad_norm": 0.9650558233261108 + }, + { + "bits_per_byte": 1.4398309278128196, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 0.9980147480964661, + "step": 7660, + "unclipped_grad_norm": 0.8797776103019714 + }, + { + "bits_per_byte": 1.4199840439119262, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 0.9842579364776611, + "step": 7670, + "unclipped_grad_norm": 0.886380136013031 + }, + { + "bits_per_byte": 1.3935578777109503, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 0.9659407138824463, + "step": 7680, + "unclipped_grad_norm": 0.8691937923431396 + }, + { + "bits_per_byte": 1.4174700875129391, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 0.9825153946876526, + "step": 7690, + "unclipped_grad_norm": 0.8523685336112976 + }, + { + "bits_per_byte": 1.4521485832415904, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 1.0065526962280273, + "step": 7700, + "unclipped_grad_norm": 0.9244040846824646 + }, + { + "bits_per_byte": 1.3647152692740268, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 0.9459485411643982, + "step": 7710, + "unclipped_grad_norm": 0.8665258884429932 + }, + { + "bits_per_byte": 1.3670638643542092, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 0.9475764632225037, + "step": 7720, + "unclipped_grad_norm": 0.8315685391426086 + }, + { + "bits_per_byte": 1.4242406145207807, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 0.987208366394043, + "step": 7730, + "unclipped_grad_norm": 0.8546077013015747 + }, + { + "bits_per_byte": 1.4185985516765753, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 0.98329758644104, + "step": 7740, + "unclipped_grad_norm": 0.85493403673172 + }, + { + "bits_per_byte": 1.4220051840248658, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9856588840484619, + "step": 7750, + "unclipped_grad_norm": 0.8603043556213379 + }, + { + "bits_per_byte": 1.420401617788221, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 0.9845473766326904, + "step": 7760, + "unclipped_grad_norm": 0.8968905210494995 + }, + { + "bits_per_byte": 1.4019977543193918, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 0.9717907905578613, + "step": 7770, + "unclipped_grad_norm": 0.9049434661865234 + }, + { + "bits_per_byte": 1.5368209737235277, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 1.065243124961853, + "step": 7780, + "unclipped_grad_norm": 0.9317148923873901 + }, + { + "bits_per_byte": 1.4095231991731958, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 0.9770070314407349, + "step": 7790, + "unclipped_grad_norm": 0.8973956108093262 + }, + { + "bits_per_byte": 1.4568208359175803, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 1.0097912549972534, + "step": 7800, + "unclipped_grad_norm": 0.9193503856658936 + }, + { + "bits_per_byte": 1.3830012386356896, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9586234092712402, + "step": 7810, + "unclipped_grad_norm": 0.9014843106269836 + }, + { + "bits_per_byte": 1.405294919710408, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 0.9740762114524841, + "step": 7820, + "unclipped_grad_norm": 0.8997668623924255 + }, + { + "bits_per_byte": 1.4454388521008288, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 1.0019018650054932, + "step": 7830, + "unclipped_grad_norm": 0.8825055956840515 + }, + { + "bits_per_byte": 1.3985925838236337, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.9694305062294006, + "step": 7840, + "unclipped_grad_norm": 0.8272459506988525 + }, + { + "bits_per_byte": 1.3833228461928029, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.9588463306427002, + "step": 7850, + "unclipped_grad_norm": 0.843379557132721 + }, + { + "bits_per_byte": 1.4053173634463458, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 0.9740917682647705, + "step": 7860, + "unclipped_grad_norm": 0.8791112899780273 + }, + { + "bits_per_byte": 1.4490683739646375, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0044176578521729, + "step": 7870, + "unclipped_grad_norm": 0.910906970500946 + }, + { + "bits_per_byte": 1.423160735456013, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9864598512649536, + "step": 7880, + "unclipped_grad_norm": 0.892792284488678 + }, + { + "bits_per_byte": 1.4853603249323581, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 1.0295733213424683, + "step": 7890, + "unclipped_grad_norm": 0.9216833114624023 + }, + { + "bits_per_byte": 1.4699596225128955, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 1.018898367881775, + "step": 7900, + "unclipped_grad_norm": 0.8876175284385681 + }, + { + "bits_per_byte": 1.4520292272818915, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.006469964981079, + "step": 7910, + "unclipped_grad_norm": 0.9382548928260803 + }, + { + "bits_per_byte": 1.378490907625487, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 0.9554970860481262, + "step": 7920, + "unclipped_grad_norm": 0.9017024040222168 + }, + { + "bits_per_byte": 1.3912675847494105, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.9643532037734985, + "step": 7930, + "unclipped_grad_norm": 0.8948923349380493 + }, + { + "bits_per_byte": 1.4533316518968755, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 1.0073727369308472, + "step": 7940, + "unclipped_grad_norm": 0.9608394503593445 + }, + { + "bits_per_byte": 1.400602459072942, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 0.9708236455917358, + "step": 7950, + "unclipped_grad_norm": 0.8757987022399902 + }, + { + "bits_per_byte": 1.4099571973926481, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 0.9773078560829163, + "step": 7960, + "unclipped_grad_norm": 0.8763889074325562 + }, + { + "bits_per_byte": 1.4042331848153067, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 0.9733402729034424, + "step": 7970, + "unclipped_grad_norm": 0.9077334403991699 + }, + { + "bits_per_byte": 1.433415802952986, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 0.9935681223869324, + "step": 7980, + "unclipped_grad_norm": 0.9200549125671387 + }, + { + "bits_per_byte": 1.447165901880793, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 1.003098964691162, + "step": 7990, + "unclipped_grad_norm": 0.9113723039627075 + }, + { + "bits_per_byte": 1.3669772710894998, + "learning_rate": 3e-05, + "loss_nats": 0.9475164413452148, + "step": 8000, + "unclipped_grad_norm": 0.863850474357605 + } + ], + "variant": "uniform_groups_6_7_forward" +} diff --git a/experiments/k3/attnres_forward/results/raw/replay-uniform_groups_6_7_forward-seed-2026073001.json b/experiments/k3/attnres_forward/results/raw/replay-uniform_groups_6_7_forward-seed-2026073001.json new file mode 100644 index 0000000..9f2f0e1 --- /dev/null +++ b/experiments/k3/attnres_forward/results/raw/replay-uniform_groups_6_7_forward-seed-2026073001.json @@ -0,0 +1,14077 @@ +{ + "architecture": "block", + "batch_size": 32, + "canonical_sha256_without_self": "9466924939ae219e8bffcba1bad3353c55386df0081790d2035ad882dc235275", + "depth": 32, + "diagnostics": [ + { + "activation_grad_rms_by_block": [ + 0.0002677410375326872, + 0.0002473295899108052, + 0.00023034414334688336, + 0.00021438254043459892, + 0.00020044577831868082, + 0.00018819731485564262, + 0.00017973134526982903, + 0.000171217747265473, + 0.0001640921109355986, + 0.00016061548376455903, + 0.00015463090676348656, + 0.00015024171443656087, + 0.0001478576596127823, + 0.00014403100067283958, + 0.00013951722939964384, + 0.00013661097909789532, + 0.00013379570737015456, + 0.0001311724045081064, + 0.0001286817860091105, + 0.00012687484559137374, + 0.0001249351626029238, + 0.00012369138130452484, + 0.00012227990373503417, + 0.00012101328320568427, + 0.00011953213106608018, + 0.0001181897969217971, + 0.0001171672047348693, + 0.00011609455395955592, + 0.00011492414341773838, + 0.0001140913664130494, + 0.00011346775136189535, + 0.00011285000800853595 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00021242368711682502, + "first_to_last_ratio": 1.8345658968465297, + "imbalance_abs_log_ratio": 0.6068078850437921, + "last_quartile_mean": 0.0001157896194854402, + "mean": 0.0001511171253696375, + "normalized": [ + 1.7717451736709768, + 1.636674793183293, + 1.5242755762025908, + 1.4186515254985965, + 1.3264266232459345, + 1.2453738409548603, + 1.1893512719369177, + 1.1330135273991528, + 1.085860457802012, + 1.0628542818802846, + 1.023252039669589, + 0.9942070699734702, + 0.9784308644775869, + 0.9531083940389613, + 0.9232390376562555, + 0.9040072643238835, + 0.8853775311228679, + 0.8680181295617843, + 0.8515367513400655, + 0.8395795332993111, + 0.8267439067367663, + 0.8185133286645813, + 0.8091730400239778, + 0.800791325997512, + 0.7909899739933552, + 0.7821072339266707, + 0.7753403490721149, + 0.7682422073314642, + 0.7604971517068638, + 0.7549863467424895, + 0.7508596466770359, + 0.7467718018887739 + ], + "population_cv": 0.2714186609524326 + }, + "activation_output_rms_by_block": [ + 0.0034885562490671873, + 0.004831995815038681, + 0.005952076520770788, + 0.006925336085259914, + 0.00385080324485898, + 0.005542940925806761, + 0.006729288958013058, + 0.007927131839096546, + 0.004081308841705322, + 0.005811207927763462, + 0.007534808944910765, + 0.008581922389566898, + 0.00499920267611742, + 0.007102519739419222, + 0.008542952127754688, + 0.010447698645293713, + 0.005041283555328846, + 0.007636935915797949, + 0.009387043304741383, + 0.011020516976714134, + 0.005807018838822842, + 0.008367395959794521, + 0.01024632342159748, + 0.012378672137856483, + 0.006453148555010557, + 0.009127611294388771, + 0.011037957854568958, + 0.012611051090061665, + 0.007010402157902718, + 0.009794782847166061, + 0.012445803731679916, + 0.014087481424212456 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.0056560162047389895, + "first_to_last_ratio": 0.5480088979804595, + "imbalance_abs_log_ratio": 0.6014637549753238, + "last_quartile_mean": 0.010321029869373888, + "mean": 0.007962599374877755, + "normalized": [ + 0.4381177659237371, + 0.606836484864952, + 0.747504205668035, + 0.8697330807712821, + 0.4836113263475722, + 0.6961220406611074, + 0.845112084785299, + 0.9955457339856744, + 0.512559862622497, + 0.7298129234151854, + 0.9462750278110585, + 1.077778999737589, + 0.6278355146046978, + 0.8919850673170736, + 1.072884836415099, + 1.3120964842531853, + 0.6331203314377778, + 0.9591008609440675, + 1.1788918244911069, + 1.384035094304029, + 0.7292868277593074, + 1.0508372411895934, + 1.2868063479276561, + 1.554601902603762, + 0.8104324041933391, + 1.1463105029730196, + 1.3862254440922965, + 1.5837857082010076, + 0.8804162846646224, + 1.2300986632667847, + 1.5630327667805075, + 1.7692063559870779 + ], + "population_cv": 0.3451071044649734 + }, + "bits_per_byte": 7.959341947457508, + "branch_output_rms_by_sublayer": [ + 0.0022265110164880753, + 0.0027073738165199757, + 0.0021613112185150385, + 0.002722575329244137, + 0.0025601957459002733, + 0.0026850115973502398, + 0.0025323182344436646, + 0.0026861224323511124, + 0.0027253651060163975, + 0.00271838391199708, + 0.002904455177485943, + 0.0026645169127732515, + 0.0029208383057266474, + 0.0026859452482312918, + 0.00299624796025455, + 0.0027359756641089916, + 0.0030937574338167906, + 0.0026767742820084095, + 0.0033804215490818024, + 0.0027451682835817337, + 0.0034370317589491606, + 0.0026999011170119047, + 0.0038969130255281925, + 0.002663193503394723, + 0.004255065228790045, + 0.002626648638397455, + 0.004119740333408117, + 0.0026531771291047335, + 0.003970506135374308, + 0.0026517712976783514, + 0.004385598469525576, + 0.002676903735846281, + 0.0041999719105660915, + 0.002670086920261383, + 0.005232699681073427, + 0.0026355870068073273, + 0.004664790350943804, + 0.0026610493659973145, + 0.004898969549685717, + 0.0026388936676084995, + 0.005131533369421959, + 0.0026469926815479994, + 0.005069608334451914, + 0.00264862016774714, + 0.0051318020559847355, + 0.002641494618728757, + 0.0058372304774820805, + 0.002674198243767023, + 0.0059690922498703, + 0.002557571977376938, + 0.00554921617731452, + 0.00252733426168561, + 0.0058550527319312096, + 0.0025468147359788418, + 0.006255472078919411, + 0.0026856842450797558, + 0.006502739619463682, + 0.002551089273765683, + 0.006122888531535864, + 0.0025778315030038357, + 0.007071305997669697, + 0.0026001501828432083, + 0.006241494789719582, + 0.0025662274565547705 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0056993408498569815, + 0.005586029321860134, + 0.005830344684528519, + 0.00540288712216694, + 0.00538025811909819, + 0.0052226633069546126, + 0.005297605878918106, + 0.0052023753769565, + 0.0048107019260181325, + 0.005248672753719287, + 0.004962353375385338, + 0.005501319898161136, + 0.005588445553926757, + 0.005229606380765812, + 0.004884302670286647, + 0.005031817046498285, + 0.005000470961685071, + 0.0053937604587735305, + 0.004980398173225761, + 0.004658922297714218, + 0.00496598185096497, + 0.004578385934052906, + 0.004596161734371138, + 0.004990442293511754, + 0.005201771295872185, + 0.00459813521112035, + 0.004706502430387731, + 0.004781878207244352, + 0.0048378722082161795, + 0.004473110720597102, + 0.005084432926394394, + 0.004685705425265166 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.005452688082542498, + "first_to_last_ratio": 1.1368823875795573, + "imbalance_abs_log_ratio": 0.12828976841498158, + "last_quartile_mean": 0.004796176053137182, + "mean": 0.005075395512328069, + "normalized": [ + 1.122935313319594, + 1.1006096585560166, + 1.1487468652180286, + 1.064525337787646, + 1.060066768398564, + 1.0290160233362764, + 1.0437818818356701, + 1.025018713185997, + 0.947847692723257, + 1.0341406380981206, + 0.9777274230809889, + 1.0839194472230789, + 1.1010857262951226, + 1.0303840100861434, + 0.9623491722808084, + 0.9914137793352396, + 0.9852376922229987, + 1.0627271206100406, + 0.9812827711906272, + 0.9179427074003902, + 0.9784423379227619, + 0.9020747098294245, + 0.9055770576316904, + 0.9832617539638114, + 1.024899691706223, + 0.905965889742295, + 0.9273173724009683, + 0.9421685848185097, + 0.9532010257062827, + 0.8813324419214177, + 1.00178063247374, + 0.9232197596982638 + ], + "population_cv": 0.06921897980134169 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6931471824645996, + "mean_weights": [ + 0.5, + 0.5 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0986123085021973, + "mean_weights": [ + 0.3333333432674408, + 0.3333333432674408, + 0.3333333432674408 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862943649291992, + "mean_weights": [ + 0.25, + 0.25, + 0.25, + 0.25 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6094379425048828, + "mean_weights": [ + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343, + 0.20000001788139343 + ], + "sources": 5 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.02802751213312149, + 0.014061377383768559, + 0.01413465291261673, + 0.014165290631353855, + 0.014225807972252369, + 0.014281988143920898, + 0.01433455292135477, + 0.014350161887705326, + 0.01440478302538395, + 0.00966146495193243, + 0.009703479707241058, + 0.009741888381540775, + 0.00977881159633398, + 0.009820535778999329, + 0.009858808480203152, + 0.00991273857653141, + 0.009950309991836548, + 0.0074927168898284435, + 0.00752287870272994, + 0.007546746172010899, + 0.007571155205368996, + 0.00766373984515667, + 0.007693085819482803, + 0.007762390188872814, + 0.007788818795233965, + 0.006325984839349985, + 0.006353048607707024, + 0.006375366821885109, + 0.006389216054230928, + 0.006445344537496567, + 0.0064707002602517605, + 0.006555322092026472, + 0.00656962301582098, + 0.005531130358576775, + 0.005553483963012695, + 0.005611324217170477, + 0.0056249531917274, + 0.0056777093559503555, + 0.005688921548426151, + 0.00574630219489336, + 0.005768724251538515, + 0.004946626722812653, + 0.004966245498508215, + 0.0050392793491482735, + 0.005057469941675663, + 0.005084683652967215, + 0.005098348017781973, + 0.005203457549214363, + 0.005216081161051989, + 0.004667261149734259, + 0.0046758754178881645, + 0.004717131145298481, + 0.004727107938379049, + 0.004768412560224533, + 0.004777040798217058, + 0.004826710093766451, + 0.004837613552808762, + 0.0044052014127373695, + 0.0044186655431985855, + 0.004504534415900707, + 0.004510571248829365, + 0.004574235528707504, + 0.0045849340967834, + 0.004644096828997135 + ], + "loss_nats": 5.516995429992676, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1972246170043945, + "mean_weights": [ + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519, + 0.11111113429069519 + ], + "sources": 9 + }, + "step": 0, + "stream_state_rms_by_sublayer": [ + 0.0022265110164880753, + 0.0034885562490671873, + 0.0040087271481752396, + 0.004831995815038681, + 0.005379301495850086, + 0.005952076520770788, + 0.006410556845366955, + 0.006925336085259914, + 0.0027253651060163975, + 0.00385080324485898, + 0.004838948138058186, + 0.005542940925806761, + 0.006175771821290255, + 0.006729288958013058, + 0.00742313964292407, + 0.007927131839096546, + 0.0030937574338167906, + 0.004081308841705322, + 0.005150026176124811, + 0.005811207927763462, + 0.007019025273621082, + 0.007534808944910765, + 0.008143599145114422, + 0.008581922389566898, + 0.004255065228790045, + 0.00499920267611742, + 0.0066224560141563416, + 0.007102519739419222, + 0.008129785768687725, + 0.008542952127754688, + 0.010099578648805618, + 0.010447698645293713, + 0.0041999719105660915, + 0.005041283555328846, + 0.00719039561226964, + 0.007636935915797949, + 0.009030314162373543, + 0.009387043304741383, + 0.010736005380749702, + 0.011020516976714134, + 0.005131533369421959, + 0.005807018838822842, + 0.00787449348717928, + 0.008367395959794521, + 0.009912566281855106, + 0.01024632342159748, + 0.012113703414797783, + 0.012378672137856483, + 0.0059690922498703, + 0.006453148555010557, + 0.00876442901790142, + 0.009127611294388771, + 0.010806876234710217, + 0.011037957854568958, + 0.012353328987956047, + 0.012611051090061665, + 0.006502739619463682, + 0.007010402157902718, + 0.009470963850617409, + 0.009794782847166061, + 0.012123330496251583, + 0.012445803731679916, + 0.01384514570236206, + 0.014087481424212456 + ] + }, + { + "activation_grad_rms_by_block": [ + 6.941999163245782e-05, + 6.621817010454834e-05, + 6.276910426095128e-05, + 5.9923673688899726e-05, + 5.571635483647697e-05, + 5.268222594168037e-05, + 4.969063229509629e-05, + 4.642214116756804e-05, + 4.2499988921917975e-05, + 3.978446693508886e-05, + 3.700290835695341e-05, + 3.4550401323940605e-05, + 3.204427412129007e-05, + 2.98763407045044e-05, + 2.8043992642778903e-05, + 2.6406481993035413e-05, + 2.5202029064530507e-05, + 2.3980048354133032e-05, + 2.2970654754317366e-05, + 2.2038069801055826e-05, + 2.1245627067401074e-05, + 2.046030931524001e-05, + 1.9785346012213267e-05, + 1.923364652611781e-05, + 1.899822564155329e-05, + 1.857742972788401e-05, + 1.8227287000627257e-05, + 1.7915044736582786e-05, + 1.7522566849947907e-05, + 1.7321037375950254e-05, + 1.712825905997306e-05, + 1.6981100998236798e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 5.7855286740959855e-05, + "first_to_last_ratio": 3.244124255259362, + "imbalance_abs_log_ratio": 1.1768454386205345, + "last_quartile_mean": 1.783386892384442e-05, + "mean": 3.283243222540477e-05, + "normalized": [ + 2.114372494729241, + 2.016852411357775, + 1.9118018375861412, + 1.8251365990038515, + 1.6969913911332248, + 1.6045788377784698, + 1.513461809772568, + 1.41391112449013, + 1.2944514323563494, + 1.2117429090222809, + 1.1270230637473653, + 1.052325368000197, + 0.9759945258181376, + 0.9099642846863768, + 0.8541551978314688, + 0.80428040821791, + 0.7675955558671622, + 0.7303768477919212, + 0.6996330517525091, + 0.6712286695593455, + 0.6470926954647555, + 0.6231737318384967, + 0.6026159096706807, + 0.585812418466987, + 0.5786420424513363, + 0.5658255715063759, + 0.5551610333188632, + 0.545650855641473, + 0.5336968863485374, + 0.5275587643655515, + 0.5216871824293211, + 0.5172050879951964 + ], + "population_cv": 0.4964530316937415 + }, + "activation_output_rms_by_block": [ + 0.0058692265301942825, + 0.010152247734367847, + 0.014839091338217258, + 0.019591957330703735, + 0.0067398143000900745, + 0.011955589056015015, + 0.01716931350529194, + 0.02306033857166767, + 0.007508765906095505, + 0.013779806904494762, + 0.020409800112247467, + 0.02685040980577469, + 0.0091559374704957, + 0.016716359183192253, + 0.02419254370033741, + 0.03126180171966553, + 0.009785215370357037, + 0.01850821264088154, + 0.02761545218527317, + 0.037425313144922256, + 0.012438409961760044, + 0.023153970018029213, + 0.03432171046733856, + 0.04660208150744438, + 0.017327340319752693, + 0.02931905910372734, + 0.044478677213191986, + 0.06014470010995865, + 0.019144421443343163, + 0.03802234306931496, + 0.05717705935239792, + 0.07674983143806458 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.013672197295818478, + "first_to_last_ratio": 0.3194779819553082, + "imbalance_abs_log_ratio": 1.141066921607124, + "last_quartile_mean": 0.04279542900621891, + "mean": 0.02535833751608152, + "normalized": [ + 0.23145155026319017, + 0.40035147130325827, + 0.585176032490566, + 0.7726041708483092, + 0.26578297160907655, + 0.4714658067956201, + 0.6770677886278491, + 0.9093789589732953, + 0.29610639504004016, + 0.5434034031511731, + 0.804855606141537, + 1.058839515356515, + 0.36106221335248306, + 0.6592056428222561, + 0.9540271985494003, + 1.2328017047584527, + 0.38587763744967657, + 0.7298669571356627, + 1.0890087793712917, + 1.4758583097645188, + 0.49050573421352905, + 0.9130712927590635, + 1.3534684774020669, + 1.8377419843824816, + 0.6832995384166726, + 1.1561901124288627, + 1.7540060417992664, + 2.3717919233394786, + 0.7549556997260695, + 1.4994020426300538, + 2.254763716909198, + 3.0266113221890856 + ], + "population_cv": 0.6561149265484539 + }, + "bits_per_byte": 5.151485772272156, + "branch_output_rms_by_sublayer": [ + 0.0029127895832061768, + 0.005085675045847893, + 0.0026882688980549574, + 0.005228722468018532, + 0.0028465059585869312, + 0.005698997061699629, + 0.0029055613558739424, + 0.0058838739059865475, + 0.003085965756326914, + 0.005918111652135849, + 0.0032896376214921474, + 0.005925621837377548, + 0.0038608317263424397, + 0.005900311749428511, + 0.0034083875361829996, + 0.006688539404422045, + 0.003751161741092801, + 0.006295192055404186, + 0.004302044399082661, + 0.006429112516343594, + 0.004344411194324493, + 0.006875772960484028, + 0.004838971421122551, + 0.006519583985209465, + 0.005485833622515202, + 0.006734788883477449, + 0.005453970283269882, + 0.007042968180030584, + 0.00535221304744482, + 0.006980822887271643, + 0.0056844716891646385, + 0.006854009814560413, + 0.005563640967011452, + 0.007032194174826145, + 0.007088648620992899, + 0.007161186542361975, + 0.006791478954255581, + 0.007514364551752806, + 0.00717391399666667, + 0.00770252151414752, + 0.0077114407904446125, + 0.0078731095418334, + 0.008122370578348637, + 0.008188363164663315, + 0.00878527294844389, + 0.008078450337052345, + 0.009222123771905899, + 0.00825345516204834, + 0.01271942537277937, + 0.007952263578772545, + 0.00897612702101469, + 0.008032192476093769, + 0.012183033861219883, + 0.008434710092842579, + 0.012307075783610344, + 0.00904827006161213, + 0.013551505282521248, + 0.00846139620989561, + 0.014996928162872791, + 0.008829282596707344, + 0.015547323040664196, + 0.008883194997906685, + 0.014805778861045837, + 0.009233531542122364 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.000395177475103743, + 0.0003332755344654961, + 0.000315893459816185, + 0.000310100133167921, + 0.00028528332620524554, + 0.00028335812413191985, + 0.00029518873948296287, + 0.00026293067142410537, + 0.0002424619071402874, + 0.00026201411214491806, + 0.00023865248769426136, + 0.0002448734927312214, + 0.0002522633348151854, + 0.00023781239740430576, + 0.00021251402376915363, + 0.0002122590435785732, + 0.00020152644174535755, + 0.0002078757425859546, + 0.00020034669926835373, + 0.00018987579939082946, + 0.0001906458879908057, + 0.0001799030931959508, + 0.00017670666675783845, + 0.00017209476706835628, + 0.00018834348956749082, + 0.00015120127877278264, + 0.00017384489513614726, + 0.00016007662150962894, + 0.00016217445507190053, + 0.00017021204806660572, + 0.00017090328227710973, + 0.00016319949033805103 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0003101509329746973, + "first_to_last_ratio": 1.8517087704220867, + "imbalance_abs_log_ratio": 0.6161088724667294, + "last_quartile_mean": 0.00016749444509246459, + "mean": 0.00022634340380683275, + "normalized": [ + 1.745919997920494, + 1.47243316509423, + 1.39563801950145, + 1.3700427224844969, + 1.260400442014708, + 1.251894773013774, + 1.3041632074018137, + 1.1616449474644162, + 1.0712126046633574, + 1.1575955284675652, + 1.0543823397563354, + 1.081867147938637, + 1.1145159548385752, + 1.0506707657681995, + 0.9389008921617108, + 0.9377743729599494, + 0.8903570342935435, + 0.9184086617490346, + 0.8851448546710675, + 0.8388837324054484, + 0.8422860343370455, + 0.794823662497736, + 0.7807016408953734, + 0.7603259656518481, + 0.8321138871280206, + 0.6680171643164899, + 0.7680581462162288, + 0.7072290105093693, + 0.7164973767484046, + 0.7520079896468688, + 0.7550619077150706, + 0.7210260497687383 + ], + "population_cv": 0.26063317009417175 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6930886507034302, + "mean_weights": [ + 0.502555787563324, + 0.497444212436676 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929051876068115, + "mean_weights": [ + 0.5103161931037903, + 0.4896838068962097 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928611993789673, + "mean_weights": [ + 0.5053715109825134, + 0.4946284890174866 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927852630615234, + "mean_weights": [ + 0.5126319527626038, + 0.48736804723739624 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6928048729896545, + "mean_weights": [ + 0.5031334757804871, + 0.49686652421951294 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6927545666694641, + "mean_weights": [ + 0.5128438472747803, + 0.4871561527252197 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6925621032714844, + "mean_weights": [ + 0.5113073587417603, + 0.48869261145591736 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6929246187210083, + "mean_weights": [ + 0.5095134973526001, + 0.4904865026473999 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0983444452285767, + "mean_weights": [ + 0.3399057686328888, + 0.3283795118331909, + 0.3317147493362427 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984482765197754, + "mean_weights": [ + 0.3407744765281677, + 0.33049434423446655, + 0.3287311792373657 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0980393886566162, + "mean_weights": [ + 0.3472622334957123, + 0.3314278721809387, + 0.3213099241256714 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985229015350342, + "mean_weights": [ + 0.3373940885066986, + 0.33421406149864197, + 0.3283918499946594 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0982658863067627, + "mean_weights": [ + 0.3445737361907959, + 0.33073461055755615, + 0.32469165325164795 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0985300540924072, + "mean_weights": [ + 0.3371811807155609, + 0.3333006501197815, + 0.32951819896698 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0983130931854248, + "mean_weights": [ + 0.34229734539985657, + 0.33283179998397827, + 0.32487088441848755 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0984888076782227, + "mean_weights": [ + 0.33794093132019043, + 0.33444374799728394, + 0.32761532068252563 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3861021995544434, + "mean_weights": [ + 0.2562229633331299, + 0.25146764516830444, + 0.24576731026172638, + 0.24654211103916168 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3861823081970215, + "mean_weights": [ + 0.2537113130092621, + 0.25214654207229614, + 0.24717503786087036, + 0.2469671070575714 + ], + "sources": 4 + }, + { + "entropy_mean": 1.386094093322754, + "mean_weights": [ + 0.2571265399456024, + 0.2500755786895752, + 0.24622224271297455, + 0.24657560884952545 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862230777740479, + "mean_weights": [ + 0.2514685392379761, + 0.2528022825717926, + 0.24948549270629883, + 0.24624371528625488 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3860026597976685, + "mean_weights": [ + 0.25493288040161133, + 0.25381627678871155, + 0.24853172898292542, + 0.2427191287279129 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862276077270508, + "mean_weights": [ + 0.25173667073249817, + 0.25301432609558105, + 0.24867257475852966, + 0.2465764284133911 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3859740495681763, + "mean_weights": [ + 0.2546749711036682, + 0.2548622786998749, + 0.24870140850543976, + 0.24176132678985596 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3862535953521729, + "mean_weights": [ + 0.2502487599849701, + 0.25214675068855286, + 0.24993090331554413, + 0.24767357110977173 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6090558767318726, + "mean_weights": [ + 0.20371176302433014, + 0.2064536213874817, + 0.19979709386825562, + 0.19547516107559204, + 0.1945623755455017 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609304666519165, + "mean_weights": [ + 0.20178237557411194, + 0.20363003015518188, + 0.20010292530059814, + 0.19683599472045898, + 0.19764868915081024 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6086277961730957, + "mean_weights": [ + 0.20440815389156342, + 0.20845328271389008, + 0.2041691243648529, + 0.19542333483695984, + 0.18754610419273376 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093450784683228, + "mean_weights": [ + 0.20030716061592102, + 0.20382772386074066, + 0.20014742016792297, + 0.19763237237930298, + 0.19808533787727356 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6085963249206543, + "mean_weights": [ + 0.20527315139770508, + 0.21035820245742798, + 0.20175111293792725, + 0.19382911920547485, + 0.18878838419914246 + ], + "sources": 5 + }, + { + "entropy_mean": 1.609370470046997, + "mean_weights": [ + 0.2003176212310791, + 0.2033592313528061, + 0.20006224513053894, + 0.19803357124328613, + 0.19822733104228973 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6083728075027466, + "mean_weights": [ + 0.20376041531562805, + 0.21104899048805237, + 0.2038523405790329, + 0.19410842657089233, + 0.18722981214523315 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6093378067016602, + "mean_weights": [ + 0.2008873075246811, + 0.20332497358322144, + 0.20115891098976135, + 0.19900420308113098, + 0.19562460482120514 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7908473014831543, + "mean_weights": [ + 0.17166128754615784, + 0.17726510763168335, + 0.16891533136367798, + 0.16100504994392395, + 0.15850719809532166, + 0.16264605522155762 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7916767597198486, + "mean_weights": [ + 0.16709068417549133, + 0.16979600489139557, + 0.1685849130153656, + 0.1652715802192688, + 0.16460880637168884, + 0.16464802622795105 + ], + "sources": 6 + }, + { + "entropy_mean": 1.789864420890808, + "mean_weights": [ + 0.17288848757743835, + 0.18083803355693817, + 0.17338326573371887, + 0.16238118708133698, + 0.15700742602348328, + 0.15350158512592316 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791719913482666, + "mean_weights": [ + 0.16619735956192017, + 0.168890118598938, + 0.16766680777072906, + 0.16658610105514526, + 0.16604891419410706, + 0.16461071372032166 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7891879081726074, + "mean_weights": [ + 0.17246663570404053, + 0.18376711010932922, + 0.17454420030117035, + 0.16218964755535126, + 0.15576335787773132, + 0.15126904845237732 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7917189598083496, + "mean_weights": [ + 0.16667470335960388, + 0.16900496184825897, + 0.166001096367836, + 0.16513821482658386, + 0.16654902696609497, + 0.16663199663162231 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7891875505447388, + "mean_weights": [ + 0.17235296964645386, + 0.18330925703048706, + 0.1743936687707901, + 0.1641043871641159, + 0.15562792122364044, + 0.15021178126335144 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079333782196045, + "mean_weights": [ + 0.12415941059589386, + 0.12247946858406067, + 0.12275533378124237, + 0.1243581771850586, + 0.1256372034549713, + 0.12609991431236267, + 0.12752515077590942, + 0.1269853413105011 + ], + "sources": 8 + }, + { + "entropy_mean": 2.19067645072937, + "mean_weights": [ + 0.11858826875686646, + 0.13251525163650513, + 0.12347469478845596, + 0.11762590706348419, + 0.11183994263410568, + 0.10465474426746368, + 0.0985296368598938, + 0.09454642236232758, + 0.09822513163089752 + ], + "sources": 9 + }, + { + "entropy_mean": 2.197180986404419, + "mean_weights": [ + 0.11080950498580933, + 0.11013305932283401, + 0.11013011634349823, + 0.11041422188282013, + 0.11128482222557068, + 0.11163365840911865, + 0.1125442311167717, + 0.11194804310798645, + 0.11110235750675201 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1899502277374268, + "mean_weights": [ + 0.11909156292676926, + 0.1331091821193695, + 0.12361659109592438, + 0.11795277148485184, + 0.11277814209461212, + 0.1057085245847702, + 0.09961889684200287, + 0.0943179503083229, + 0.09380637109279633 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1971864700317383, + "mean_weights": [ + 0.11064367741346359, + 0.11015763878822327, + 0.11035164445638657, + 0.11099889129400253, + 0.1101146712899208, + 0.11141451448202133, + 0.11247441172599792, + 0.11148296296596527, + 0.11236158758401871 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1897201538085938, + "mean_weights": [ + 0.11971956491470337, + 0.13387462496757507, + 0.12342339754104614, + 0.1180819645524025, + 0.11173982918262482, + 0.10558837652206421, + 0.10011725127696991, + 0.09448471665382385, + 0.09297028183937073 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1971964836120605, + "mean_weights": [ + 0.11113918572664261, + 0.11165578663349152, + 0.11136722564697266, + 0.11112368106842041, + 0.11216380447149277, + 0.11136430501937866, + 0.1116628348827362, + 0.10975746810436249, + 0.10976571589708328 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1886677742004395, + "mean_weights": [ + 0.11989010870456696, + 0.13561508059501648, + 0.1242879331111908, + 0.1177452951669693, + 0.11283841729164124, + 0.1049613356590271, + 0.09941282868385315, + 0.09397745132446289, + 0.0912715420126915 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.028571274131536484, + 0.014370470307767391, + 0.015031064860522747, + 0.014924541115760803, + 0.01576058566570282, + 0.015609913505613804, + 0.01673153042793274, + 0.01674462854862213, + 0.01786557026207447, + 0.012058191932737827, + 0.013009775429964066, + 0.013223569840192795, + 0.014060634188354015, + 0.01422955747693777, + 0.015225647017359734, + 0.015402527526021004, + 0.016711384057998657, + 0.012696896679699421, + 0.013651188462972641, + 0.01374440360814333, + 0.014770817942917347, + 0.01499935146421194, + 0.01605224795639515, + 0.016234049573540688, + 0.017371296882629395, + 0.014139331877231598, + 0.0150116803124547, + 0.015255129896104336, + 0.01619553007185459, + 0.01641164906322956, + 0.017412541434168816, + 0.017549853771924973, + 0.01851520501077175, + 0.0156314168125391, + 0.01649715006351471, + 0.016789749264717102, + 0.01767360419034958, + 0.017833195626735687, + 0.018907228484749794, + 0.019026586785912514, + 0.02030135691165924, + 0.017719922587275505, + 0.018491322174668312, + 0.018898772075772285, + 0.019703106954693794, + 0.020232124254107475, + 0.021055491641163826, + 0.021651869639754295, + 0.02249983325600624, + 0.02048179879784584, + 0.021194644272327423, + 0.021763430908322334, + 0.02249440737068653, + 0.023286480456590652, + 0.024062326177954674, + 0.02487023174762726, + 0.02590898610651493, + 0.022659940645098686, + 0.024567829445004463, + 0.024109894409775734, + 0.02627130225300789, + 0.02551914006471634, + 0.02790611796081066, + 0.02684609591960907 + ], + "loss_nats": 3.570737838745117, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.1932034492492676, + "mean_weights": [ + 0.10075974464416504, + 0.09453918039798737, + 0.10217029601335526, + 0.10980333387851715, + 0.11425476521253586, + 0.11821146309375763, + 0.11953863501548767, + 0.12103148549795151, + 0.11969110369682312 + ], + "sources": 9 + }, + "step": 100, + "stream_state_rms_by_sublayer": [ + 0.0029127895832061768, + 0.0058692265301942825, + 0.006536925211548805, + 0.010152247734367847, + 0.010624829679727554, + 0.014839091338217258, + 0.0151302395388484, + 0.019591957330703735, + 0.003085965756326914, + 0.0067398143000900745, + 0.007633335888385773, + 0.011955589056015015, + 0.012725084088742733, + 0.01716931350529194, + 0.01780773513019085, + 0.02306033857166767, + 0.003751161741092801, + 0.007508765906095505, + 0.0091110710054636, + 0.013779806904494762, + 0.015157311223447323, + 0.020409800112247467, + 0.02179115265607834, + 0.02685040980577469, + 0.005485833622515202, + 0.0091559374704957, + 0.011829552240669727, + 0.016716359183192253, + 0.018926752731204033, + 0.02419254370033741, + 0.02609706111252308, + 0.03126180171966553, + 0.005563640967011452, + 0.009785215370357037, + 0.01357173826545477, + 0.01850821264088154, + 0.021842118352651596, + 0.02761545218527317, + 0.03138817846775055, + 0.037425313144922256, + 0.0077114407904446125, + 0.012438409961760044, + 0.017470013350248337, + 0.023153970018029213, + 0.028209656476974487, + 0.03432171046733856, + 0.04007846489548683, + 0.04660208150744438, + 0.01271942537277937, + 0.017327340319752693, + 0.02364915981888771, + 0.02931905910372734, + 0.03804082050919533, + 0.044478677213191986, + 0.05299994722008705, + 0.06014470010995865, + 0.013551505282521248, + 0.019144421443343163, + 0.031610529869794846, + 0.03802234306931496, + 0.05028318613767624, + 0.05717705935239792, + 0.06944417208433151, + 0.07674983143806458 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00013502020738087595, + 0.0001386140938848257, + 0.00013852481788489968, + 0.00013219378888607025, + 0.00013201223919168115, + 0.00011907621228601784, + 0.00010749710054369643, + 9.981176117435098e-05, + 9.781593689695e-05, + 8.850422454997897e-05, + 8.08831537142396e-05, + 7.543387619080022e-05, + 7.733904203632846e-05, + 7.133937469916418e-05, + 6.59702272969298e-05, + 6.017514533596113e-05, + 5.727267125621438e-05, + 5.206368223298341e-05, + 4.759432704304345e-05, + 4.3677500798366964e-05, + 3.97721778426785e-05, + 3.6940233258064836e-05, + 3.430873766774312e-05, + 3.225093678338453e-05, + 2.9220358555903658e-05, + 2.7307431082590483e-05, + 2.608104477985762e-05, + 2.5369541617692448e-05, + 2.8449645469663665e-05, + 2.820042209350504e-05, + 2.805530675686896e-05, + 2.801550544972997e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00012534377765405225, + "first_to_last_ratio": 4.543514284047766, + "imbalance_abs_log_ratio": 1.5137007839906835, + "last_quartile_mean": 2.758740697572648e-05, + "mean": 6.827471014503317e-05, + "normalized": [ + 1.9776020592992347, + 2.03024067902117, + 2.0289330791831572, + 1.9362043222923453, + 1.9335452162484903, + 1.7440749588400906, + 1.574479046712095, + 1.4619140961905945, + 1.432681833275654, + 1.2962958665364244, + 1.1846722387018975, + 1.1048582415151837, + 1.13276265650986, + 1.0448872583658118, + 0.9662469039676911, + 0.8813680088591155, + 0.8388563076218469, + 0.7625617468369571, + 0.6971003896163128, + 0.6397317645960655, + 0.5825316249339191, + 0.5410529488824517, + 0.5025101914729843, + 0.4723701750600653, + 0.42798216929565935, + 0.39996407197602785, + 0.38200154529334107, + 0.3715803635606943, + 0.41669375687175014, + 0.413043453917271, + 0.41091799140958957, + 0.41033503313624886 + ], + "population_cv": 0.5682770161254035 + }, + "activation_output_rms_by_block": [ + 0.021690133959054947, + 0.038645509630441666, + 0.050416164100170135, + 0.059038665145635605, + 0.015957212075591087, + 0.0259401835501194, + 0.03326375409960747, + 0.03896600753068924, + 0.01569177769124508, + 0.02718697488307953, + 0.03437623754143715, + 0.03927858546376228, + 0.0174946840852499, + 0.027429770678281784, + 0.036497704684734344, + 0.042413465678691864, + 0.01978003978729248, + 0.02902580052614212, + 0.04184097424149513, + 0.049806781113147736, + 0.02072436735033989, + 0.035371601581573486, + 0.05015818774700165, + 0.06557824462652206, + 0.024349043145775795, + 0.0433179996907711, + 0.06327643990516663, + 0.08553080260753632, + 0.032196998596191406, + 0.06154516339302063, + 0.0982988178730011, + 0.1370837539434433 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.03548970376141369, + "first_to_last_ratio": 0.5203778234995319, + "imbalance_abs_log_ratio": 0.6532001475878298, + "last_quartile_mean": 0.06819987739436328, + "mean": 0.043192870216444135, + "normalized": [ + 0.5021693129065826, + 0.894719647867457, + 1.1672334773662711, + 1.366861355816052, + 0.3694408821554987, + 0.600566330047768, + 0.770121409689127, + 0.9021398053758959, + 0.36329555347009557, + 0.6294320045609996, + 0.7958775920464214, + 0.9093766000484119, + 0.4050363867365644, + 0.6350532053282925, + 0.8449937339621194, + 0.9819552501640527, + 0.45794687151311236, + 0.6720044391745837, + 0.9687009460548811, + 1.1531250612325734, + 0.4798099141475861, + 0.8189222296254584, + 1.1612608167888254, + 1.5182654984006019, + 0.5637282964470782, + 1.002896993732992, + 1.4649741864358992, + 1.9802065058175629, + 0.7454239191526002, + 1.4248917259865153, + 2.2758112017195224, + 3.1737588462285977 + ], + "population_cv": 0.5898779055306158 + }, + "bits_per_byte": 3.0783016453895327, + "branch_output_rms_by_sublayer": [ + 0.010510989464819431, + 0.01773845963180065, + 0.01288721151649952, + 0.015410542488098145, + 0.013293960131704807, + 0.011829170398414135, + 0.012563494965434074, + 0.010719931684434414, + 0.012978821992874146, + 0.008852306753396988, + 0.012484307400882244, + 0.008082501590251923, + 0.017479149624705315, + 0.007394776213914156, + 0.013811159878969193, + 0.008539069443941116, + 0.012748918496072292, + 0.008203585632145405, + 0.015691176056861877, + 0.00825700256973505, + 0.019360050559043884, + 0.008359342813491821, + 0.01773277297616005, + 0.008825630880892277, + 0.014515543356537819, + 0.008455797098577023, + 0.014256616123020649, + 0.009114319458603859, + 0.017845245078206062, + 0.009323488920927048, + 0.011321012862026691, + 0.00932974461466074, + 0.01630145125091076, + 0.010338325053453445, + 0.014692036435008049, + 0.01085656601935625, + 0.022584766149520874, + 0.012223388068377972, + 0.021950112655758858, + 0.014179684221744537, + 0.015464341267943382, + 0.013388797640800476, + 0.016217676922678947, + 0.014884707517921925, + 0.019293656572699547, + 0.01591324806213379, + 0.015093626454472542, + 0.018570836633443832, + 0.016100121662020683, + 0.01750165969133377, + 0.011022309772670269, + 0.01947851851582527, + 0.008948881179094315, + 0.021226298063993454, + 0.007377106696367264, + 0.02357090450823307, + 0.007519193924963474, + 0.02960866689682007, + 0.008151640184223652, + 0.02826736681163311, + 0.009839511476457119, + 0.034673310816287994, + 0.011650451458990574, + 0.03381374478340149 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.000534027543251826, + 0.0003551799374971097, + 0.0003169502727458517, + 0.00031266717764008215, + 0.0002927028358434817, + 0.00025320354741983973, + 0.00021793521292911338, + 0.0002553923951558317, + 0.0002247035985789251, + 0.0001890734727443061, + 0.00017489844348413705, + 0.00016649772261059797, + 0.0001703301962196103, + 0.00015541374025934292, + 0.00014978136198409063, + 0.00013453045687611686, + 0.00014882006004230677, + 0.00013684889755545492, + 0.00014149569788357677, + 0.00013256541433128886, + 0.00011810236114568028, + 0.00011399650055164618, + 0.00011714792127563267, + 0.0001104246640026569, + 0.0001026434139331144, + 9.221143034651731e-05, + 8.781994013379414e-05, + 7.90832897193947e-05, + 0.00010462192413175276, + 9.52877304114312e-05, + 0.00011873406905510886, + 0.0001114991774094402 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.000317257365310392, + "first_to_last_ratio": 3.20502057979239, + "imbalance_abs_log_ratio": 1.1647185115040897, + "last_quartile_mean": 9.89876218925692e-05, + "mean": 0.00017858095022403313, + "normalized": [ + 2.9903947906082844, + 1.9889015992552947, + 1.774826891380249, + 1.7508428376477747, + 1.6390484846019722, + 1.417864263249754, + 1.2203721206305018, + 1.4301211570183558, + 1.2582730593438451, + 1.058754993223577, + 0.979379061790878, + 0.9323375332123807, + 0.9537982411109802, + 0.8702705415352169, + 0.8387309049267972, + 0.753330389984743, + 0.8333479010813261, + 0.7663129655418197, + 0.7923336599232325, + 0.7423267384622104, + 0.6613379590461282, + 0.6383463656601415, + 0.6559933807534813, + 0.6183451474758485, + 0.5747724704362158, + 0.5163564771653211, + 0.49176544319885407, + 0.44284280949442395, + 0.5858515367988725, + 0.533582838997614, + 0.6648753347216193, + 0.6243621017222857 + ], + "population_cv": 0.5428104396751369 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.6472191214561462, + "mean_weights": [ + 0.3721027076244354, + 0.627897322177887 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6685923337936401, + "mean_weights": [ + 0.4305521249771118, + 0.5694478750228882 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6887905597686768, + "mean_weights": [ + 0.5073041915893555, + 0.49269580841064453 + ], + "sources": 2 + }, + { + "entropy_mean": 0.674967885017395, + "mean_weights": [ + 0.4601387083530426, + 0.539861261844635 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6842271089553833, + "mean_weights": [ + 0.46463143825531006, + 0.5353685021400452 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6764289140701294, + "mean_weights": [ + 0.45219624042510986, + 0.5478037595748901 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6803587675094604, + "mean_weights": [ + 0.4517097771167755, + 0.5482901930809021 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6795487403869629, + "mean_weights": [ + 0.4564407467842102, + 0.543559193611145 + ], + "sources": 2 + }, + { + "entropy_mean": 1.09000825881958, + "mean_weights": [ + 0.3023155927658081, + 0.3300391733646393, + 0.367645263671875 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0830318927764893, + "mean_weights": [ + 0.3015379011631012, + 0.3218569755554199, + 0.3766051232814789 + ], + "sources": 3 + }, + { + "entropy_mean": 1.092214822769165, + "mean_weights": [ + 0.30797675251960754, + 0.3204318583011627, + 0.37159138917922974 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0806152820587158, + "mean_weights": [ + 0.31027084589004517, + 0.3089178800582886, + 0.38081130385398865 + ], + "sources": 3 + }, + { + "entropy_mean": 1.088780164718628, + "mean_weights": [ + 0.29294297099113464, + 0.3224499821662903, + 0.38460707664489746 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0694758892059326, + "mean_weights": [ + 0.2983662486076355, + 0.3019666075706482, + 0.3996671736240387 + ], + "sources": 3 + }, + { + "entropy_mean": 1.090681791305542, + "mean_weights": [ + 0.3049081563949585, + 0.31428080797195435, + 0.38081103563308716 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0703673362731934, + "mean_weights": [ + 0.29188719391822815, + 0.30882728099823, + 0.39928552508354187 + ], + "sources": 3 + }, + { + "entropy_mean": 1.375261664390564, + "mean_weights": [ + 0.22141581773757935, + 0.22935809195041656, + 0.2638857364654541, + 0.2853403687477112 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3544485569000244, + "mean_weights": [ + 0.22965481877326965, + 0.2214069962501526, + 0.2419331669807434, + 0.30700498819351196 + ], + "sources": 4 + }, + { + "entropy_mean": 1.375445008277893, + "mean_weights": [ + 0.22887249290943146, + 0.22569704055786133, + 0.24806134402751923, + 0.297369122505188 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3552662134170532, + "mean_weights": [ + 0.2435493767261505, + 0.22663038969039917, + 0.22198355197906494, + 0.3078366816043854 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3799614906311035, + "mean_weights": [ + 0.2553842067718506, + 0.22196413576602936, + 0.24428126215934753, + 0.27837038040161133 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3527708053588867, + "mean_weights": [ + 0.25185543298721313, + 0.22270265221595764, + 0.2234145700931549, + 0.3020273447036743 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3809083700180054, + "mean_weights": [ + 0.2529875934123993, + 0.2247350960969925, + 0.24474212527275085, + 0.27753520011901855 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3513743877410889, + "mean_weights": [ + 0.24927198886871338, + 0.2232775092124939, + 0.2234012484550476, + 0.3040492534637451 + ], + "sources": 4 + }, + { + "entropy_mean": 1.6052519083023071, + "mean_weights": [ + 0.20885375142097473, + 0.18223194777965546, + 0.1860320121049881, + 0.20962435007095337, + 0.21325793862342834 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5736613273620605, + "mean_weights": [ + 0.20247238874435425, + 0.1819916069507599, + 0.1644536554813385, + 0.20117707550525665, + 0.2499052882194519 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6082277297973633, + "mean_weights": [ + 0.19722914695739746, + 0.19568710029125214, + 0.1940370351076126, + 0.20102860033512115, + 0.21201811730861664 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5787183046340942, + "mean_weights": [ + 0.20195573568344116, + 0.1856016367673874, + 0.17212294042110443, + 0.1921299397945404, + 0.24818973243236542 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6036648750305176, + "mean_weights": [ + 0.20087790489196777, + 0.19327759742736816, + 0.18064075708389282, + 0.1987924724817276, + 0.22641125321388245 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5773308277130127, + "mean_weights": [ + 0.19965466856956482, + 0.19064198434352875, + 0.17014144361019135, + 0.18579250574111938, + 0.2537693977355957 + ], + "sources": 5 + }, + { + "entropy_mean": 1.596920132637024, + "mean_weights": [ + 0.20649376511573792, + 0.19138765335083008, + 0.17314665019512177, + 0.18983504176139832, + 0.23913687467575073 + ], + "sources": 5 + }, + { + "entropy_mean": 1.574884295463562, + "mean_weights": [ + 0.20621415972709656, + 0.19019779562950134, + 0.16562730073928833, + 0.17967543005943298, + 0.2582853138446808 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7829394340515137, + "mean_weights": [ + 0.15515035390853882, + 0.167438805103302, + 0.150580033659935, + 0.15603643655776978, + 0.18237966299057007, + 0.18841469287872314 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7675386667251587, + "mean_weights": [ + 0.16129285097122192, + 0.15975908935070038, + 0.14229260385036469, + 0.14818856120109558, + 0.18669070303440094, + 0.2017761766910553 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7799508571624756, + "mean_weights": [ + 0.15204058587551117, + 0.16960464417934418, + 0.15014684200286865, + 0.15069472789764404, + 0.17520248889923096, + 0.2023106813430786 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7687162160873413, + "mean_weights": [ + 0.15768879652023315, + 0.16014868021011353, + 0.14229220151901245, + 0.14887431263923645, + 0.17495375871658325, + 0.21604226529598236 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7769694328308105, + "mean_weights": [ + 0.1434900015592575, + 0.17719288170337677, + 0.15652287006378174, + 0.15137143433094025, + 0.17447316646575928, + 0.19694967567920685 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7790402173995972, + "mean_weights": [ + 0.1672755777835846, + 0.16318361461162567, + 0.14764192700386047, + 0.15013208985328674, + 0.1656753122806549, + 0.20609146356582642 + ], + "sources": 6 + }, + { + "entropy_mean": 1.772436261177063, + "mean_weights": [ + 0.137189120054245, + 0.1825355887413025, + 0.15700370073318481, + 0.15137675404548645, + 0.17622745037078857, + 0.1956673562526703 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0557358264923096, + "mean_weights": [ + 0.11572219431400299, + 0.14279131591320038, + 0.13401660323143005, + 0.1240810751914978, + 0.12964043021202087, + 0.13645057380199432, + 0.10849803686141968, + 0.1087997779250145 + ], + "sources": 8 + }, + { + "entropy_mean": 2.128070592880249, + "mean_weights": [ + 0.08253289759159088, + 0.16997724771499634, + 0.15496571362018585, + 0.12397155165672302, + 0.11783402413129807, + 0.10891544073820114, + 0.08614566922187805, + 0.07161472737789154, + 0.08404272049665451 + ], + "sources": 9 + }, + { + "entropy_mean": 2.190647602081299, + "mean_weights": [ + 0.111246258020401, + 0.11339963972568512, + 0.10967698693275452, + 0.10811547189950943, + 0.11347523331642151, + 0.11441484093666077, + 0.10874959081411362, + 0.1088113933801651, + 0.11211058497428894 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1061489582061768, + "mean_weights": [ + 0.0818711519241333, + 0.18386806547641754, + 0.16306090354919434, + 0.1236560046672821, + 0.11738234013319016, + 0.10990595817565918, + 0.08861973881721497, + 0.07051397860050201, + 0.061121866106987 + ], + "sources": 9 + }, + { + "entropy_mean": 2.174285888671875, + "mean_weights": [ + 0.10868722200393677, + 0.129492849111557, + 0.11783719062805176, + 0.11122089624404907, + 0.11352718621492386, + 0.12009987980127335, + 0.09868897497653961, + 0.10157625377178192, + 0.09886953234672546 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0911030769348145, + "mean_weights": [ + 0.07921186089515686, + 0.1853443682193756, + 0.17000551521778107, + 0.12975995242595673, + 0.12159739434719086, + 0.11123217642307281, + 0.08325955271720886, + 0.0661969929933548, + 0.05339217931032181 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1573359966278076, + "mean_weights": [ + 0.10092437267303467, + 0.1396944522857666, + 0.12192441523075104, + 0.12009409070014954, + 0.13081315159797668, + 0.12593936920166016, + 0.09386305510997772, + 0.08600614964962006, + 0.08074095100164413 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0948352813720703, + "mean_weights": [ + 0.08049570769071579, + 0.187056303024292, + 0.1619722694158554, + 0.1283482015132904, + 0.12387267500162125, + 0.11433333158493042, + 0.08365806937217712, + 0.06816720962524414, + 0.05209621414542198 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.043896060436964035, + 0.017306910827755928, + 0.0239819698035717, + 0.027399448677897453, + 0.030875300988554955, + 0.03204025700688362, + 0.03548785299062729, + 0.03623313829302788, + 0.03888553008437157, + 0.025523660704493523, + 0.027128782123327255, + 0.027281278744339943, + 0.028869546949863434, + 0.029556388035416603, + 0.03074541874229908, + 0.03038080967962742, + 0.03226381912827492, + 0.02304573729634285, + 0.024761522188782692, + 0.02376183681190014, + 0.025758439674973488, + 0.024560648947954178, + 0.026881994679570198, + 0.025268739089369774, + 0.027781862765550613, + 0.020820777863264084, + 0.02243473008275032, + 0.02128320187330246, + 0.022841596975922585, + 0.02157461643218994, + 0.023407133296132088, + 0.02214578352868557, + 0.023906925693154335, + 0.018881559371948242, + 0.019972745329141617, + 0.019260240718722343, + 0.020448695868253708, + 0.020204225555062294, + 0.021090438589453697, + 0.020976155996322632, + 0.021342279389500618, + 0.018290864303708076, + 0.019086189568042755, + 0.019081290811300278, + 0.020024435594677925, + 0.020085478201508522, + 0.021220820024609566, + 0.02111699804663658, + 0.022582143545150757, + 0.019801219925284386, + 0.02100391499698162, + 0.021013697609305382, + 0.022442955523729324, + 0.022457340732216835, + 0.02410208247601986, + 0.024221593514084816, + 0.025193551555275917, + 0.021587364375591278, + 0.02587711811065674, + 0.02333785593509674, + 0.027406372129917145, + 0.02415860816836357, + 0.028257466852664948, + 0.02561909332871437 + ], + "loss_nats": 2.133716106414795, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 2.174569606781006, + "mean_weights": [ + 0.12282125651836395, + 0.09209376573562622, + 0.09000569581985474, + 0.09502357244491577, + 0.09669246524572372, + 0.10431371629238129, + 0.12377016246318817, + 0.1270047426223755, + 0.14827463030815125 + ], + "sources": 9 + }, + "step": 500, + "stream_state_rms_by_sublayer": [ + 0.010510989464819431, + 0.021690133959054947, + 0.02805406041443348, + 0.038645509630441666, + 0.04236667975783348, + 0.050416164100170135, + 0.05213770642876625, + 0.059038665145635605, + 0.012978821992874146, + 0.015957212075591087, + 0.02230519987642765, + 0.0259401835501194, + 0.029941538348793983, + 0.03326375409960747, + 0.03444092720746994, + 0.03896600753068924, + 0.012748918496072292, + 0.01569177769124508, + 0.02370862103998661, + 0.02718697488307953, + 0.03030574880540371, + 0.03437623754143715, + 0.034412682056427, + 0.03927858546376228, + 0.014515543356537819, + 0.0174946840852499, + 0.02315252088010311, + 0.027429770678281784, + 0.03182410076260567, + 0.036497704684734344, + 0.03731905296444893, + 0.042413465678691864, + 0.01630145125091076, + 0.01978003978729248, + 0.023382632061839104, + 0.02902580052614212, + 0.035387299954891205, + 0.04184097424149513, + 0.04106847196817398, + 0.049806781113147736, + 0.015464341267943382, + 0.02072436735033989, + 0.026616448536515236, + 0.035371601581573486, + 0.03925501927733421, + 0.05015818774700165, + 0.05158501863479614, + 0.06557824462652206, + 0.016100121662020683, + 0.024349043145775795, + 0.029167983680963516, + 0.0433179996907711, + 0.04585959017276764, + 0.06327643990516663, + 0.06569604575634003, + 0.08553080260753632, + 0.007519193924963474, + 0.032196998596191406, + 0.03536655753850937, + 0.06154516339302063, + 0.06592655926942825, + 0.0982988178730011, + 0.10538331419229507, + 0.1370837539434433 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.912946603028104e-05, + 8.98468351806514e-05, + 8.289486868306994e-05, + 7.69269245211035e-05, + 8.701786282472312e-05, + 8.263266499852762e-05, + 7.924547389848158e-05, + 7.185961294453591e-05, + 6.937592843314633e-05, + 6.650048453593627e-05, + 6.397806282620877e-05, + 6.214729364728555e-05, + 6.411123467842117e-05, + 6.199828203534707e-05, + 5.916267400607467e-05, + 5.662695912178606e-05, + 5.848028013133444e-05, + 5.5864726164145395e-05, + 5.7232024119002745e-05, + 5.329703708412126e-05, + 6.049815419828519e-05, + 5.729254553443752e-05, + 5.153644451638684e-05, + 4.557902502710931e-05, + 5.965012314845808e-05, + 5.673339182976633e-05, + 5.410864105215296e-05, + 5.2005798352183774e-05, + 4.5126816985430196e-05, + 3.954961721319705e-05, + 3.792164716287516e-05, + 3.6557423300109804e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 8.119421363517176e-05, + "first_to_last_ratio": 1.7019463434397786, + "imbalance_abs_log_ratio": 0.5317725040606909, + "last_quartile_mean": 4.770668238052167e-05, + "mean": 6.1715260130768e-05, + "normalized": [ + 1.282170177402059, + 1.4558285096793826, + 1.3431826829770244, + 1.2464814108877387, + 1.4099894035987472, + 1.3389340792445479, + 1.2840499048463692, + 1.1643734919414277, + 1.1241292394481721, + 1.077537134171174, + 1.0366652055042163, + 1.007000432560798, + 1.038823048668617, + 1.0045859306754825, + 0.9586393038077669, + 0.9175519798798485, + 0.9475821702350604, + 0.9052011778897804, + 0.9273561190171481, + 0.863595761748238, + 0.9802786874769078, + 0.9283367746168578, + 0.8350680918655556, + 0.7385373557614803, + 0.9665376605731854, + 0.9192765567147311, + 0.8767465443312168, + 0.8426732422741001, + 0.731210026334027, + 0.6408401605923019, + 0.6144614327562302, + 0.5923563025198072 + ], + "population_cv": 0.22289232151468405 + }, + "activation_output_rms_by_block": [ + 0.07933542132377625, + 0.17356063425540924, + 0.19592563807964325, + 0.20413769781589508, + 0.0703900009393692, + 0.08005841821432114, + 0.0889945924282074, + 0.10390866547822952, + 0.043775007128715515, + 0.06786487251520157, + 0.09043840318918228, + 0.11542254686355591, + 0.03245418146252632, + 0.05422379821538925, + 0.0745156928896904, + 0.09330973774194717, + 0.03882657364010811, + 0.06532005220651627, + 0.0864439532160759, + 0.11046812683343887, + 0.0477026104927063, + 0.08639359474182129, + 0.11987065523862839, + 0.16759569942951202, + 0.06834843754768372, + 0.10255371779203415, + 0.1335299015045166, + 0.16770142316818237, + 0.060653913766145706, + 0.1185198500752449, + 0.19719740748405457, + 0.280917763710022 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.12453888356685638, + "first_to_last_ratio": 0.8821421066737113, + "imbalance_abs_log_ratio": 0.12540211727588868, + "last_quartile_mean": 0.1411778018809855, + "mean": 0.10688621841836721, + "normalized": [ + 0.7422418202994759, + 1.6237887056315277, + 1.8330299357468485, + 1.909859856926291, + 0.65855076529935, + 0.7490059934664504, + 0.8326105436705642, + 0.972142779638034, + 0.4095477207115157, + 0.6349263124790253, + 0.8461184662291453, + 1.0798636959142511, + 0.3036329841467142, + 0.5073039257797476, + 0.6971496792788181, + 0.8729819346468051, + 0.3632514482656278, + 0.6111176274460819, + 0.8087474184718791, + 1.033511414923967, + 0.4462933687670733, + 0.8082762775240584, + 1.121479055133547, + 1.5679823078174555, + 0.6394504226930232, + 0.9594662371777898, + 1.2492714540789758, + 1.568971431955579, + 0.5674624349487043, + 1.1088412690524991, + 1.844928283571574, + 2.628194428307601 + ], + "population_cv": 0.5270692473404438 + }, + "bits_per_byte": 2.066011762411041, + "branch_output_rms_by_sublayer": [ + 0.04519475996494293, + 0.06201421841979027, + 0.11058512330055237, + 0.024957187473773956, + 0.04467621445655823, + 0.007390463724732399, + 0.03156944364309311, + 0.015626417472958565, + 0.06896122545003891, + 0.009649131447076797, + 0.02159368060529232, + 0.00823317188769579, + 0.02192658744752407, + 0.01007626112550497, + 0.03447022661566734, + 0.017795119434595108, + 0.039527297019958496, + 0.016687674447894096, + 0.027605466544628143, + 0.01717737689614296, + 0.024575263261795044, + 0.01972619630396366, + 0.02325247973203659, + 0.019825972616672516, + 0.02363201417028904, + 0.01986200176179409, + 0.02095704898238182, + 0.022178664803504944, + 0.02059737965464592, + 0.02332259714603424, + 0.019563060253858566, + 0.025621045380830765, + 0.027532115578651428, + 0.02510117180645466, + 0.030061373487114906, + 0.027030576020479202, + 0.02861795388162136, + 0.02360380068421364, + 0.0402015745639801, + 0.030928917229175568, + 0.043986596167087555, + 0.023246120661497116, + 0.06151795759797096, + 0.023922353982925415, + 0.06635378301143646, + 0.030496269464492798, + 0.07754725962877274, + 0.05318857729434967, + 0.04700809717178345, + 0.04513707384467125, + 0.0281364768743515, + 0.04732036218047142, + 0.01650506816804409, + 0.05235907435417175, + 0.011390995234251022, + 0.060677725821733475, + 0.013558072037994862, + 0.05817825347185135, + 0.010303095914423466, + 0.06757662445306778, + 0.0185127891600132, + 0.0921226441860199, + 0.017877288162708282, + 0.09247501194477081 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0002499867152056645, + 0.00022023184426304317, + 0.00012862347080347888, + 0.00013874508711477122, + 0.00016561428986663383, + 0.00010237510824701164, + 0.00011183016649318567, + 0.00018676622342607996, + 0.00015642584072286578, + 0.0001222540107587513, + 0.00012523380872062526, + 0.00012743167233473566, + 0.00013086282613597257, + 0.00013887000222485716, + 0.0001429926281348834, + 0.00015194658148812596, + 0.0001557433490681304, + 0.00016747093812728608, + 0.00015864942110219157, + 0.00019577301560792646, + 0.00019437764302629505, + 0.00022437945703715723, + 0.00027218714798262134, + 0.0002948071371943306, + 0.00028787668401285504, + 0.00025299422855138884, + 0.0002433303889943037, + 0.00023362869890304787, + 0.00021691172073067734, + 0.0001988072477910395, + 0.00019662747032228586, + 0.00019539784029560395 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.0001630216131774836, + "first_to_last_ratio": 0.7143904907034329, + "imbalance_abs_log_ratio": 0.3363255603938907, + "last_quartile_mean": 0.00022819678495015027, + "mean": 0.00018403602077149458, + "normalized": [ + 1.358357533257343, + 1.1966779293521597, + 0.6989037812504226, + 0.7539017988605713, + 0.8999014930466586, + 0.5562775581532708, + 0.6076536866227826, + 1.0148351537005644, + 0.849974043489505, + 0.6642939259730358, + 0.6804853104062701, + 0.6924278834139714, + 0.7110718085915166, + 0.7545805524521904, + 0.7769817426797547, + 0.8256350080334982, + 0.8462655757021973, + 0.9099899977473634, + 0.8620563541697966, + 1.063775530394699, + 1.0561934680583047, + 1.2192148911747795, + 1.4789884439013068, + 1.6018991062644918, + 1.5642409719904375, + 1.3746995152949713, + 1.3221889278753263, + 1.269472671293677, + 1.1786373114424287, + 1.080262695083401, + 1.0684183971051258, + 1.06173693321818 + ], + "population_cv": 0.28586106954832347 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.4698029160499573, + "mean_weights": [ + 0.8109943866729736, + 0.18900558352470398 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6435889005661011, + "mean_weights": [ + 0.5100059509277344, + 0.4899940490722656 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5176190733909607, + "mean_weights": [ + 0.7797738909721375, + 0.22022610902786255 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6866215467453003, + "mean_weights": [ + 0.4749618172645569, + 0.5250381827354431 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6593912839889526, + "mean_weights": [ + 0.55280601978302, + 0.4471939504146576 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6897172927856445, + "mean_weights": [ + 0.5164542198181152, + 0.48354578018188477 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6356666684150696, + "mean_weights": [ + 0.6334895491600037, + 0.36651045083999634 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6826819777488708, + "mean_weights": [ + 0.5319589376449585, + 0.4680410921573639 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0528697967529297, + "mean_weights": [ + 0.35035771131515503, + 0.23919537663459778, + 0.4104468822479248 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0909645557403564, + "mean_weights": [ + 0.3614423871040344, + 0.28591710329055786, + 0.3526404798030853 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0500558614730835, + "mean_weights": [ + 0.3499464988708496, + 0.22412750124931335, + 0.4259260296821594 + ], + "sources": 3 + }, + { + "entropy_mean": 1.086782693862915, + "mean_weights": [ + 0.3487471342086792, + 0.2704853415489197, + 0.3807675242424011 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0477923154830933, + "mean_weights": [ + 0.29735344648361206, + 0.26350510120391846, + 0.4391414523124695 + ], + "sources": 3 + }, + { + "entropy_mean": 1.083518147468567, + "mean_weights": [ + 0.3840525150299072, + 0.25850173830986023, + 0.35744574666023254 + ], + "sources": 3 + }, + { + "entropy_mean": 1.041725993156433, + "mean_weights": [ + 0.3748757243156433, + 0.20431557297706604, + 0.42080870270729065 + ], + "sources": 3 + }, + { + "entropy_mean": 1.088074803352356, + "mean_weights": [ + 0.37115857005119324, + 0.27379852533340454, + 0.35504287481307983 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3461657762527466, + "mean_weights": [ + 0.24980032444000244, + 0.17032209038734436, + 0.26108914613723755, + 0.31878846883773804 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3758656978607178, + "mean_weights": [ + 0.28765738010406494, + 0.2253328114748001, + 0.21813109517097473, + 0.2688787281513214 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3665999174118042, + "mean_weights": [ + 0.2572351098060608, + 0.2049572765827179, + 0.26986706256866455, + 0.26794055104255676 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3734302520751953, + "mean_weights": [ + 0.2780992090702057, + 0.21354396641254425, + 0.21350377798080444, + 0.2948530316352844 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3616993427276611, + "mean_weights": [ + 0.3108273148536682, + 0.20910179615020752, + 0.2097688913345337, + 0.2703019976615906 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3674063682556152, + "mean_weights": [ + 0.29994261264801025, + 0.20192782580852509, + 0.21637186408042908, + 0.2817577123641968 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3652420043945312, + "mean_weights": [ + 0.31152600049972534, + 0.20496675372123718, + 0.22167494893074036, + 0.2618322968482971 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3603029251098633, + "mean_weights": [ + 0.2523220181465149, + 0.19280630350112915, + 0.22273331880569458, + 0.33213838934898376 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5933597087860107, + "mean_weights": [ + 0.23010152578353882, + 0.1523759663105011, + 0.1853415071964264, + 0.21938833594322205, + 0.21279269456863403 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5951242446899414, + "mean_weights": [ + 0.20269069075584412, + 0.16687172651290894, + 0.1650846004486084, + 0.22331196069717407, + 0.24204102158546448 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6055580377578735, + "mean_weights": [ + 0.19624704122543335, + 0.18407827615737915, + 0.19682693481445312, + 0.21266505122184753, + 0.21018269658088684 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5947179794311523, + "mean_weights": [ + 0.19966183602809906, + 0.18184176087379456, + 0.15833750367164612, + 0.2062276005744934, + 0.25393128395080566 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5722358226776123, + "mean_weights": [ + 0.17954319715499878, + 0.15006548166275024, + 0.1787385493516922, + 0.22400569915771484, + 0.2676470875740051 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5913262367248535, + "mean_weights": [ + 0.19785037636756897, + 0.18112456798553467, + 0.15975478291511536, + 0.19423411786556244, + 0.2670361399650574 + ], + "sources": 5 + }, + { + "entropy_mean": 1.568877935409546, + "mean_weights": [ + 0.18879836797714233, + 0.15164746344089508, + 0.17191189527511597, + 0.2049403190612793, + 0.2827019691467285 + ], + "sources": 5 + }, + { + "entropy_mean": 1.573760747909546, + "mean_weights": [ + 0.19746656715869904, + 0.16800090670585632, + 0.13806992769241333, + 0.21452516317367554, + 0.28193745017051697 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7648463249206543, + "mean_weights": [ + 0.14480476081371307, + 0.13240447640419006, + 0.16316933929920197, + 0.17283213138580322, + 0.22918567061424255, + 0.15760360658168793 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7635750770568848, + "mean_weights": [ + 0.16815733909606934, + 0.14792580902576447, + 0.11194154620170593, + 0.14750641584396362, + 0.20471270382404327, + 0.21975621581077576 + ], + "sources": 6 + }, + { + "entropy_mean": 1.763267159461975, + "mean_weights": [ + 0.13464021682739258, + 0.13171157240867615, + 0.1645362675189972, + 0.15497304499149323, + 0.2008623480796814, + 0.21327653527259827 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7614943981170654, + "mean_weights": [ + 0.15073809027671814, + 0.14079315960407257, + 0.11966855823993683, + 0.16534580290317535, + 0.18966889381408691, + 0.2337855100631714 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7334396839141846, + "mean_weights": [ + 0.12032384425401688, + 0.125158429145813, + 0.1547280251979828, + 0.15251007676124573, + 0.1917954981327057, + 0.25548410415649414 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7752734422683716, + "mean_weights": [ + 0.16270102560520172, + 0.14989414811134338, + 0.12619557976722717, + 0.1607460379600525, + 0.18175114691257477, + 0.21871207654476166 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7268621921539307, + "mean_weights": [ + 0.12063982337713242, + 0.13393959403038025, + 0.16096177697181702, + 0.14662346243858337, + 0.17392438650131226, + 0.26391100883483887 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0249087810516357, + "mean_weights": [ + 0.10775190591812134, + 0.12611693143844604, + 0.1291588544845581, + 0.13433250784873962, + 0.11416041851043701, + 0.10145977884531021, + 0.10813163220882416, + 0.1788879632949829 + ], + "sources": 8 + }, + { + "entropy_mean": 2.1254401206970215, + "mean_weights": [ + 0.08790647983551025, + 0.11165842413902283, + 0.11382804811000824, + 0.10206783562898636, + 0.09298259764909744, + 0.08998951315879822, + 0.1232423186302185, + 0.20247796177864075, + 0.07584682106971741 + ], + "sources": 9 + }, + { + "entropy_mean": 2.164797782897949, + "mean_weights": [ + 0.09679380059242249, + 0.10930618643760681, + 0.10940437018871307, + 0.1065194308757782, + 0.10231906175613403, + 0.0894792377948761, + 0.09792773425579071, + 0.15008091926574707, + 0.13816925883293152 + ], + "sources": 9 + }, + { + "entropy_mean": 2.097749948501587, + "mean_weights": [ + 0.08595120906829834, + 0.11955319344997406, + 0.1062607616186142, + 0.09907947480678558, + 0.07950928807258606, + 0.07379093766212463, + 0.1002209484577179, + 0.20458786189556122, + 0.131046324968338 + ], + "sources": 9 + }, + { + "entropy_mean": 2.149176597595215, + "mean_weights": [ + 0.09992527961730957, + 0.11369793117046356, + 0.12063031643629074, + 0.11137115955352783, + 0.09962157905101776, + 0.0898594781756401, + 0.09756124764680862, + 0.14056888222694397, + 0.12676411867141724 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1050100326538086, + "mean_weights": [ + 0.076517753303051, + 0.1342213898897171, + 0.1254764199256897, + 0.12790073454380035, + 0.0898401141166687, + 0.09525885432958603, + 0.1059754490852356, + 0.16228562593460083, + 0.08252367377281189 + ], + "sources": 9 + }, + { + "entropy_mean": 2.140446662902832, + "mean_weights": [ + 0.11036726832389832, + 0.12699949741363525, + 0.11147990077733994, + 0.12464220076799393, + 0.11014757305383682, + 0.10495586693286896, + 0.08858617395162582, + 0.10959435999393463, + 0.11322715878486633 + ], + "sources": 9 + }, + { + "entropy_mean": 2.140577793121338, + "mean_weights": [ + 0.08426377177238464, + 0.12632998824119568, + 0.1283624768257141, + 0.11601665616035461, + 0.09646609425544739, + 0.08777236193418503, + 0.10991427302360535, + 0.15994463860988617, + 0.09092973917722702 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05581815540790558, + 0.04701259359717369, + 0.05504828691482544, + 0.06247949227690697, + 0.10042016953229904, + 0.09893567860126495, + 0.10539961606264114, + 0.08913368731737137, + 0.10551442950963974, + 0.0704219788312912, + 0.07794100791215897, + 0.07123440504074097, + 0.07851417362689972, + 0.07974004000425339, + 0.07635531574487686, + 0.07136041671037674, + 0.07898814976215363, + 0.051502104848623276, + 0.05948945879936218, + 0.05734645202755928, + 0.05630601570010185, + 0.054764918982982635, + 0.05502825230360031, + 0.054255250841379166, + 0.057338930666446686, + 0.04283996671438217, + 0.04555618762969971, + 0.04723348096013069, + 0.04675706475973129, + 0.04486935958266258, + 0.04799116402864456, + 0.04580603912472725, + 0.0505371131002903, + 0.041227150708436966, + 0.04136831313371658, + 0.04064388945698738, + 0.04442180320620537, + 0.043157923966646194, + 0.04518991708755493, + 0.045875295996665955, + 0.04627705737948418, + 0.04033881053328514, + 0.04083601385354996, + 0.041378311812877655, + 0.04169000685214996, + 0.041933998465538025, + 0.04239153489470482, + 0.04077022150158882, + 0.04231491684913635, + 0.03644668683409691, + 0.0375138483941555, + 0.037383291870355606, + 0.038913119584321976, + 0.03884527459740639, + 0.04077959060668945, + 0.0407528392970562, + 0.04941263794898987, + 0.048238202929496765, + 0.04445720463991165, + 0.05065429210662842, + 0.049143001437187195, + 0.04975641518831253, + 0.05452987924218178, + 0.0528714694082737 + ], + "loss_nats": 1.4320502281188965, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.933801293373108, + "mean_weights": [ + 0.11528845131397247, + 0.09230738878250122, + 0.0814325213432312, + 0.06491375714540482, + 0.06580930203199387, + 0.05624303221702576, + 0.05220801383256912, + 0.15835918486118317, + 0.313438355922699 + ], + "sources": 9 + }, + "step": 2000, + "stream_state_rms_by_sublayer": [ + 0.04519475996494293, + 0.07933542132377625, + 0.1634819060564041, + 0.17356063425540924, + 0.19396348297595978, + 0.19592563807964325, + 0.19977088272571564, + 0.20413769781589508, + 0.06896122545003891, + 0.0703900009393692, + 0.07872548699378967, + 0.08005841821432114, + 0.0875031054019928, + 0.0889945924282074, + 0.10048835724592209, + 0.10390866547822952, + 0.039527297019958496, + 0.043775007128715515, + 0.061073824763298035, + 0.06786487251520157, + 0.08160728216171265, + 0.09043840318918228, + 0.10582270473241806, + 0.11542254686355591, + 0.02363201417028904, + 0.03245418146252632, + 0.04137130454182625, + 0.05422379821538925, + 0.06148746609687805, + 0.0745156928896904, + 0.07912598550319672, + 0.09330973774194717, + 0.027532115578651428, + 0.03882657364010811, + 0.051289595663547516, + 0.06532005220651627, + 0.0751892700791359, + 0.0864439532160759, + 0.09719052165746689, + 0.11046812683343887, + 0.043986596167087555, + 0.0477026104927063, + 0.08199729770421982, + 0.08639359474182129, + 0.1128886342048645, + 0.11987065523862839, + 0.15118840336799622, + 0.16759569942951202, + 0.04700809717178345, + 0.06834843754768372, + 0.07857025414705276, + 0.10255371779203415, + 0.10558044165372849, + 0.1335299015045166, + 0.13421936333179474, + 0.16770142316818237, + 0.013558072037994862, + 0.060653913766145706, + 0.06223326548933983, + 0.1185198500752449, + 0.12151430547237396, + 0.19719740748405457, + 0.2012815624475479, + 0.280917763710022 + ] + }, + { + "activation_grad_rms_by_block": [ + 7.722850568825379e-05, + 8.501399133820087e-05, + 7.900782657088712e-05, + 7.458877371391281e-05, + 8.851825259625912e-05, + 8.496154623571783e-05, + 8.129017805913463e-05, + 7.308541535167024e-05, + 7.228730828501284e-05, + 6.910465162945911e-05, + 6.638290506089106e-05, + 6.404728628695011e-05, + 7.102679228410125e-05, + 6.824541924288496e-05, + 6.386093446053565e-05, + 6.002810187055729e-05, + 6.220424256753176e-05, + 5.8480411098571494e-05, + 5.7781468058237806e-05, + 5.275792864267714e-05, + 5.794492972199805e-05, + 5.5180109484354034e-05, + 4.93703773827292e-05, + 4.439146141521633e-05, + 6.059826409909874e-05, + 5.768534538219683e-05, + 5.5251708545256406e-05, + 5.3433937864610925e-05, + 5.2880252042086795e-05, + 4.163408812019043e-05, + 3.907483187504113e-05, + 3.544480205164291e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 8.046181119425455e-05, + "first_to_last_ratio": 1.6254778770020237, + "imbalance_abs_log_ratio": 0.4858018507042873, + "last_quartile_mean": 4.950040374751552e-05, + "mean": 6.28997514695584e-05, + "normalized": [ + 1.2278030339377428, + 1.3515791295192177, + 1.2560912360540017, + 1.18583574610802, + 1.4072909753721254, + 1.3507453408116674, + 1.2923767766948435, + 1.1619348827958629, + 1.1492463260367212, + 1.09864745114142, + 1.0553762695392908, + 1.018243868864046, + 1.1292062468398798, + 1.0849871048522937, + 1.0152811890114135, + 0.954345612949029, + 0.988942581078985, + 0.9297399390659639, + 0.9186279231357903, + 0.838762116066713, + 0.9212266879948113, + 0.8772707076761593, + 0.7849057624118434, + 0.7057493929320289, + 0.9634102310948954, + 0.9170997346486308, + 0.8784090120291904, + 0.8495095229505216, + 0.8407068518812711, + 0.6619118064455324, + 0.6212239470286648, + 0.563512593031423 + ], + "population_cv": 0.21279267414040023 + }, + "activation_output_rms_by_block": [ + 0.11307977139949799, + 0.24579407274723053, + 0.2624119520187378, + 0.2694927752017975, + 0.08423247188329697, + 0.09299755096435547, + 0.1023314967751503, + 0.11912773549556732, + 0.047540031373500824, + 0.07420121878385544, + 0.10375737398862839, + 0.13788114488124847, + 0.03878077492117882, + 0.06997464597225189, + 0.0936177521944046, + 0.1194845587015152, + 0.04525832086801529, + 0.07861223816871643, + 0.10294434428215027, + 0.13677391409873962, + 0.06066577881574631, + 0.10652344673871994, + 0.15397457778453827, + 0.2117626667022705, + 0.0858822911977768, + 0.13233982026576996, + 0.1727868914604187, + 0.21057960391044617, + 0.07290095835924149, + 0.13914754986763, + 0.22899514436721802, + 0.3399814963340759 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.16118347831070423, + "first_to_last_ratio": 0.9326305492848443, + "imbalance_abs_log_ratio": 0.06974613803006748, + "last_quartile_mean": 0.17282671947032213, + "mean": 0.13293232407886535, + "normalized": [ + 0.8506566945479013, + 1.8490165913401708, + 1.9740266623417755, + 2.0272930385382737, + 0.6336492833249797, + 0.6995856847367116, + 0.7698014571267079, + 0.8961532593448971, + 0.35762581977650926, + 0.5581879298208444, + 0.7805277964377708, + 1.037228122179276, + 0.2917332198162064, + 0.5263930083005073, + 0.7042512259009598, + 0.8988375064489812, + 0.3404613677044019, + 0.5913703737104438, + 0.7744116789914545, + 1.028898840417438, + 0.45636589321762594, + 0.8013359239512152, + 1.1582929799165258, + 1.593011091693824, + 0.6460602550424457, + 0.9955428161118746, + 1.2998109576261425, + 1.5841113541768417, + 0.5484065584830309, + 1.0467548117572767, + 1.7226445558219614, + 2.5575532413950244 + ], + "population_cv": 0.5475104567779164 + }, + "bits_per_byte": 1.94431735460847, + "branch_output_rms_by_sublayer": [ + 0.06706089526414871, + 0.08734722435474396, + 0.16490890085697174, + 0.01817459426820278, + 0.05402721092104912, + 0.006570694502443075, + 0.03852631151676178, + 0.02017863281071186, + 0.08292478322982788, + 0.007697225548326969, + 0.027649108320474625, + 0.00738769443705678, + 0.027229653671383858, + 0.010129167698323727, + 0.04429584741592407, + 0.02217743918299675, + 0.04215651750564575, + 0.020272625610232353, + 0.029113495722413063, + 0.02010158821940422, + 0.028859032317996025, + 0.02570836804807186, + 0.03370937705039978, + 0.025632938370108604, + 0.0246309582144022, + 0.027024641633033752, + 0.025226473808288574, + 0.029663097113370895, + 0.022274626418948174, + 0.032867297530174255, + 0.025765124708414078, + 0.03647798299789429, + 0.02585558034479618, + 0.036331918090581894, + 0.035017866641283035, + 0.03689613565802574, + 0.03337937593460083, + 0.034316401928663254, + 0.05114835500717163, + 0.046151019632816315, + 0.05665922909975052, + 0.030194124206900597, + 0.07309310138225555, + 0.034518927335739136, + 0.08364450931549072, + 0.046253010630607605, + 0.08640947192907333, + 0.07801470160484314, + 0.05644892156124115, + 0.061151664704084396, + 0.04434261471033096, + 0.06492369621992111, + 0.03316858038306236, + 0.06979183107614517, + 0.014417458325624466, + 0.07597717642784119, + 0.016864154487848282, + 0.07042016088962555, + 0.013014601543545723, + 0.08321022987365723, + 0.01720009371638298, + 0.1123834028840065, + 0.02818106859922409, + 0.1276838332414627 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.00028301765084205076, + 0.00018640727245129113, + 0.00010306090203068865, + 0.00014222708269657445, + 0.00015838688665849668, + 9.970135661519603e-05, + 0.00012138718434767938, + 0.00021048722070999366, + 0.00017274196027943569, + 0.00013106145358904433, + 0.00013638371514977, + 0.0001547949615264904, + 0.00016587079599548126, + 0.00017931423985143718, + 0.00019278834870391096, + 0.00020784376871445687, + 0.00020123613260398355, + 0.00021726968219813378, + 0.0002071508396997652, + 0.0002559233234473849, + 0.0002295854626557122, + 0.0002641860573878995, + 0.0003281630733843104, + 0.0003582708629313403, + 0.000355412897731539, + 0.00032680906284868124, + 0.0003121616071976194, + 0.00029556208910195224, + 0.0002908363635073086, + 0.0002515380376246955, + 0.00023503421639413057, + 0.00023619860105820258 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00016308444454399634, + "first_to_last_ratio": 0.5663753457749909, + "imbalance_abs_log_ratio": 0.56849826548963, + "last_quartile_mean": 0.00028794410943301614, + "mean": 0.000219087909685458, + "normalized": [ + 1.291799493857858, + 0.8508332236083402, + 0.47040889740858816, + 0.6491781445209288, + 0.7229375956249293, + 0.45507466276133685, + 0.5540569743074986, + 0.960743205830882, + 0.7884595755532076, + 0.5982139944518515, + 0.622506807178792, + 0.7065426921491367, + 0.7570969855599063, + 0.8184579427905291, + 0.8799588666517243, + 0.9486774921211115, + 0.9185177442830869, + 0.9917009227486143, + 0.9455147022816972, + 1.1681307463054946, + 1.0479147981526018, + 1.2058449860021414, + 1.4978602600912592, + 1.6352835875138234, + 1.6222387544881012, + 1.4916800444071845, + 1.4248235224197732, + 1.3490570498677328, + 1.3274870526851938, + 1.1481146448739494, + 1.0727849689723477, + 1.0780996605303788 + ], + "population_cv": 0.3276215791506264 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.298600435256958, + "mean_weights": [ + 0.9062461256980896, + 0.09375390410423279 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6085544228553772, + "mean_weights": [ + 0.5902345180511475, + 0.40976545214653015 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5791133642196655, + "mean_weights": [ + 0.7206420302391052, + 0.2793579697608948 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6809916496276855, + "mean_weights": [ + 0.4366615414619446, + 0.5633384585380554 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6643537878990173, + "mean_weights": [ + 0.43601688742637634, + 0.5639830827713013 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6867029070854187, + "mean_weights": [ + 0.5107938051223755, + 0.4892061948776245 + ], + "sources": 2 + }, + { + "entropy_mean": 0.630885124206543, + "mean_weights": [ + 0.6297253370285034, + 0.3702746331691742 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6814190149307251, + "mean_weights": [ + 0.5384589433670044, + 0.461541086435318 + ], + "sources": 2 + }, + { + "entropy_mean": 1.0193029642105103, + "mean_weights": [ + 0.2793715000152588, + 0.22541894018650055, + 0.49520957469940186 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0885753631591797, + "mean_weights": [ + 0.3747631013393402, + 0.27809539437294006, + 0.3471415042877197 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0166544914245605, + "mean_weights": [ + 0.29693084955215454, + 0.2092972993850708, + 0.49377185106277466 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0730276107788086, + "mean_weights": [ + 0.3801417946815491, + 0.2446514070034027, + 0.3752068281173706 + ], + "sources": 3 + }, + { + "entropy_mean": 1.01689612865448, + "mean_weights": [ + 0.25329312682151794, + 0.25301480293273926, + 0.4936920702457428 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0735528469085693, + "mean_weights": [ + 0.4058016240596771, + 0.23908020555973053, + 0.35511815547943115 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0387647151947021, + "mean_weights": [ + 0.3897400498390198, + 0.19914782047271729, + 0.41111212968826294 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0789580345153809, + "mean_weights": [ + 0.4085901975631714, + 0.2620368003845215, + 0.3293730318546295 + ], + "sources": 3 + }, + { + "entropy_mean": 1.330599308013916, + "mean_weights": [ + 0.25999969244003296, + 0.1543980836868286, + 0.24968576431274414, + 0.3359164595603943 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3653013706207275, + "mean_weights": [ + 0.3057574927806854, + 0.2060212939977646, + 0.20908181369304657, + 0.2791393995285034 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3513377904891968, + "mean_weights": [ + 0.2617591619491577, + 0.19308003783226013, + 0.2539125084877014, + 0.29124826192855835 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3677287101745605, + "mean_weights": [ + 0.29450443387031555, + 0.20663464069366455, + 0.20358902215957642, + 0.2952719032764435 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3450267314910889, + "mean_weights": [ + 0.3230239152908325, + 0.18811741471290588, + 0.2109987735748291, + 0.2778598666191101 + ], + "sources": 4 + }, + { + "entropy_mean": 1.348590612411499, + "mean_weights": [ + 0.3295133709907532, + 0.18519240617752075, + 0.19538277387619019, + 0.2899114787578583 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3515987396240234, + "mean_weights": [ + 0.33015355467796326, + 0.19018611311912537, + 0.21045629680156708, + 0.2692040503025055 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3623924255371094, + "mean_weights": [ + 0.2646353542804718, + 0.19441348314285278, + 0.2166558802127838, + 0.3242952823638916 + ], + "sources": 4 + }, + { + "entropy_mean": 1.5844863653182983, + "mean_weights": [ + 0.23668400943279266, + 0.13988922536373138, + 0.18485453724861145, + 0.23012810945510864, + 0.20844411849975586 + ], + "sources": 5 + }, + { + "entropy_mean": 1.593141794204712, + "mean_weights": [ + 0.20686954259872437, + 0.17310559749603271, + 0.1597880721092224, + 0.2032153457403183, + 0.257021427154541 + ], + "sources": 5 + }, + { + "entropy_mean": 1.6000027656555176, + "mean_weights": [ + 0.18989244103431702, + 0.17239899933338165, + 0.1975756287574768, + 0.20976364612579346, + 0.23036929965019226 + ], + "sources": 5 + }, + { + "entropy_mean": 1.582854986190796, + "mean_weights": [ + 0.19638973474502563, + 0.1686723232269287, + 0.15907755494117737, + 0.19003787636756897, + 0.2858225107192993 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5503523349761963, + "mean_weights": [ + 0.17854918539524078, + 0.139242485165596, + 0.17552749812602997, + 0.2097170352935791, + 0.29696381092071533 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5834622383117676, + "mean_weights": [ + 0.19760790467262268, + 0.17277662456035614, + 0.15623365342617035, + 0.19005371630191803, + 0.283328115940094 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5523425340652466, + "mean_weights": [ + 0.191162571310997, + 0.13771341741085052, + 0.18561774492263794, + 0.18527957797050476, + 0.30022668838500977 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5726368427276611, + "mean_weights": [ + 0.20087841153144836, + 0.16328519582748413, + 0.1421341896057129, + 0.19957204163074493, + 0.2941301763057709 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7497403621673584, + "mean_weights": [ + 0.14710332453250885, + 0.11305859684944153, + 0.15883512794971466, + 0.16542699933052063, + 0.2511061429977417, + 0.16446979343891144 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7714827060699463, + "mean_weights": [ + 0.17014864087104797, + 0.14715781807899475, + 0.12150244414806366, + 0.1485304832458496, + 0.20396047830581665, + 0.20870016515254974 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7444989681243896, + "mean_weights": [ + 0.13433504104614258, + 0.11546684801578522, + 0.15696288645267487, + 0.141643226146698, + 0.2109500616788864, + 0.24064195156097412 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7677668333053589, + "mean_weights": [ + 0.15450707077980042, + 0.1484912931919098, + 0.12180650234222412, + 0.162429541349411, + 0.18306714296340942, + 0.22969844937324524 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6891989707946777, + "mean_weights": [ + 0.11655238270759583, + 0.10491175204515457, + 0.1503290832042694, + 0.1321149468421936, + 0.194725900888443, + 0.3013659715652466 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7821013927459717, + "mean_weights": [ + 0.1637197732925415, + 0.1602180004119873, + 0.1371064931154251, + 0.15685361623764038, + 0.17574524879455566, + 0.20635688304901123 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7014548778533936, + "mean_weights": [ + 0.12046012282371521, + 0.11536401510238647, + 0.15961314737796783, + 0.13822567462921143, + 0.1780020147562027, + 0.28833502531051636 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.021324634552002, + "mean_weights": [ + 0.10935264825820923, + 0.12541475892066956, + 0.13264349102973938, + 0.15386636555194855, + 0.11865094304084778, + 0.10020800679922104, + 0.10042205452919006, + 0.159441739320755 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0959906578063965, + "mean_weights": [ + 0.08627711981534958, + 0.10818275809288025, + 0.10734419524669647, + 0.09354669600725174, + 0.09311383217573166, + 0.08255733549594879, + 0.10319340229034424, + 0.20966464281082153, + 0.11612001806497574 + ], + "sources": 9 + }, + { + "entropy_mean": 2.150697708129883, + "mean_weights": [ + 0.08980904519557953, + 0.10636033117771149, + 0.12191382050514221, + 0.11199057847261429, + 0.09992339462041855, + 0.08958576619625092, + 0.09268327802419662, + 0.13146400451660156, + 0.15626978874206543 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0447325706481934, + "mean_weights": [ + 0.08423449099063873, + 0.10816886276006699, + 0.1022123247385025, + 0.09324942529201508, + 0.07462728023529053, + 0.06389737129211426, + 0.08259779959917068, + 0.20694155991077423, + 0.1840709000825882 + ], + "sources": 9 + }, + { + "entropy_mean": 2.142486333847046, + "mean_weights": [ + 0.0978899598121643, + 0.10836011171340942, + 0.12591950595378876, + 0.1209651455283165, + 0.10007082670927048, + 0.09053842723369598, + 0.08933036029338837, + 0.11989083886146545, + 0.14703482389450073 + ], + "sources": 9 + }, + { + "entropy_mean": 2.067331314086914, + "mean_weights": [ + 0.07690603286027908, + 0.1283465027809143, + 0.11419844627380371, + 0.12572115659713745, + 0.09080883860588074, + 0.09361410140991211, + 0.09618563950061798, + 0.16779401898384094, + 0.1064252257347107 + ], + "sources": 9 + }, + { + "entropy_mean": 2.126595973968506, + "mean_weights": [ + 0.10529343783855438, + 0.12262256443500519, + 0.11240119487047195, + 0.1594175547361374, + 0.11083489656448364, + 0.1012311726808548, + 0.08822474628686905, + 0.0945647805929184, + 0.1054096445441246 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1250782012939453, + "mean_weights": [ + 0.08339430391788483, + 0.1166929081082344, + 0.1221323236823082, + 0.12451616674661636, + 0.09800273925065994, + 0.07886499911546707, + 0.09880783408880234, + 0.15631729364395142, + 0.12127144634723663 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.05882728472352028, + 0.05434255301952362, + 0.06656001508235931, + 0.08629672974348068, + 0.1458369940519333, + 0.15356051921844482, + 0.13871629536151886, + 0.1133713349699974, + 0.1344493180513382, + 0.09049845486879349, + 0.09826874732971191, + 0.08978800475597382, + 0.09347957372665405, + 0.10162461549043655, + 0.09102483093738556, + 0.08678106218576431, + 0.09482892602682114, + 0.060551125556230545, + 0.07042072713375092, + 0.06859821081161499, + 0.06875573098659515, + 0.06323139369487762, + 0.06267102062702179, + 0.06210179254412651, + 0.06717807799577713, + 0.048520777374506, + 0.053565312176942825, + 0.05379631742835045, + 0.05197212100028992, + 0.049394212663173676, + 0.05453816428780556, + 0.051639657467603683, + 0.05769462510943413, + 0.04657750949263573, + 0.048166461288928986, + 0.04587066173553467, + 0.05144380033016205, + 0.049700506031513214, + 0.05281500518321991, + 0.05409446731209755, + 0.05452948063611984, + 0.04756034538149834, + 0.048005420714616776, + 0.048697564750909805, + 0.04912369325757027, + 0.049967169761657715, + 0.05079125985503197, + 0.05016079545021057, + 0.052426256239414215, + 0.04546017944812775, + 0.04680249094963074, + 0.04691625386476517, + 0.04869557544589043, + 0.04886418953537941, + 0.051024626940488815, + 0.051005132496356964, + 0.05881999060511589, + 0.05965002626180649, + 0.05215844139456749, + 0.06303735822439194, + 0.057204701006412506, + 0.06299907714128494, + 0.06047141179442406, + 0.06585761159658432 + ], + "loss_nats": 1.3476980924606323, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.8709112405776978, + "mean_weights": [ + 0.10687088966369629, + 0.08648081868886948, + 0.0809902623295784, + 0.06436935067176819, + 0.058983318507671356, + 0.0546012744307518, + 0.04820964112877846, + 0.13850103318691254, + 0.3609933853149414 + ], + "sources": 9 + }, + "step": 4000, + "stream_state_rms_by_sublayer": [ + 0.06706089526414871, + 0.11307977139949799, + 0.239619642496109, + 0.24579407274723053, + 0.2610020935535431, + 0.2624119520187378, + 0.2643596827983856, + 0.2694927752017975, + 0.08292478322982788, + 0.08423247188329697, + 0.09236455708742142, + 0.09299755096435547, + 0.10117614269256592, + 0.1023314967751503, + 0.11657007038593292, + 0.11912773549556732, + 0.04215651750564575, + 0.047540031373500824, + 0.06610929220914841, + 0.07420121878385544, + 0.09169621020555496, + 0.10375737398862839, + 0.12501466274261475, + 0.13788114488124847, + 0.0246309582144022, + 0.03878077492117882, + 0.05176131799817085, + 0.06997464597225189, + 0.0750405415892601, + 0.0936177521944046, + 0.09971214830875397, + 0.1194845587015152, + 0.02585558034479618, + 0.04525832086801529, + 0.059157054871320724, + 0.07861223816871643, + 0.08684703707695007, + 0.10294434428215027, + 0.11631578952074051, + 0.13677391409873962, + 0.05665922909975052, + 0.06066577881574631, + 0.09965206682682037, + 0.10652344673871994, + 0.14260981976985931, + 0.15397457778453827, + 0.18696846067905426, + 0.2117626667022705, + 0.05644892156124115, + 0.0858822911977768, + 0.10220976173877716, + 0.13233982026576996, + 0.1401691436767578, + 0.1727868914604187, + 0.1740308403968811, + 0.21057960391044617, + 0.016864154487848282, + 0.07290095835924149, + 0.07443612068891525, + 0.13914754986763, + 0.1412491649389267, + 0.22899514436721802, + 0.23334667086601257, + 0.3399814963340759 + ] + }, + { + "activation_grad_rms_by_block": [ + 0.00012893932580482215, + 0.00013874190335627645, + 0.00013047004176769406, + 0.0001248678017873317, + 0.0001568332372698933, + 0.00015078592696227133, + 0.00014506626757793128, + 0.00013033850700594485, + 0.00012732735194731504, + 0.00012182179489172995, + 0.00011695339344441891, + 0.00011271165567450225, + 0.00013322403538040817, + 0.00012706179404631257, + 0.00011747308599296957, + 0.00010898760228883475, + 0.00011914948117919266, + 0.00010761221346911043, + 0.00010350234515499324, + 9.157581371255219e-05, + 0.00010028973338194191, + 9.595853771315888e-05, + 8.392572635784745e-05, + 7.49087703297846e-05, + 9.893974493024871e-05, + 9.361770207760856e-05, + 8.900452667148784e-05, + 8.53935198392719e-05, + 9.635069000069052e-05, + 6.989986286498606e-05, + 6.29150090389885e-05, + 5.117334512760863e-05 + ], + "activation_grad_statistics": { + "first_quartile_mean": 0.00013825537644152064, + "first_to_last_ratio": 1.7087171008907982, + "imbalance_abs_log_ratio": 0.5357428555514637, + "last_quartile_mean": 8.091180006886134e-05, + "mean": 0.00010924439834525401, + "normalized": [ + 1.1802831793473259, + 1.2700138904861653, + 1.1942950278819688, + 1.1430133139888945, + 1.4356181153951746, + 1.3802623223364756, + 1.3279057761796307, + 1.1930909866337074, + 1.1655275133184586, + 1.115130814366578, + 1.0705665024105082, + 1.0317385365452825, + 1.2195044999869864, + 1.163096652743226, + 1.0753236575271325, + 0.9976493434875472, + 1.0906690227048055, + 0.9850593266028592, + 0.947438465704084, + 0.8382655321432377, + 0.9180309004493582, + 0.8783840559942786, + 0.7682382587032985, + 0.6856989599873513, + 0.9056733931341846, + 0.8569565441858251, + 0.814728517156546, + 0.781674128218417, + 0.8819737346732008, + 0.6398484858150421, + 0.5759106186859398, + 0.4684299232065091 + ], + "population_cv": 0.23039819631834382 + }, + "activation_output_rms_by_block": [ + 0.09847520291805267, + 0.2336176484823227, + 0.2445971816778183, + 0.25124794244766235, + 0.06568645685911179, + 0.0732743963599205, + 0.08026880025863647, + 0.09760069102048874, + 0.03844119608402252, + 0.06061003357172012, + 0.08555752038955688, + 0.11737477779388428, + 0.032784610986709595, + 0.056893471628427505, + 0.08006435632705688, + 0.10290847718715668, + 0.042495425790548325, + 0.07159165292978287, + 0.09270426630973816, + 0.12490540742874146, + 0.05401705205440521, + 0.09582175314426422, + 0.13860832154750824, + 0.18824194371700287, + 0.07648079842329025, + 0.11732161790132523, + 0.1534760445356369, + 0.1852438896894455, + 0.0618281289935112, + 0.1136249303817749, + 0.17999152839183807, + 0.2661416530609131 + ], + "activation_output_statistics": { + "first_quartile_mean": 0.1430960400030017, + "first_to_last_ratio": 0.991906938893357, + "imbalance_abs_log_ratio": 0.008125987697286065, + "last_quartile_mean": 0.1442635739222169, + "mean": 0.11505928682163358, + "normalized": [ + 0.855864881821406, + 2.030411059686819, + 2.1258360662099016, + 2.1836389689877898, + 0.5708922649671886, + 0.6368403488673751, + 0.6976299130297081, + 0.8482643488985867, + 0.33409902969079386, + 0.526772199324324, + 0.7435950869588587, + 1.0201243292585345, + 0.2849366782320903, + 0.494470921905022, + 0.6958530557483266, + 0.8943952290151663, + 0.36933503556671, + 0.6222153370441552, + 0.8057086817637722, + 1.085574323282322, + 0.46947146594210304, + 0.8328032946424252, + 1.2046687006011143, + 1.6360430254432048, + 0.6647077392531713, + 1.0196623075128106, + 1.3338866316245945, + 1.6099864235784203, + 0.5373588755973951, + 0.987533762119678, + 1.5643372505068913, + 2.3130827629193407 + ], + "population_cv": 0.5584622573890647 + }, + "bits_per_byte": 1.8658982533119042, + "branch_output_rms_by_sublayer": [ + 0.06009881943464279, + 0.0740160420536995, + 0.16605307161808014, + 0.01321887131780386, + 0.03974800556898117, + 0.0063072373159229755, + 0.0285708699375391, + 0.01633937656879425, + 0.06428490579128265, + 0.006369625683873892, + 0.019304754212498665, + 0.005901239812374115, + 0.02134516090154648, + 0.008837349712848663, + 0.04194158315658569, + 0.019798006862401962, + 0.032778121531009674, + 0.01838836632668972, + 0.02474922500550747, + 0.018625546246767044, + 0.02529645897448063, + 0.02233816310763359, + 0.031109536066651344, + 0.02339978888630867, + 0.019492624327540398, + 0.024492669850587845, + 0.021350549533963203, + 0.025974294170737267, + 0.021464798599481583, + 0.03068808652460575, + 0.025218263268470764, + 0.034959711134433746, + 0.023065272718667984, + 0.03464050218462944, + 0.03318355977535248, + 0.036559805274009705, + 0.028468552976846695, + 0.036484114825725555, + 0.04525674507021904, + 0.04875451326370239, + 0.04778723791241646, + 0.02997542731463909, + 0.06364032626152039, + 0.03515400364995003, + 0.07339563965797424, + 0.047455791383981705, + 0.07156641036272049, + 0.0753600150346756, + 0.045828815549612045, + 0.05875468626618385, + 0.03798433765769005, + 0.062369704246520996, + 0.031021421775221825, + 0.06693152338266373, + 0.013733532279729843, + 0.07132836431264877, + 0.014834110625088215, + 0.059611398726701736, + 0.01227854285389185, + 0.07063451409339905, + 0.015306126326322556, + 0.0886390432715416, + 0.023551054298877716, + 0.10633378475904465 + ], + "capture": { + "all_gradients_finite": true, + "all_gradients_present": true, + "count": 32, + "dtypes": [ + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32", + "torch.float32" + ], + "position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset", + "shape": [ + 16, + 256, + 192 + ], + "storage_unique": true + }, + "core_parameter_grad_rms_by_block": [ + 0.0004111790560190594, + 0.000318877405155091, + 0.00014742903942172142, + 0.00020755451403694258, + 0.0002463606891270598, + 0.00014456987080996807, + 0.00019212787762562836, + 0.00036555868126082633, + 0.0002991879962661554, + 0.00022983523357009925, + 0.00024435205456627054, + 0.0002669921054596226, + 0.0003006866868649958, + 0.0003291358425384917, + 0.00036871190632583635, + 0.00039276994518724825, + 0.00039201298161452774, + 0.0004400376459388244, + 0.00039498460757089043, + 0.0004741323980951389, + 0.00038284106289177163, + 0.0004510554379638166, + 0.0005610915320148874, + 0.0005919697796700657, + 0.000569093514379326, + 0.0005200314300946135, + 0.0005020604670293563, + 0.00047426498818203583, + 0.00048720678977716534, + 0.0004023984378573175, + 0.00036024062252050477, + 0.00034111967523395124 + ], + "core_parameter_grad_statistics": { + "first_quartile_mean": 0.00025420714168203714, + "first_to_last_ratio": 0.5561886763237934, + "imbalance_abs_log_ratio": 0.5866476963902508, + "last_quartile_mean": 0.0004570519906342838, + "mean": 0.00036905844609591283, + "normalized": [ + 1.1141299172765715, + 0.8640295555577653, + 0.3994734194036215, + 0.5623892807021743, + 0.6675384122303336, + 0.3917262220640156, + 0.5205892986816976, + 0.9905170444624455, + 0.8106791740742356, + 0.6227610721321047, + 0.6620958201909489, + 0.7234412551291004, + 0.8147400230121052, + 0.8918257962126523, + 0.9990610165578315, + 1.0642486287528918, + 1.062197561826425, + 1.192325092661516, + 1.0702494733537131, + 1.2847081623811933, + 1.0373453498806444, + 1.222178879078123, + 1.5203324512699758, + 1.6040000870654008, + 1.5420146060860698, + 1.409076084278166, + 1.3603820000338869, + 1.2850674282056165, + 1.3201345069624757, + 1.0903379725192364, + 0.9761072435309706, + 0.9242971644260901 + ], + "population_cv": 0.32188168615263085 + }, + "depth_weights": [ + { + "entropy_mean": 0.0, + "mean_weights": [ + 1.0 + ], + "sources": 1 + }, + { + "entropy_mean": 0.30903440713882446, + "mean_weights": [ + 0.9020393490791321, + 0.09796065837144852 + ], + "sources": 2 + }, + { + "entropy_mean": 0.5943206548690796, + "mean_weights": [ + 0.6391856670379639, + 0.36081427335739136 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6354048252105713, + "mean_weights": [ + 0.6418365240097046, + 0.3581635057926178 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6752665042877197, + "mean_weights": [ + 0.4143034815788269, + 0.5856965184211731 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6511313915252686, + "mean_weights": [ + 0.39545655250549316, + 0.6045434474945068 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6874249577522278, + "mean_weights": [ + 0.48418158292770386, + 0.5158184170722961 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6541168689727783, + "mean_weights": [ + 0.5819840431213379, + 0.4180159270763397 + ], + "sources": 2 + }, + { + "entropy_mean": 0.6862896680831909, + "mean_weights": [ + 0.5171412229537964, + 0.4828587770462036 + ], + "sources": 2 + }, + { + "entropy_mean": 0.9821022748947144, + "mean_weights": [ + 0.22901444137096405, + 0.21936020255088806, + 0.5516253709793091 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0879732370376587, + "mean_weights": [ + 0.3412911891937256, + 0.2740474343299866, + 0.38466137647628784 + ], + "sources": 3 + }, + { + "entropy_mean": 0.966917097568512, + "mean_weights": [ + 0.23901140689849854, + 0.1963682770729065, + 0.564620316028595 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0720024108886719, + "mean_weights": [ + 0.3473166227340698, + 0.24676883220672607, + 0.4059145450592041 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0106569528579712, + "mean_weights": [ + 0.23335079848766327, + 0.2564984858036041, + 0.5101507306098938 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0801734924316406, + "mean_weights": [ + 0.3693524897098541, + 0.2494254857301712, + 0.3812220096588135 + ], + "sources": 3 + }, + { + "entropy_mean": 1.0216292142868042, + "mean_weights": [ + 0.34189894795417786, + 0.1885349005460739, + 0.4695661664009094 + ], + "sources": 3 + }, + { + "entropy_mean": 1.085930585861206, + "mean_weights": [ + 0.37305235862731934, + 0.2697691321372986, + 0.3571785092353821 + ], + "sources": 3 + }, + { + "entropy_mean": 1.3077852725982666, + "mean_weights": [ + 0.22748318314552307, + 0.13374337553977966, + 0.2605305314064026, + 0.3782429099082947 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3685073852539062, + "mean_weights": [ + 0.2904655933380127, + 0.2061430811882019, + 0.2159208357334137, + 0.2874704897403717 + ], + "sources": 4 + }, + { + "entropy_mean": 1.337707281112671, + "mean_weights": [ + 0.23166599869728088, + 0.1705857813358307, + 0.2691241204738617, + 0.3286241292953491 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3637475967407227, + "mean_weights": [ + 0.27879974246025085, + 0.19106647372245789, + 0.2128581553697586, + 0.31727564334869385 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3454006910324097, + "mean_weights": [ + 0.2911474108695984, + 0.18621329963207245, + 0.21682560443878174, + 0.30581367015838623 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3548822402954102, + "mean_weights": [ + 0.30711978673934937, + 0.18216806650161743, + 0.20504707098007202, + 0.30566510558128357 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3559226989746094, + "mean_weights": [ + 0.30007290840148926, + 0.1815715879201889, + 0.22795470058918, + 0.29040080308914185 + ], + "sources": 4 + }, + { + "entropy_mean": 1.3654773235321045, + "mean_weights": [ + 0.2562333047389984, + 0.19376778602600098, + 0.2279227077960968, + 0.3220762014389038 + ], + "sources": 4 + }, + { + "entropy_mean": 1.581005334854126, + "mean_weights": [ + 0.21280249953269958, + 0.13066443800926208, + 0.1934673935174942, + 0.23154397308826447, + 0.23152171075344086 + ], + "sources": 5 + }, + { + "entropy_mean": 1.585540771484375, + "mean_weights": [ + 0.2017359882593155, + 0.1621151715517044, + 0.1603015810251236, + 0.1971220076084137, + 0.2787252962589264 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5922611951828003, + "mean_weights": [ + 0.18180608749389648, + 0.15913578867912292, + 0.20041614770889282, + 0.20758184790611267, + 0.2510601282119751 + ], + "sources": 5 + }, + { + "entropy_mean": 1.58207368850708, + "mean_weights": [ + 0.1940680742263794, + 0.16344153881072998, + 0.16575275361537933, + 0.18803098797798157, + 0.28870663046836853 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5402436256408691, + "mean_weights": [ + 0.17004631459712982, + 0.1271815001964569, + 0.18410228192806244, + 0.20228636264801025, + 0.3163835406303406 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5844833850860596, + "mean_weights": [ + 0.1959158182144165, + 0.16973957419395447, + 0.16372722387313843, + 0.1855185478925705, + 0.2850988507270813 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5373270511627197, + "mean_weights": [ + 0.18315774202346802, + 0.12609878182411194, + 0.18130838871002197, + 0.18354278802871704, + 0.32589229941368103 + ], + "sources": 5 + }, + { + "entropy_mean": 1.5777552127838135, + "mean_weights": [ + 0.20232322812080383, + 0.16796565055847168, + 0.146327942609787, + 0.19361403584480286, + 0.28976911306381226 + ], + "sources": 5 + }, + { + "entropy_mean": 1.7422258853912354, + "mean_weights": [ + 0.14358624815940857, + 0.10591697692871094, + 0.16278813779354095, + 0.15928007662296295, + 0.26065593957901, + 0.16777263581752777 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7709391117095947, + "mean_weights": [ + 0.16574758291244507, + 0.14620095491409302, + 0.12474840879440308, + 0.14619803428649902, + 0.19964303076267242, + 0.2174619883298874 + ], + "sources": 6 + }, + { + "entropy_mean": 1.729372501373291, + "mean_weights": [ + 0.13057398796081543, + 0.10316480696201324, + 0.16024847328662872, + 0.13796097040176392, + 0.20938652753829956, + 0.2586652338504791 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7713168859481812, + "mean_weights": [ + 0.15559720993041992, + 0.1570751816034317, + 0.12412820756435394, + 0.15546472370624542, + 0.17953522503376007, + 0.22819945216178894 + ], + "sources": 6 + }, + { + "entropy_mean": 1.6778056621551514, + "mean_weights": [ + 0.11762547492980957, + 0.09853287786245346, + 0.15266761183738708, + 0.12690392136573792, + 0.18736225366592407, + 0.3169078230857849 + ], + "sources": 6 + }, + { + "entropy_mean": 1.7828495502471924, + "mean_weights": [ + 0.1644119918346405, + 0.16709990799427032, + 0.13716958463191986, + 0.1527465432882309, + 0.17479197680950165, + 0.20378002524375916 + ], + "sources": 6 + }, + { + "entropy_mean": 1.691389560699463, + "mean_weights": [ + 0.12016548216342926, + 0.11117547750473022, + 0.15797217190265656, + 0.1312505006790161, + 0.17261159420013428, + 0.3068247437477112 + ], + "sources": 6 + }, + { + "entropy_mean": 1.791759729385376, + "mean_weights": [ + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592, + 0.1666666567325592 + ], + "sources": 6 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 1.9459102153778076, + "mean_weights": [ + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548, + 0.1428571492433548 + ], + "sources": 7 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.079441547393799, + "mean_weights": [ + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125, + 0.125 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0278921127319336, + "mean_weights": [ + 0.11234381794929504, + 0.1314859390258789, + 0.1422475129365921, + 0.16262540221214294, + 0.12343616783618927, + 0.1039048433303833, + 0.09523597359657288, + 0.12872034311294556 + ], + "sources": 8 + }, + { + "entropy_mean": 2.0601582527160645, + "mean_weights": [ + 0.08340039104223251, + 0.09872499108314514, + 0.10069122910499573, + 0.0878734141588211, + 0.08539019525051117, + 0.08006646484136581, + 0.1009698286652565, + 0.2190263718366623, + 0.14385712146759033 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1482632160186768, + "mean_weights": [ + 0.08904889225959778, + 0.10518711805343628, + 0.11570116132497787, + 0.12019158899784088, + 0.10622677206993103, + 0.08939772844314575, + 0.0865967720746994, + 0.12295456230640411, + 0.16469541192054749 + ], + "sources": 9 + }, + { + "entropy_mean": 1.9992003440856934, + "mean_weights": [ + 0.08175462484359741, + 0.09771331399679184, + 0.09001943469047546, + 0.08729717880487442, + 0.07171636074781418, + 0.06423543393611908, + 0.07570543885231018, + 0.20757217705249786, + 0.22398602962493896 + ], + "sources": 9 + }, + { + "entropy_mean": 2.137761354446411, + "mean_weights": [ + 0.09767191112041473, + 0.11270089447498322, + 0.12036637961864471, + 0.12793320417404175, + 0.1046464592218399, + 0.08895810693502426, + 0.08073794841766357, + 0.10764030367136002, + 0.15934479236602783 + ], + "sources": 9 + }, + { + "entropy_mean": 2.0407042503356934, + "mean_weights": [ + 0.07716986536979675, + 0.12590588629245758, + 0.11102603375911713, + 0.11707732081413269, + 0.08211341500282288, + 0.09504722058773041, + 0.09234312176704407, + 0.17442387342453003, + 0.12489324063062668 + ], + "sources": 9 + }, + { + "entropy_mean": 2.128730297088623, + "mean_weights": [ + 0.10417187213897705, + 0.13345485925674438, + 0.11557041108608246, + 0.1635882556438446, + 0.10930082947015762, + 0.09852401167154312, + 0.08385953307151794, + 0.08640560507774353, + 0.10512463748455048 + ], + "sources": 9 + }, + { + "entropy_mean": 2.1099185943603516, + "mean_weights": [ + 0.08170980215072632, + 0.11237261444330215, + 0.11802452802658081, + 0.11393916606903076, + 0.0935322567820549, + 0.07808645069599152, + 0.09232928603887558, + 0.15550142526626587, + 0.15450447797775269 + ], + "sources": 9 + } + ], + "layer_input_rms_by_sublayer": [ + 0.06159411370754242, + 0.05641147121787071, + 0.0601210743188858, + 0.0968460738658905, + 0.14369840919971466, + 0.15035921335220337, + 0.13629312813282013, + 0.11518421769142151, + 0.13050560653209686, + 0.07938458025455475, + 0.08858852833509445, + 0.07773812115192413, + 0.0843547061085701, + 0.0894903838634491, + 0.08370587974786758, + 0.07600551843643188, + 0.0868152379989624, + 0.04898906499147415, + 0.06335873156785965, + 0.055966150015592575, + 0.05818251520395279, + 0.05542164295911789, + 0.055110398679971695, + 0.053531844168901443, + 0.05831512063741684, + 0.04076668992638588, + 0.04529546573758125, + 0.04488367959856987, + 0.04510748013854027, + 0.04101870581507683, + 0.047199174761772156, + 0.043394725769758224, + 0.05005872622132301, + 0.039048757404088974, + 0.04125509411096573, + 0.037758756428956985, + 0.04443074017763138, + 0.04148557782173157, + 0.0460357666015625, + 0.04605577141046524, + 0.04641414061188698, + 0.04017103090882301, + 0.04042011499404907, + 0.04096098616719246, + 0.04127231240272522, + 0.041848864406347275, + 0.04266415163874626, + 0.042332492768764496, + 0.04439258202910423, + 0.03856244310736656, + 0.039745256304740906, + 0.03987186402082443, + 0.04138168320059776, + 0.04157072678208351, + 0.04337858036160469, + 0.04334103316068649, + 0.0472542941570282, + 0.05266488343477249, + 0.0427602119743824, + 0.05429914966225624, + 0.046690016984939575, + 0.055435240268707275, + 0.04864690452814102, + 0.0566694401204586 + ], + "loss_nats": 1.293342113494873, + "loss_scale": 1.0, + "output_weights": { + "entropy_mean": 1.7431397438049316, + "mean_weights": [ + 0.0995081216096878, + 0.0776149332523346, + 0.06879925727844238, + 0.060406532138586044, + 0.05441401153802872, + 0.04954121261835098, + 0.03926881030201912, + 0.12004305422306061, + 0.43040403723716736 + ], + "sources": 9 + }, + "step": 8000, + "stream_state_rms_by_sublayer": [ + 0.06009881943464279, + 0.09847520291805267, + 0.22995010018348694, + 0.2336176484823227, + 0.2430322766304016, + 0.2445971816778183, + 0.2476799637079239, + 0.25124794244766235, + 0.06428490579128265, + 0.06568645685911179, + 0.07276169210672379, + 0.0732743963599205, + 0.07930935174226761, + 0.08026880025863647, + 0.09504730254411697, + 0.09760069102048874, + 0.032778121531009674, + 0.03844119608402252, + 0.05227602273225784, + 0.06061003357172012, + 0.07424736768007278, + 0.08555752038955688, + 0.10497704148292542, + 0.11737477779388428, + 0.019492624327540398, + 0.032784610986709595, + 0.040958914905786514, + 0.056893471628427505, + 0.06350068747997284, + 0.08006435632705688, + 0.08542235195636749, + 0.10290847718715668, + 0.023065272718667984, + 0.042495425790548325, + 0.05381188914179802, + 0.07159165292978287, + 0.07674155384302139, + 0.09270426630973816, + 0.10431230813264847, + 0.12490540742874146, + 0.04778723791241646, + 0.05401705205440521, + 0.08787702769041061, + 0.09582175314426422, + 0.12634232640266418, + 0.13860832154750824, + 0.16462822258472443, + 0.18824194371700287, + 0.045828815549612045, + 0.07648079842329025, + 0.08903630077838898, + 0.11732161790132523, + 0.12432269752025604, + 0.1534760445356369, + 0.15425331890583038, + 0.1852438896894455, + 0.014834110625088215, + 0.0618281289935112, + 0.06328272074460983, + 0.1136249303817749, + 0.1152983158826828, + 0.17999152839183807, + 0.18247950077056885, + 0.2661416530609131 + ] + } + ], + "environment": { + "autocast": "cuda-bfloat16-forward-fp32-cross-entropy", + "compile": false, + "compute_capability": [ + 12, + 0 + ], + "cublas_workspace_config": ":4096:8", + "cuda": "12.8", + "deterministic_algorithms": true, + "gpu": "NVIDIA GeForce RTX 5090", + "python": "3.10.14", + "torch": "2.11.0+cu128" + }, + "evaluations": [ + { + "bits_per_byte": 7.9557801867581395, + "cross_entropy_nats": 5.514526605606079, + "step": 0 + }, + { + "bits_per_byte": 5.165781185190186, + "cross_entropy_nats": 3.58064666390419, + "step": 100 + }, + { + "bits_per_byte": 3.1104177286073162, + "cross_entropy_nats": 2.15597727894783, + "step": 500 + }, + { + "bits_per_byte": 1.9543160174708378, + "cross_entropy_nats": 1.354628637433052, + "step": 2000 + }, + { + "bits_per_byte": 1.8041839773784627, + "cross_entropy_nats": 1.2505650371313095, + "step": 4000 + }, + { + "bits_per_byte": 1.7180484874899726, + "cross_entropy_nats": 1.190860465168953, + "step": 8000 + } + ], + "forward_intervention": { + "depth_visit_counts": [ + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054, + 8054 + ], + "forward_calls": 8054, + "output_mixer_selected": false, + "output_visit_count": 8054, + "passed": true, + "selected_depth_indices": [ + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55 + ], + "selected_parameter_reachability_gate": true, + "selected_parameters": { + "mixers.40.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.40.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.41.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.42.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.43.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.44.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.45.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.46.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.47.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.48.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.48.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.49.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.50.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.51.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.52.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.53.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.54.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.key_norm.weight": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + }, + "mixers.55.query": { + "final_equals_initial": true, + "gradient_hook_calls": 0, + "in_optimizer_param_group": true, + "optimizer_state_present": false + } + }, + "selected_source_count_checks": { + "40": true, + "41": true, + "42": true, + "43": true, + "44": true, + "45": true, + "46": true, + "47": true, + "48": true, + "49": true, + "50": true, + "51": true, + "52": true, + "53": true, + "54": true, + "55": true + }, + "selector_gate": true, + "semantics": "selected depth mixers use parameter-free constant-zero logits with the parent softmax+einsum arithmetic kernel", + "source_counts_by_depth_index": { + "0": [ + 1 + ], + "1": [ + 2 + ], + "10": [ + 3 + ], + "11": [ + 3 + ], + "12": [ + 3 + ], + "13": [ + 3 + ], + "14": [ + 3 + ], + "15": [ + 3 + ], + "16": [ + 3 + ], + "17": [ + 4 + ], + "18": [ + 4 + ], + "19": [ + 4 + ], + "2": [ + 2 + ], + "20": [ + 4 + ], + "21": [ + 4 + ], + "22": [ + 4 + ], + "23": [ + 4 + ], + "24": [ + 4 + ], + "25": [ + 5 + ], + "26": [ + 5 + ], + "27": [ + 5 + ], + "28": [ + 5 + ], + "29": [ + 5 + ], + "3": [ + 2 + ], + "30": [ + 5 + ], + "31": [ + 5 + ], + "32": [ + 5 + ], + "33": [ + 6 + ], + "34": [ + 6 + ], + "35": [ + 6 + ], + "36": [ + 6 + ], + "37": [ + 6 + ], + "38": [ + 6 + ], + "39": [ + 6 + ], + "4": [ + 2 + ], + "40": [ + 6 + ], + "41": [ + 7 + ], + "42": [ + 7 + ], + "43": [ + 7 + ], + "44": [ + 7 + ], + "45": [ + 7 + ], + "46": [ + 7 + ], + "47": [ + 7 + ], + "48": [ + 7 + ], + "49": [ + 8 + ], + "5": [ + 2 + ], + "50": [ + 8 + ], + "51": [ + 8 + ], + "52": [ + 8 + ], + "53": [ + 8 + ], + "54": [ + 8 + ], + "55": [ + 8 + ], + "56": [ + 8 + ], + "57": [ + 9 + ], + "58": [ + 9 + ], + "59": [ + 9 + ], + "6": [ + 2 + ], + "60": [ + 9 + ], + "61": [ + 9 + ], + "62": [ + 9 + ], + "63": [ + 9 + ], + "7": [ + 2 + ], + "8": [ + 2 + ], + "9": [ + 3 + ] + }, + "uniform_weight_gate": true, + "uniform_weight_max_abs_error": 0.0, + "uniform_weight_threshold": 1e-12, + "unselected_parameter_reachability_gate": true, + "unselected_parameters": { + "mixers.0.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.0.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.1.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.10.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.11.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.12.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.13.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.14.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.15.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.16.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.17.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.18.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.19.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.2.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.20.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.21.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.22.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.23.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.24.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.25.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.26.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.27.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.28.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.29.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.3.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.30.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.31.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.32.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.33.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.34.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.35.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.36.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.37.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.38.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.39.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.4.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.5.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.56.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.57.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.58.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.59.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.6.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.60.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.61.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.62.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.63.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.7.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.8.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "mixers.9.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.key_norm.weight": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + }, + "output_mixer.query": { + "gradient_hook_calls": 8006, + "in_optimizer_param_group": true, + "optimizer_state_present": true + } + }, + "variant": "uniform_groups_6_7_forward", + "visit_gate": true + }, + "gradient_gate": null, + "hashes": { + "final_mixer_parameters": "22b43094ce3c67188311a7ad094214b5f64646f275a5093592a80f382624f2f1", + "final_model_state": "1c164cc59ddd419b6b4aeb37945f715ddbf0a1a6582dbc6c28a517527da49371", + "final_optimizer_state": "c7f66bb2b9115ed25ee91eadc1084d5bd947b04bbd5ac6888fc156e75f84147b", + "final_public_parameters": "5141274aafe289f5c2ce5cb835abb664bbe5d21b114a2d39a8e5b603decb17b0", + "initial_mixer_parameters": "c5a06c218c4501b16fcccee115aa8c79d5dceade78b38bbb547e4f5b2202516d", + "initial_public_parameter_elements": 18985152, + "initial_public_parameter_structure": "7625a2d62805070dfbe720e9243f103d23c47e27d3c14de1af55fd6187e81670", + "initial_public_parameter_tensors": 227, + "initial_public_parameters": "73bbe569e1a47981386ebaf59ca891174746bf81897fe5403676ad6d50423c58" + }, + "manifest": { + "diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716", + "file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371", + "formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e", + "input_gate_tensor_hashes": { + "0": "65136111a29a042e61a7909132560d95cd4bcf0f9b52f64d0fb2e57773856434", + "1": "d995676b4e7dec8f661cd8c2345fe7fc7a513c17f528c02fc946a441a6995a94", + "7999": "2345e7ac3decca2bdaebf13094fdc92fcabef42e3e461a2fefd6e8e7e76baccc" + }, + "path": "experiments/k3/attnres_gradient/manifest.json", + "validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338" + }, + "model": { + "attnres_aggregation_groups": 8, + "context": 256, + "d_ff": 768, + "d_head": 32, + "d_model": 192, + "heads": 6, + "layers": 32, + "parameters": { + "core": 18985152, + "embedding": 98304, + "mixer": 24960, + "total": 19010112 + }, + "sublayers": 64, + "sublayers_per_attnres_group": 8, + "transformer_blocks_per_attnres_group": 4, + "vocabulary": 256 + }, + "optimizer": { + "betas": [ + 0.9, + 0.95 + ], + "epsilon": 1e-08, + "grad_clip": 1.0, + "min_lr": 3e-05, + "name": "AdamW", + "peak_lr": 0.0003, + "warmup_steps": 400, + "weight_decay_ndim_ge_2": 0.1 + }, + "parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1", + "parent_runner_canonical_sha256": "95ee350ad9afbde622fbc162237bc49475fbfdd41633c54ad4c3ef5e0a579041", + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "run_kind": "replay", + "schema_version": 2, + "seed": 2026073001, + "steps": 8000, + "study_manifest": { + "file_sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", + "path": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/manifest.json", + "status": "frozen-before-model-output" + }, + "target_bytes_seen": 65536000, + "timing": { + "mean_ms": 119.32800952218578, + "measured_steps": 7980, + "median_ms": 95.19774450745899, + "p95_ms": 235.78832300263457, + "peak_allocated_bytes": 10566180864, + "peak_reserved_bytes": 11068768256, + "warmup_steps_excluded": 20 + }, + "training_history": [ + { + "bits_per_byte": 7.958777844362678, + "learning_rate": 7.499999999999999e-07, + "loss_nats": 5.516604423522949, + "step": 1, + "unclipped_grad_norm": 21.321983337402344 + }, + { + "bits_per_byte": 7.24805334829669, + "learning_rate": 7.499999999999999e-06, + "loss_nats": 5.023967742919922, + "step": 10, + "unclipped_grad_norm": 11.155019760131836 + }, + { + "bits_per_byte": 6.733684884373789, + "learning_rate": 1.4999999999999999e-05, + "loss_nats": 4.6674346923828125, + "step": 20, + "unclipped_grad_norm": 3.4893624782562256 + }, + { + "bits_per_byte": 6.611928046868954, + "learning_rate": 2.2499999999999998e-05, + "loss_nats": 4.583039283752441, + "step": 30, + "unclipped_grad_norm": 3.0757408142089844 + }, + { + "bits_per_byte": 6.392097070813699, + "learning_rate": 2.9999999999999997e-05, + "loss_nats": 4.4306640625, + "step": 40, + "unclipped_grad_norm": 3.2317423820495605 + }, + { + "bits_per_byte": 6.094256140188279, + "learning_rate": 3.75e-05, + "loss_nats": 4.224216461181641, + "step": 50, + "unclipped_grad_norm": 2.873729705810547 + }, + { + "bits_per_byte": 5.886236932594281, + "learning_rate": 4.4999999999999996e-05, + "loss_nats": 4.080028533935547, + "step": 60, + "unclipped_grad_norm": 2.6709868907928467 + }, + { + "bits_per_byte": 5.715815884201307, + "learning_rate": 5.2499999999999995e-05, + "loss_nats": 3.9619016647338867, + "step": 70, + "unclipped_grad_norm": 3.1735682487487793 + }, + { + "bits_per_byte": 5.566368464046354, + "learning_rate": 5.9999999999999995e-05, + "loss_nats": 3.8583126068115234, + "step": 80, + "unclipped_grad_norm": 2.5606157779693604 + }, + { + "bits_per_byte": 5.355355038500834, + "learning_rate": 6.749999999999999e-05, + "loss_nats": 3.7120492458343506, + "step": 90, + "unclipped_grad_norm": 2.426494598388672 + }, + { + "bits_per_byte": 5.181467851762371, + "learning_rate": 7.5e-05, + "loss_nats": 3.591519832611084, + "step": 100, + "unclipped_grad_norm": 2.2765960693359375 + }, + { + "bits_per_byte": 4.992471301215007, + "learning_rate": 8.249999999999999e-05, + "loss_nats": 3.460517406463623, + "step": 110, + "unclipped_grad_norm": 2.09186053276062 + }, + { + "bits_per_byte": 4.7761191902214035, + "learning_rate": 8.999999999999999e-05, + "loss_nats": 3.310553550720215, + "step": 120, + "unclipped_grad_norm": 1.8941689729690552 + }, + { + "bits_per_byte": 4.582826513055486, + "learning_rate": 9.75e-05, + "loss_nats": 3.1765732765197754, + "step": 130, + "unclipped_grad_norm": 1.6992738246917725 + }, + { + "bits_per_byte": 4.428815017311936, + "learning_rate": 0.00010499999999999999, + "loss_nats": 3.0698206424713135, + "step": 140, + "unclipped_grad_norm": 1.4459766149520874 + }, + { + "bits_per_byte": 4.272379254121823, + "learning_rate": 0.0001125, + "loss_nats": 2.9613876342773438, + "step": 150, + "unclipped_grad_norm": 1.2247486114501953 + }, + { + "bits_per_byte": 4.168162583213185, + "learning_rate": 0.00011999999999999999, + "loss_nats": 2.8891501426696777, + "step": 160, + "unclipped_grad_norm": 1.0640714168548584 + }, + { + "bits_per_byte": 4.028077554605902, + "learning_rate": 0.00012749999999999998, + "loss_nats": 2.79205060005188, + "step": 170, + "unclipped_grad_norm": 0.9820604920387268 + }, + { + "bits_per_byte": 3.9491010575378898, + "learning_rate": 0.00013499999999999997, + "loss_nats": 2.7373082637786865, + "step": 180, + "unclipped_grad_norm": 0.8641575574874878 + }, + { + "bits_per_byte": 3.839261929807245, + "learning_rate": 0.0001425, + "loss_nats": 2.6611735820770264, + "step": 190, + "unclipped_grad_norm": 0.7899060249328613 + }, + { + "bits_per_byte": 3.783438081163686, + "learning_rate": 0.00015, + "loss_nats": 2.6224794387817383, + "step": 200, + "unclipped_grad_norm": 1.0962475538253784 + }, + { + "bits_per_byte": 3.7344969782076562, + "learning_rate": 0.0001575, + "loss_nats": 2.5885560512542725, + "step": 210, + "unclipped_grad_norm": 0.7674247622489929 + }, + { + "bits_per_byte": 3.635042162936108, + "learning_rate": 0.00016499999999999997, + "loss_nats": 2.5196192264556885, + "step": 220, + "unclipped_grad_norm": 0.7568824291229248 + }, + { + "bits_per_byte": 3.6286846522643135, + "learning_rate": 0.0001725, + "loss_nats": 2.5152125358581543, + "step": 230, + "unclipped_grad_norm": 1.0865727663040161 + }, + { + "bits_per_byte": 3.5048798453530052, + "learning_rate": 0.00017999999999999998, + "loss_nats": 2.4293975830078125, + "step": 240, + "unclipped_grad_norm": 0.734052836894989 + }, + { + "bits_per_byte": 3.540276626622985, + "learning_rate": 0.0001875, + "loss_nats": 2.453932762145996, + "step": 250, + "unclipped_grad_norm": 1.3704512119293213 + }, + { + "bits_per_byte": 3.541249360496265, + "learning_rate": 0.000195, + "loss_nats": 2.4546070098876953, + "step": 260, + "unclipped_grad_norm": 1.0168581008911133 + }, + { + "bits_per_byte": 3.519328795782593, + "learning_rate": 0.00020249999999999996, + "loss_nats": 2.439412832260132, + "step": 270, + "unclipped_grad_norm": 0.8324092030525208 + }, + { + "bits_per_byte": 3.468906578237333, + "learning_rate": 0.00020999999999999998, + "loss_nats": 2.4044628143310547, + "step": 280, + "unclipped_grad_norm": 0.9137193560600281 + }, + { + "bits_per_byte": 3.367204981614913, + "learning_rate": 0.00021749999999999997, + "loss_nats": 2.3339686393737793, + "step": 290, + "unclipped_grad_norm": 0.7944438457489014 + }, + { + "bits_per_byte": 3.413081697697869, + "learning_rate": 0.000225, + "loss_nats": 2.3657679557800293, + "step": 300, + "unclipped_grad_norm": 1.5879573822021484 + }, + { + "bits_per_byte": 3.3687473220278505, + "learning_rate": 0.00023249999999999996, + "loss_nats": 2.3350377082824707, + "step": 310, + "unclipped_grad_norm": 1.0807567834854126 + }, + { + "bits_per_byte": 3.345624254619346, + "learning_rate": 0.00023999999999999998, + "loss_nats": 2.319010019302368, + "step": 320, + "unclipped_grad_norm": 1.2019305229187012 + }, + { + "bits_per_byte": 3.331462171251397, + "learning_rate": 0.0002475, + "loss_nats": 2.3091936111450195, + "step": 330, + "unclipped_grad_norm": 1.149749517440796 + }, + { + "bits_per_byte": 3.414775382843566, + "learning_rate": 0.00025499999999999996, + "loss_nats": 2.3669419288635254, + "step": 340, + "unclipped_grad_norm": 1.1368536949157715 + }, + { + "bits_per_byte": 3.351615098279501, + "learning_rate": 0.0002625, + "loss_nats": 2.32316255569458, + "step": 350, + "unclipped_grad_norm": 0.9630405306816101 + }, + { + "bits_per_byte": 3.2885696981262127, + "learning_rate": 0.00026999999999999995, + "loss_nats": 2.2794628143310547, + "step": 360, + "unclipped_grad_norm": 1.1213526725769043 + }, + { + "bits_per_byte": 3.276858367533059, + "learning_rate": 0.00027749999999999997, + "loss_nats": 2.2713451385498047, + "step": 370, + "unclipped_grad_norm": 0.8436188101768494 + }, + { + "bits_per_byte": 3.338454985835479, + "learning_rate": 0.000285, + "loss_nats": 2.3140406608581543, + "step": 380, + "unclipped_grad_norm": 1.504451036453247 + }, + { + "bits_per_byte": 3.2545759513219714, + "learning_rate": 0.0002925, + "loss_nats": 2.2559001445770264, + "step": 390, + "unclipped_grad_norm": 1.3384807109832764 + }, + { + "bits_per_byte": 3.2093125253719923, + "learning_rate": 0.0003, + "loss_nats": 2.2245259284973145, + "step": 400, + "unclipped_grad_norm": 1.2285854816436768 + }, + { + "bits_per_byte": 3.1867102214303005, + "learning_rate": 0.0002999988466112388, + "loss_nats": 2.2088592052459717, + "step": 410, + "unclipped_grad_norm": 0.8287063837051392 + }, + { + "bits_per_byte": 3.2421075530997197, + "learning_rate": 0.00029999538646466355, + "loss_nats": 2.247257709503174, + "step": 420, + "unclipped_grad_norm": 1.8037424087524414 + }, + { + "bits_per_byte": 3.205863241326288, + "learning_rate": 0.0002999896196193985, + "loss_nats": 2.222135066986084, + "step": 430, + "unclipped_grad_norm": 1.0546784400939941 + }, + { + "bits_per_byte": 3.252657656834248, + "learning_rate": 0.00029998154617398326, + "loss_nats": 2.254570484161377, + "step": 440, + "unclipped_grad_norm": 1.1148794889450073 + }, + { + "bits_per_byte": 3.155373262616588, + "learning_rate": 0.0002999711662663707, + "loss_nats": 2.187138080596924, + "step": 450, + "unclipped_grad_norm": 0.8904770612716675 + }, + { + "bits_per_byte": 3.184654684787188, + "learning_rate": 0.0002999584800739247, + "loss_nats": 2.2074344158172607, + "step": 460, + "unclipped_grad_norm": 1.0755161046981812 + }, + { + "bits_per_byte": 3.137023057734828, + "learning_rate": 0.0002999434878134173, + "loss_nats": 2.1744186878204346, + "step": 470, + "unclipped_grad_norm": 1.1856991052627563 + }, + { + "bits_per_byte": 3.137933189923194, + "learning_rate": 0.000299926189741025, + "loss_nats": 2.1750495433807373, + "step": 480, + "unclipped_grad_norm": 0.9833663702011108 + }, + { + "bits_per_byte": 3.080759621435664, + "learning_rate": 0.000299906586152324, + "loss_nats": 2.1354198455810547, + "step": 490, + "unclipped_grad_norm": 1.625687837600708 + }, + { + "bits_per_byte": 3.0477578705615995, + "learning_rate": 0.0002998846773822856, + "loss_nats": 2.1125447750091553, + "step": 500, + "unclipped_grad_norm": 1.4587886333465576 + }, + { + "bits_per_byte": 3.0781640592688424, + "learning_rate": 0.0002998604638052702, + "loss_nats": 2.1336207389831543, + "step": 510, + "unclipped_grad_norm": 0.9339342713356018 + }, + { + "bits_per_byte": 3.005261301498701, + "learning_rate": 0.0002998339458350211, + "loss_nats": 2.0830883979797363, + "step": 520, + "unclipped_grad_norm": 1.0892436504364014 + }, + { + "bits_per_byte": 3.074576501171844, + "learning_rate": 0.0002998051239246572, + "loss_nats": 2.131134033203125, + "step": 530, + "unclipped_grad_norm": 1.2632100582122803 + }, + { + "bits_per_byte": 3.0146178456362414, + "learning_rate": 0.00029977399856666537, + "loss_nats": 2.089573860168457, + "step": 540, + "unclipped_grad_norm": 0.8534988164901733 + }, + { + "bits_per_byte": 2.991983208956187, + "learning_rate": 0.0002997405702928922, + "loss_nats": 2.0738847255706787, + "step": 550, + "unclipped_grad_norm": 0.9598786234855652 + }, + { + "bits_per_byte": 2.9897278284727724, + "learning_rate": 0.0002997048396745345, + "loss_nats": 2.0723214149475098, + "step": 560, + "unclipped_grad_norm": 1.29025137424469 + }, + { + "bits_per_byte": 2.9395350357837633, + "learning_rate": 0.0002996668073221302, + "loss_nats": 2.0375304222106934, + "step": 570, + "unclipped_grad_norm": 0.9295416474342346 + }, + { + "bits_per_byte": 3.001635907218513, + "learning_rate": 0.000299626473885547, + "loss_nats": 2.080575466156006, + "step": 580, + "unclipped_grad_norm": 0.898005485534668 + }, + { + "bits_per_byte": 2.900255574187905, + "learning_rate": 0.00029958384005397226, + "loss_nats": 2.0103039741516113, + "step": 590, + "unclipped_grad_norm": 1.2257674932479858 + }, + { + "bits_per_byte": 2.9531189134795115, + "learning_rate": 0.0002995389065559004, + "loss_nats": 2.0469460487365723, + "step": 600, + "unclipped_grad_norm": 0.852761447429657 + }, + { + "bits_per_byte": 2.8686083586720357, + "learning_rate": 0.000299491674159121, + "loss_nats": 1.9883677959442139, + "step": 610, + "unclipped_grad_norm": 0.9190118908882141 + }, + { + "bits_per_byte": 2.921888927874636, + "learning_rate": 0.00029944214367070535, + "loss_nats": 2.025299072265625, + "step": 620, + "unclipped_grad_norm": 0.8095329999923706 + }, + { + "bits_per_byte": 2.845826504842843, + "learning_rate": 0.00029939031593699284, + "loss_nats": 1.97257661819458, + "step": 630, + "unclipped_grad_norm": 0.8052086234092712 + }, + { + "bits_per_byte": 2.797842313356214, + "learning_rate": 0.00029933619184357654, + "loss_nats": 1.9393165111541748, + "step": 640, + "unclipped_grad_norm": 1.0781192779541016 + }, + { + "bits_per_byte": 2.7552171012704707, + "learning_rate": 0.00029927977231528774, + "loss_nats": 1.9097709655761719, + "step": 650, + "unclipped_grad_norm": 1.3079663515090942 + }, + { + "bits_per_byte": 2.687411565306601, + "learning_rate": 0.00029922105831618057, + "loss_nats": 1.86277174949646, + "step": 660, + "unclipped_grad_norm": 0.8526151776313782 + }, + { + "bits_per_byte": 2.7299691882105557, + "learning_rate": 0.0002991600508495153, + "loss_nats": 1.8922704458236694, + "step": 670, + "unclipped_grad_norm": 1.1361027956008911 + }, + { + "bits_per_byte": 2.655181672569609, + "learning_rate": 0.00029909675095774114, + "loss_nats": 1.8404316902160645, + "step": 680, + "unclipped_grad_norm": 0.9145796895027161 + }, + { + "bits_per_byte": 2.7342496644078804, + "learning_rate": 0.0002990311597224787, + "loss_nats": 1.8952374458312988, + "step": 690, + "unclipped_grad_norm": 1.2074719667434692 + }, + { + "bits_per_byte": 2.640863428954677, + "learning_rate": 0.00029896327826450103, + "loss_nats": 1.8305070400238037, + "step": 700, + "unclipped_grad_norm": 0.8382254242897034 + }, + { + "bits_per_byte": 2.6551386769068936, + "learning_rate": 0.0002988931077437152, + "loss_nats": 1.8404018878936768, + "step": 710, + "unclipped_grad_norm": 1.0521663427352905 + }, + { + "bits_per_byte": 2.673591039483266, + "learning_rate": 0.00029882064935914164, + "loss_nats": 1.8531920909881592, + "step": 720, + "unclipped_grad_norm": 1.272436261177063 + }, + { + "bits_per_byte": 2.6318969414692934, + "learning_rate": 0.0002987459043488943, + "loss_nats": 1.8242919445037842, + "step": 730, + "unclipped_grad_norm": 1.0338865518569946 + }, + { + "bits_per_byte": 2.5773001890396405, + "learning_rate": 0.0002986688739901591, + "loss_nats": 1.786448359489441, + "step": 740, + "unclipped_grad_norm": 1.0496065616607666 + }, + { + "bits_per_byte": 2.5505205984911417, + "learning_rate": 0.0002985895595991723, + "loss_nats": 1.7678861618041992, + "step": 750, + "unclipped_grad_norm": 1.0196244716644287 + }, + { + "bits_per_byte": 2.481558823165522, + "learning_rate": 0.00029850796253119824, + "loss_nats": 1.7200855016708374, + "step": 760, + "unclipped_grad_norm": 1.0939509868621826 + }, + { + "bits_per_byte": 2.61914511583842, + "learning_rate": 0.00029842408418050544, + "loss_nats": 1.815453052520752, + "step": 770, + "unclipped_grad_norm": 0.9276040196418762 + }, + { + "bits_per_byte": 2.5086259687062635, + "learning_rate": 0.0002983379259803436, + "loss_nats": 1.738847017288208, + "step": 780, + "unclipped_grad_norm": 0.9036074876785278 + }, + { + "bits_per_byte": 2.4903447288674996, + "learning_rate": 0.0002982494894029186, + "loss_nats": 1.7261754274368286, + "step": 790, + "unclipped_grad_norm": 1.0674705505371094 + }, + { + "bits_per_byte": 2.43120591462856, + "learning_rate": 0.0002981587759593675, + "loss_nats": 1.6851835250854492, + "step": 800, + "unclipped_grad_norm": 0.9424428939819336 + }, + { + "bits_per_byte": 2.4865733213167287, + "learning_rate": 0.00029806578719973283, + "loss_nats": 1.7235612869262695, + "step": 810, + "unclipped_grad_norm": 0.8431268930435181 + }, + { + "bits_per_byte": 2.4259095369125885, + "learning_rate": 0.00029797052471293596, + "loss_nats": 1.6815123558044434, + "step": 820, + "unclipped_grad_norm": 1.1989459991455078 + }, + { + "bits_per_byte": 2.4961998782161237, + "learning_rate": 0.0002978729901267499, + "loss_nats": 1.730233907699585, + "step": 830, + "unclipped_grad_norm": 1.552717685699463 + }, + { + "bits_per_byte": 2.5448967658079282, + "learning_rate": 0.0002977731851077716, + "loss_nats": 1.7639880180358887, + "step": 840, + "unclipped_grad_norm": 1.1160975694656372 + }, + { + "bits_per_byte": 2.436699728427721, + "learning_rate": 0.0002976711113613935, + "loss_nats": 1.6889915466308594, + "step": 850, + "unclipped_grad_norm": 1.051925778388977 + }, + { + "bits_per_byte": 2.4492720041710943, + "learning_rate": 0.0002975667706317743, + "loss_nats": 1.6977059841156006, + "step": 860, + "unclipped_grad_norm": 0.9328287839889526 + }, + { + "bits_per_byte": 2.3406037343275234, + "learning_rate": 0.00029746016470180915, + "loss_nats": 1.6223828792572021, + "step": 870, + "unclipped_grad_norm": 0.865578293800354 + }, + { + "bits_per_byte": 2.3792623385537675, + "learning_rate": 0.00029735129539309926, + "loss_nats": 1.6491789817810059, + "step": 880, + "unclipped_grad_norm": 0.9337467551231384 + }, + { + "bits_per_byte": 2.312986416286672, + "learning_rate": 0.0002972401645659207, + "loss_nats": 1.6032400131225586, + "step": 890, + "unclipped_grad_norm": 1.0183653831481934 + }, + { + "bits_per_byte": 2.3254228257358633, + "learning_rate": 0.00029712677411919283, + "loss_nats": 1.6118602752685547, + "step": 900, + "unclipped_grad_norm": 1.4078937768936157 + }, + { + "bits_per_byte": 2.347192389682078, + "learning_rate": 0.0002970111259904454, + "loss_nats": 1.6269497871398926, + "step": 910, + "unclipped_grad_norm": 0.9840359687805176 + }, + { + "bits_per_byte": 2.320075197189935, + "learning_rate": 0.0002968932221557859, + "loss_nats": 1.6081535816192627, + "step": 920, + "unclipped_grad_norm": 0.9145527482032776 + }, + { + "bits_per_byte": 2.346458711693497, + "learning_rate": 0.0002967730646298655, + "loss_nats": 1.626441240310669, + "step": 930, + "unclipped_grad_norm": 0.8572312593460083 + }, + { + "bits_per_byte": 2.351096739821965, + "learning_rate": 0.0002966506554658449, + "loss_nats": 1.6296560764312744, + "step": 940, + "unclipped_grad_norm": 1.0435532331466675 + }, + { + "bits_per_byte": 2.320259906556962, + "learning_rate": 0.00029652599675535876, + "loss_nats": 1.6082816123962402, + "step": 950, + "unclipped_grad_norm": 0.8369219899177551 + }, + { + "bits_per_byte": 2.3634616044883976, + "learning_rate": 0.0002963990906284808, + "loss_nats": 1.6382267475128174, + "step": 960, + "unclipped_grad_norm": 0.9031174778938293 + }, + { + "bits_per_byte": 2.266536997976343, + "learning_rate": 0.0002962699392536863, + "loss_nats": 1.5710437297821045, + "step": 970, + "unclipped_grad_norm": 1.1165263652801514 + }, + { + "bits_per_byte": 2.3488327602060073, + "learning_rate": 0.0002961385448378161, + "loss_nats": 1.628086805343628, + "step": 980, + "unclipped_grad_norm": 0.8563165068626404 + }, + { + "bits_per_byte": 2.3144214395254714, + "learning_rate": 0.00029600490962603817, + "loss_nats": 1.6042346954345703, + "step": 990, + "unclipped_grad_norm": 1.1368024349212646 + }, + { + "bits_per_byte": 2.279426409727907, + "learning_rate": 0.00029586903590180956, + "loss_nats": 1.5799779891967773, + "step": 1000, + "unclipped_grad_norm": 1.021753191947937 + }, + { + "bits_per_byte": 2.2576482466491496, + "learning_rate": 0.00029573092598683735, + "loss_nats": 1.564882516860962, + "step": 1010, + "unclipped_grad_norm": 1.0570216178894043 + }, + { + "bits_per_byte": 2.22758757108749, + "learning_rate": 0.00029559058224103895, + "loss_nats": 1.5440460443496704, + "step": 1020, + "unclipped_grad_norm": 0.9315199255943298 + }, + { + "bits_per_byte": 2.231221736482872, + "learning_rate": 0.0002954480070625017, + "loss_nats": 1.546565055847168, + "step": 1030, + "unclipped_grad_norm": 1.043064832687378 + }, + { + "bits_per_byte": 2.2689536981862672, + "learning_rate": 0.00029530320288744205, + "loss_nats": 1.572718858718872, + "step": 1040, + "unclipped_grad_norm": 0.9336802959442139 + }, + { + "bits_per_byte": 2.259553470455408, + "learning_rate": 0.0002951561721901638, + "loss_nats": 1.5662031173706055, + "step": 1050, + "unclipped_grad_norm": 0.7879506349563599 + }, + { + "bits_per_byte": 2.1748846916091384, + "learning_rate": 0.00029500691748301594, + "loss_nats": 1.5075151920318604, + "step": 1060, + "unclipped_grad_norm": 0.8336158394813538 + }, + { + "bits_per_byte": 2.2471471579701174, + "learning_rate": 0.00029485544131634963, + "loss_nats": 1.5576037168502808, + "step": 1070, + "unclipped_grad_norm": 0.8977189660072327 + }, + { + "bits_per_byte": 2.2372222111714755, + "learning_rate": 0.0002947017462784745, + "loss_nats": 1.5507242679595947, + "step": 1080, + "unclipped_grad_norm": 0.8431094288825989 + }, + { + "bits_per_byte": 2.191021135808992, + "learning_rate": 0.00029454583499561476, + "loss_nats": 1.518700122833252, + "step": 1090, + "unclipped_grad_norm": 0.7572782039642334 + }, + { + "bits_per_byte": 2.2616172622657613, + "learning_rate": 0.00029438771013186413, + "loss_nats": 1.5676336288452148, + "step": 1100, + "unclipped_grad_norm": 1.0218000411987305 + }, + { + "bits_per_byte": 2.123016099970122, + "learning_rate": 0.0002942273743891401, + "loss_nats": 1.4715626239776611, + "step": 1110, + "unclipped_grad_norm": 0.8405666947364807 + }, + { + "bits_per_byte": 2.1662938142332395, + "learning_rate": 0.00029406483050713824, + "loss_nats": 1.5015604496002197, + "step": 1120, + "unclipped_grad_norm": 0.8585324287414551 + }, + { + "bits_per_byte": 2.1182848572448862, + "learning_rate": 0.00029390008126328504, + "loss_nats": 1.4682831764221191, + "step": 1130, + "unclipped_grad_norm": 1.0757169723510742 + }, + { + "bits_per_byte": 2.244305832595213, + "learning_rate": 0.00029373312947269045, + "loss_nats": 1.5556342601776123, + "step": 1140, + "unclipped_grad_norm": 0.8914085030555725 + }, + { + "bits_per_byte": 2.0951435596753902, + "learning_rate": 0.00029356397798809995, + "loss_nats": 1.4522428512573242, + "step": 1150, + "unclipped_grad_norm": 0.8124125599861145 + }, + { + "bits_per_byte": 2.121791583495979, + "learning_rate": 0.0002933926296998457, + "loss_nats": 1.4707138538360596, + "step": 1160, + "unclipped_grad_norm": 0.8673461675643921 + }, + { + "bits_per_byte": 2.1775493908016066, + "learning_rate": 0.0002932190875357972, + "loss_nats": 1.5093622207641602, + "step": 1170, + "unclipped_grad_norm": 1.0352803468704224 + }, + { + "bits_per_byte": 2.165567015550693, + "learning_rate": 0.00029304335446131126, + "loss_nats": 1.5010566711425781, + "step": 1180, + "unclipped_grad_norm": 0.9384984970092773 + }, + { + "bits_per_byte": 2.0844211293246975, + "learning_rate": 0.00029286543347918116, + "loss_nats": 1.4448106288909912, + "step": 1190, + "unclipped_grad_norm": 0.930260419845581 + }, + { + "bits_per_byte": 2.0516309171111944, + "learning_rate": 0.0002926853276295856, + "loss_nats": 1.4220821857452393, + "step": 1200, + "unclipped_grad_norm": 1.0661922693252563 + }, + { + "bits_per_byte": 2.101316188997507, + "learning_rate": 0.0002925030399900368, + "loss_nats": 1.4565213918685913, + "step": 1210, + "unclipped_grad_norm": 0.7982707619667053 + }, + { + "bits_per_byte": 2.11886168705588, + "learning_rate": 0.00029231857367532726, + "loss_nats": 1.4686830043792725, + "step": 1220, + "unclipped_grad_norm": 0.7903804183006287 + }, + { + "bits_per_byte": 2.0872621107343003, + "learning_rate": 0.0002921319318374776, + "loss_nats": 1.4467798471450806, + "step": 1230, + "unclipped_grad_norm": 0.7807658910751343 + }, + { + "bits_per_byte": 2.020406778916448, + "learning_rate": 0.00029194311766568174, + "loss_nats": 1.4004392623901367, + "step": 1240, + "unclipped_grad_norm": 0.9420327544212341 + }, + { + "bits_per_byte": 2.1222521530349896, + "learning_rate": 0.000291752134386253, + "loss_nats": 1.4710330963134766, + "step": 1250, + "unclipped_grad_norm": 0.8917564153671265 + }, + { + "bits_per_byte": 2.094018105208144, + "learning_rate": 0.00029155898526256856, + "loss_nats": 1.451462745666504, + "step": 1260, + "unclipped_grad_norm": 0.7565622329711914 + }, + { + "bits_per_byte": 2.0811376365544247, + "learning_rate": 0.0002913636735950141, + "loss_nats": 1.4425346851348877, + "step": 1270, + "unclipped_grad_norm": 0.8209865093231201 + }, + { + "bits_per_byte": 1.9807775046087346, + "learning_rate": 0.0002911662027209272, + "loss_nats": 1.3729703426361084, + "step": 1280, + "unclipped_grad_norm": 0.8037207722663879 + }, + { + "bits_per_byte": 2.092389601487124, + "learning_rate": 0.00029096657601454013, + "loss_nats": 1.4503339529037476, + "step": 1290, + "unclipped_grad_norm": 0.7903739213943481 + }, + { + "bits_per_byte": 2.0465137453174225, + "learning_rate": 0.00029076479688692266, + "loss_nats": 1.4185352325439453, + "step": 1300, + "unclipped_grad_norm": 0.8105548620223999 + }, + { + "bits_per_byte": 2.049106899727132, + "learning_rate": 0.00029056086878592323, + "loss_nats": 1.420332670211792, + "step": 1310, + "unclipped_grad_norm": 0.77737957239151 + }, + { + "bits_per_byte": 2.021933812873459, + "learning_rate": 0.00029035479519611045, + "loss_nats": 1.401497721672058, + "step": 1320, + "unclipped_grad_norm": 0.8686265349388123 + }, + { + "bits_per_byte": 2.0978068830066516, + "learning_rate": 0.0002901465796387134, + "loss_nats": 1.4540889263153076, + "step": 1330, + "unclipped_grad_norm": 0.7300577163696289 + }, + { + "bits_per_byte": 2.0568973698459154, + "learning_rate": 0.00028993622567156154, + "loss_nats": 1.4257326126098633, + "step": 1340, + "unclipped_grad_norm": 0.919792652130127 + }, + { + "bits_per_byte": 2.0468033641014753, + "learning_rate": 0.0002897237368890237, + "loss_nats": 1.4187359809875488, + "step": 1350, + "unclipped_grad_norm": 0.7215532064437866 + }, + { + "bits_per_byte": 2.1111642595512135, + "learning_rate": 0.0002895091169219469, + "loss_nats": 1.4633475542068481, + "step": 1360, + "unclipped_grad_norm": 0.7694162130355835 + }, + { + "bits_per_byte": 2.0903565945712748, + "learning_rate": 0.00028929236943759425, + "loss_nats": 1.4489247798919678, + "step": 1370, + "unclipped_grad_norm": 0.7001428604125977 + }, + { + "bits_per_byte": 2.04615842916074, + "learning_rate": 0.00028907349813958225, + "loss_nats": 1.4182889461517334, + "step": 1380, + "unclipped_grad_norm": 0.7102509140968323 + }, + { + "bits_per_byte": 2.0505161255683015, + "learning_rate": 0.0002888525067678175, + "loss_nats": 1.421309471130371, + "step": 1390, + "unclipped_grad_norm": 0.7400784492492676 + }, + { + "bits_per_byte": 1.9793060210479523, + "learning_rate": 0.00028862939909843273, + "loss_nats": 1.371950387954712, + "step": 1400, + "unclipped_grad_norm": 0.8035197854042053 + }, + { + "bits_per_byte": 2.0066942581978546, + "learning_rate": 0.0002884041789437225, + "loss_nats": 1.3909344673156738, + "step": 1410, + "unclipped_grad_norm": 0.7851306796073914 + }, + { + "bits_per_byte": 2.045760977254596, + "learning_rate": 0.0002881768501520778, + "loss_nats": 1.4180134534835815, + "step": 1420, + "unclipped_grad_norm": 0.7472310066223145 + }, + { + "bits_per_byte": 2.077360381593525, + "learning_rate": 0.0002879474166079205, + "loss_nats": 1.4399164915084839, + "step": 1430, + "unclipped_grad_norm": 0.771403431892395 + }, + { + "bits_per_byte": 2.025465304626276, + "learning_rate": 0.00028771588223163694, + "loss_nats": 1.4039455652236938, + "step": 1440, + "unclipped_grad_norm": 0.7264323830604553 + }, + { + "bits_per_byte": 1.9586063611727556, + "learning_rate": 0.00028748225097951054, + "loss_nats": 1.3576024770736694, + "step": 1450, + "unclipped_grad_norm": 0.6727918386459351 + }, + { + "bits_per_byte": 2.0233887860997584, + "learning_rate": 0.000287246526843655, + "loss_nats": 1.4025062322616577, + "step": 1460, + "unclipped_grad_norm": 0.822909414768219 + }, + { + "bits_per_byte": 1.9480701880329474, + "learning_rate": 0.00028700871385194533, + "loss_nats": 1.35029935836792, + "step": 1470, + "unclipped_grad_norm": 0.6954723000526428 + }, + { + "bits_per_byte": 1.999692328533277, + "learning_rate": 0.00028676881606794945, + "loss_nats": 1.3860810995101929, + "step": 1480, + "unclipped_grad_norm": 0.7725124955177307 + }, + { + "bits_per_byte": 2.0372935834220165, + "learning_rate": 0.00028652683759085854, + "loss_nats": 1.4121443033218384, + "step": 1490, + "unclipped_grad_norm": 0.7179713845252991 + }, + { + "bits_per_byte": 2.0175219419308754, + "learning_rate": 0.0002862827825554171, + "loss_nats": 1.398439645767212, + "step": 1500, + "unclipped_grad_norm": 0.7370100617408752 + }, + { + "bits_per_byte": 2.0160968936858263, + "learning_rate": 0.00028603665513185245, + "loss_nats": 1.3974518775939941, + "step": 1510, + "unclipped_grad_norm": 0.743471622467041 + }, + { + "bits_per_byte": 1.9661135758655681, + "learning_rate": 0.000285788459525803, + "loss_nats": 1.3628060817718506, + "step": 1520, + "unclipped_grad_norm": 0.783790647983551 + }, + { + "bits_per_byte": 2.0459152456924197, + "learning_rate": 0.00028553819997824707, + "loss_nats": 1.4181203842163086, + "step": 1530, + "unclipped_grad_norm": 0.8134282231330872 + }, + { + "bits_per_byte": 1.9944171046833623, + "learning_rate": 0.00028528588076542964, + "loss_nats": 1.3824245929718018, + "step": 1540, + "unclipped_grad_norm": 0.7183938026428223 + }, + { + "bits_per_byte": 2.0539129549054933, + "learning_rate": 0.00028503150619878996, + "loss_nats": 1.4236639738082886, + "step": 1550, + "unclipped_grad_norm": 0.7512509226799011 + }, + { + "bits_per_byte": 1.9679570579001664, + "learning_rate": 0.0002847750806248876, + "loss_nats": 1.3640838861465454, + "step": 1560, + "unclipped_grad_norm": 0.7425466179847717 + }, + { + "bits_per_byte": 2.039061737055537, + "learning_rate": 0.000284516608425328, + "loss_nats": 1.41336989402771, + "step": 1570, + "unclipped_grad_norm": 0.734654426574707 + }, + { + "bits_per_byte": 1.9997533823743334, + "learning_rate": 0.000284256094016688, + "loss_nats": 1.3861234188079834, + "step": 1580, + "unclipped_grad_norm": 0.7791112065315247 + }, + { + "bits_per_byte": 1.9916653822695574, + "learning_rate": 0.0002839935418504399, + "loss_nats": 1.3805172443389893, + "step": 1590, + "unclipped_grad_norm": 0.677095353603363 + }, + { + "bits_per_byte": 2.0273311444054865, + "learning_rate": 0.000283728956412876, + "loss_nats": 1.4052388668060303, + "step": 1600, + "unclipped_grad_norm": 0.7420785427093506 + }, + { + "bits_per_byte": 1.9617567393712605, + "learning_rate": 0.00028346234222503136, + "loss_nats": 1.3597861528396606, + "step": 1610, + "unclipped_grad_norm": 0.7535207271575928 + }, + { + "bits_per_byte": 1.9265297049776824, + "learning_rate": 0.00028319370384260675, + "loss_nats": 1.3353686332702637, + "step": 1620, + "unclipped_grad_norm": 0.6667174100875854 + }, + { + "bits_per_byte": 1.9517581839980493, + "learning_rate": 0.00028292304585589105, + "loss_nats": 1.3528556823730469, + "step": 1630, + "unclipped_grad_norm": 0.725364089012146 + }, + { + "bits_per_byte": 1.9890120339320458, + "learning_rate": 0.0002826503728896824, + "loss_nats": 1.3786780834197998, + "step": 1640, + "unclipped_grad_norm": 0.6893826127052307 + }, + { + "bits_per_byte": 2.0536652998882508, + "learning_rate": 0.0002823756896032095, + "loss_nats": 1.4234923124313354, + "step": 1650, + "unclipped_grad_norm": 0.7862091660499573 + }, + { + "bits_per_byte": 1.9885088126956212, + "learning_rate": 0.0002820990006900518, + "loss_nats": 1.3783292770385742, + "step": 1660, + "unclipped_grad_norm": 0.7903512120246887 + }, + { + "bits_per_byte": 1.993925922232498, + "learning_rate": 0.00028182031087805947, + "loss_nats": 1.3820841312408447, + "step": 1670, + "unclipped_grad_norm": 0.6966707706451416 + }, + { + "bits_per_byte": 2.0060665215222055, + "learning_rate": 0.0002815396249292725, + "loss_nats": 1.3904993534088135, + "step": 1680, + "unclipped_grad_norm": 0.7420340776443481 + }, + { + "bits_per_byte": 2.062234679432792, + "learning_rate": 0.0002812569476398391, + "loss_nats": 1.4294321537017822, + "step": 1690, + "unclipped_grad_norm": 0.7326623201370239 + }, + { + "bits_per_byte": 1.9473220635016941, + "learning_rate": 0.00028097228383993446, + "loss_nats": 1.349780797958374, + "step": 1700, + "unclipped_grad_norm": 0.66156405210495 + }, + { + "bits_per_byte": 1.9027651422814593, + "learning_rate": 0.0002806856383936774, + "loss_nats": 1.3188962936401367, + "step": 1710, + "unclipped_grad_norm": 0.7105585932731628 + }, + { + "bits_per_byte": 1.8903479948891646, + "learning_rate": 0.00028039701619904774, + "loss_nats": 1.3102893829345703, + "step": 1720, + "unclipped_grad_norm": 0.6643191576004028 + }, + { + "bits_per_byte": 1.8665820563317348, + "learning_rate": 0.0002801064221878024, + "loss_nats": 1.293816089630127, + "step": 1730, + "unclipped_grad_norm": 0.7081220149993896 + }, + { + "bits_per_byte": 1.96802155139424, + "learning_rate": 0.00027981386132539134, + "loss_nats": 1.364128589630127, + "step": 1740, + "unclipped_grad_norm": 0.6791313290596008 + }, + { + "bits_per_byte": 1.9944721391316382, + "learning_rate": 0.0002795193386108724, + "loss_nats": 1.382462739944458, + "step": 1750, + "unclipped_grad_norm": 0.7308006286621094 + }, + { + "bits_per_byte": 1.991308346286366, + "learning_rate": 0.00027922285907682625, + "loss_nats": 1.3802697658538818, + "step": 1760, + "unclipped_grad_norm": 0.7906081080436707 + }, + { + "bits_per_byte": 2.00654583717016, + "learning_rate": 0.00027892442778927005, + "loss_nats": 1.3908315896987915, + "step": 1770, + "unclipped_grad_norm": 0.7655326128005981 + }, + { + "bits_per_byte": 1.8592688380690956, + "learning_rate": 0.00027862404984757116, + "loss_nats": 1.288746953010559, + "step": 1780, + "unclipped_grad_norm": 0.7074741721153259 + }, + { + "bits_per_byte": 1.952668832134368, + "learning_rate": 0.0002783217303843597, + "loss_nats": 1.3534868955612183, + "step": 1790, + "unclipped_grad_norm": 0.7222051620483398 + }, + { + "bits_per_byte": 1.8861160177993834, + "learning_rate": 0.00027801747456544134, + "loss_nats": 1.3073559999465942, + "step": 1800, + "unclipped_grad_norm": 0.6845945119857788 + }, + { + "bits_per_byte": 1.9459288320470545, + "learning_rate": 0.00027771128758970843, + "loss_nats": 1.3488150835037231, + "step": 1810, + "unclipped_grad_norm": 0.6448895931243896 + }, + { + "bits_per_byte": 1.9107855532044464, + "learning_rate": 0.00027740317468905165, + "loss_nats": 1.3244556188583374, + "step": 1820, + "unclipped_grad_norm": 0.6578379273414612 + }, + { + "bits_per_byte": 1.8921146006788274, + "learning_rate": 0.0002770931411282704, + "loss_nats": 1.311513900756836, + "step": 1830, + "unclipped_grad_norm": 0.6559426188468933 + }, + { + "bits_per_byte": 1.905526151758411, + "learning_rate": 0.0002767811922049827, + "loss_nats": 1.320810079574585, + "step": 1840, + "unclipped_grad_norm": 0.677651584148407 + }, + { + "bits_per_byte": 1.9092674623452806, + "learning_rate": 0.00027646733324953503, + "loss_nats": 1.3234033584594727, + "step": 1850, + "unclipped_grad_norm": 0.7678938508033752 + }, + { + "bits_per_byte": 1.897424221058915, + "learning_rate": 0.00027615156962491097, + "loss_nats": 1.3151942491531372, + "step": 1860, + "unclipped_grad_norm": 0.6985909342765808 + }, + { + "bits_per_byte": 1.886600492926864, + "learning_rate": 0.0002758339067266396, + "loss_nats": 1.3076918125152588, + "step": 1870, + "unclipped_grad_norm": 0.7078948020935059 + }, + { + "bits_per_byte": 1.898806789589201, + "learning_rate": 0.0002755143499827035, + "loss_nats": 1.316152572631836, + "step": 1880, + "unclipped_grad_norm": 0.683250367641449 + }, + { + "bits_per_byte": 1.9117479681186746, + "learning_rate": 0.0002751929048534456, + "loss_nats": 1.3251227140426636, + "step": 1890, + "unclipped_grad_norm": 0.68494713306427 + }, + { + "bits_per_byte": 1.91050522148354, + "learning_rate": 0.0002748695768314763, + "loss_nats": 1.3242613077163696, + "step": 1900, + "unclipped_grad_norm": 0.6974118947982788 + }, + { + "bits_per_byte": 1.8642745651051085, + "learning_rate": 0.00027454437144157947, + "loss_nats": 1.2922166585922241, + "step": 1910, + "unclipped_grad_norm": 0.6859782934188843 + }, + { + "bits_per_byte": 1.8595417745360148, + "learning_rate": 0.00027421729424061787, + "loss_nats": 1.2889361381530762, + "step": 1920, + "unclipped_grad_norm": 0.6479495167732239 + }, + { + "bits_per_byte": 1.94510073558315, + "learning_rate": 0.0002738883508174385, + "loss_nats": 1.3482410907745361, + "step": 1930, + "unclipped_grad_norm": 0.6703650951385498 + }, + { + "bits_per_byte": 1.8199764777744223, + "learning_rate": 0.0002735575467927768, + "loss_nats": 1.2615115642547607, + "step": 1940, + "unclipped_grad_norm": 0.6317785382270813 + }, + { + "bits_per_byte": 1.9155513644425057, + "learning_rate": 0.0002732248878191609, + "loss_nats": 1.327759027481079, + "step": 1950, + "unclipped_grad_norm": 0.6845464706420898 + }, + { + "bits_per_byte": 1.823228669702238, + "learning_rate": 0.0002728903795808148, + "loss_nats": 1.263765811920166, + "step": 1960, + "unclipped_grad_norm": 0.662417471408844 + }, + { + "bits_per_byte": 1.9053677557369664, + "learning_rate": 0.0002725540277935614, + "loss_nats": 1.3207002878189087, + "step": 1970, + "unclipped_grad_norm": 0.6656482219696045 + }, + { + "bits_per_byte": 1.9521315583330725, + "learning_rate": 0.00027221583820472456, + "loss_nats": 1.3531144857406616, + "step": 1980, + "unclipped_grad_norm": 0.7318056225776672 + }, + { + "bits_per_byte": 1.9373622042249272, + "learning_rate": 0.0002718758165930314, + "loss_nats": 1.3428771495819092, + "step": 1990, + "unclipped_grad_norm": 0.6949125528335571 + }, + { + "bits_per_byte": 1.900603148377463, + "learning_rate": 0.0002715339687685131, + "loss_nats": 1.3173977136611938, + "step": 2000, + "unclipped_grad_norm": 0.6581599712371826 + }, + { + "bits_per_byte": 1.8758416022541884, + "learning_rate": 0.00027119030057240576, + "loss_nats": 1.300234317779541, + "step": 2010, + "unclipped_grad_norm": 0.6078773140907288 + }, + { + "bits_per_byte": 1.9067026850729634, + "learning_rate": 0.0002708448178770506, + "loss_nats": 1.3216255903244019, + "step": 2020, + "unclipped_grad_norm": 0.6958262920379639 + }, + { + "bits_per_byte": 1.983292750877603, + "learning_rate": 0.00027049752658579376, + "loss_nats": 1.3747137784957886, + "step": 2030, + "unclipped_grad_norm": 0.6581296324729919 + }, + { + "bits_per_byte": 1.9112309882701812, + "learning_rate": 0.00027014843263288514, + "loss_nats": 1.324764370918274, + "step": 2040, + "unclipped_grad_norm": 0.6548293232917786 + }, + { + "bits_per_byte": 1.9557756987222046, + "learning_rate": 0.00026979754198337724, + "loss_nats": 1.3556404113769531, + "step": 2050, + "unclipped_grad_norm": 0.6823968291282654 + }, + { + "bits_per_byte": 1.9199252272192486, + "learning_rate": 0.00026944486063302305, + "loss_nats": 1.3307907581329346, + "step": 2060, + "unclipped_grad_norm": 0.6643821597099304 + }, + { + "bits_per_byte": 1.8721622054216296, + "learning_rate": 0.0002690903946081738, + "loss_nats": 1.2976839542388916, + "step": 2070, + "unclipped_grad_norm": 0.714849591255188 + }, + { + "bits_per_byte": 1.8679991933748443, + "learning_rate": 0.00026873414996567596, + "loss_nats": 1.2947983741760254, + "step": 2080, + "unclipped_grad_norm": 0.7407627105712891 + }, + { + "bits_per_byte": 1.875685442007205, + "learning_rate": 0.00026837613279276736, + "loss_nats": 1.300126075744629, + "step": 2090, + "unclipped_grad_norm": 0.7046352028846741 + }, + { + "bits_per_byte": 1.8501363873256296, + "learning_rate": 0.0002680163492069739, + "loss_nats": 1.282416820526123, + "step": 2100, + "unclipped_grad_norm": 0.6226866245269775 + }, + { + "bits_per_byte": 1.8931537198553405, + "learning_rate": 0.00026765480535600414, + "loss_nats": 1.3122341632843018, + "step": 2110, + "unclipped_grad_norm": 0.638272225856781 + }, + { + "bits_per_byte": 1.8568935856780293, + "learning_rate": 0.0002672915074176452, + "loss_nats": 1.2871005535125732, + "step": 2120, + "unclipped_grad_norm": 0.6061417460441589 + }, + { + "bits_per_byte": 1.8254683997644243, + "learning_rate": 0.00026692646159965624, + "loss_nats": 1.2653182744979858, + "step": 2130, + "unclipped_grad_norm": 0.6564081907272339 + }, + { + "bits_per_byte": 1.948432727460966, + "learning_rate": 0.0002665596741396633, + "loss_nats": 1.350550651550293, + "step": 2140, + "unclipped_grad_norm": 0.6463961005210876 + }, + { + "bits_per_byte": 1.8285157603550626, + "learning_rate": 0.000266191151305052, + "loss_nats": 1.2674305438995361, + "step": 2150, + "unclipped_grad_norm": 0.6475794315338135 + }, + { + "bits_per_byte": 1.8751418048478277, + "learning_rate": 0.00026582089939286075, + "loss_nats": 1.2997492551803589, + "step": 2160, + "unclipped_grad_norm": 0.7653019428253174 + }, + { + "bits_per_byte": 1.9566493705885877, + "learning_rate": 0.0002654489247296732, + "loss_nats": 1.356245994567871, + "step": 2170, + "unclipped_grad_norm": 0.6685433387756348 + }, + { + "bits_per_byte": 1.8327747387210291, + "learning_rate": 0.00026507523367151007, + "loss_nats": 1.2703826427459717, + "step": 2180, + "unclipped_grad_norm": 0.6575022339820862 + }, + { + "bits_per_byte": 1.845147858554703, + "learning_rate": 0.00026469983260372037, + "loss_nats": 1.278959035873413, + "step": 2190, + "unclipped_grad_norm": 0.6102811098098755 + }, + { + "bits_per_byte": 1.8820553354298617, + "learning_rate": 0.0002643227279408727, + "loss_nats": 1.3045413494110107, + "step": 2200, + "unclipped_grad_norm": 0.6623853445053101 + }, + { + "bits_per_byte": 1.8328992541602538, + "learning_rate": 0.0002639439261266453, + "loss_nats": 1.2704689502716064, + "step": 2210, + "unclipped_grad_norm": 0.6560285687446594 + }, + { + "bits_per_byte": 1.8367568250191066, + "learning_rate": 0.000263563433633716, + "loss_nats": 1.2731428146362305, + "step": 2220, + "unclipped_grad_norm": 0.6610417366027832 + }, + { + "bits_per_byte": 1.9033600302707938, + "learning_rate": 0.00026318125696365156, + "loss_nats": 1.3193086385726929, + "step": 2230, + "unclipped_grad_norm": 0.6783599257469177 + }, + { + "bits_per_byte": 1.8456421366932827, + "learning_rate": 0.000262797402646797, + "loss_nats": 1.279301643371582, + "step": 2240, + "unclipped_grad_norm": 0.7856665253639221 + }, + { + "bits_per_byte": 1.8491072431428666, + "learning_rate": 0.00026241187724216334, + "loss_nats": 1.2817034721374512, + "step": 2250, + "unclipped_grad_norm": 0.6108865141868591 + }, + { + "bits_per_byte": 1.829634335516274, + "learning_rate": 0.0002620246873373162, + "loss_nats": 1.2682058811187744, + "step": 2260, + "unclipped_grad_norm": 0.6457281112670898 + }, + { + "bits_per_byte": 1.8200751958160175, + "learning_rate": 0.00026163583954826264, + "loss_nats": 1.261579990386963, + "step": 2270, + "unclipped_grad_norm": 0.6352243423461914 + }, + { + "bits_per_byte": 1.8463887133806782, + "learning_rate": 0.0002612453405193386, + "loss_nats": 1.279819130897522, + "step": 2280, + "unclipped_grad_norm": 0.6741878986358643 + }, + { + "bits_per_byte": 1.8767995456194944, + "learning_rate": 0.000260853196923095, + "loss_nats": 1.3008983135223389, + "step": 2290, + "unclipped_grad_norm": 0.6832013726234436 + }, + { + "bits_per_byte": 1.7966688729815419, + "learning_rate": 0.0002604594154601839, + "loss_nats": 1.2453559637069702, + "step": 2300, + "unclipped_grad_norm": 0.6234115958213806 + }, + { + "bits_per_byte": 1.8154861827430453, + "learning_rate": 0.0002600640028592441, + "loss_nats": 1.2583991289138794, + "step": 2310, + "unclipped_grad_norm": 0.5922688245773315 + }, + { + "bits_per_byte": 1.8041798712926733, + "learning_rate": 0.0002596669658767859, + "loss_nats": 1.2505621910095215, + "step": 2320, + "unclipped_grad_norm": 0.6404216289520264 + }, + { + "bits_per_byte": 1.8787744223993519, + "learning_rate": 0.000259268311297076, + "loss_nats": 1.3022671937942505, + "step": 2330, + "unclipped_grad_norm": 0.636642575263977 + }, + { + "bits_per_byte": 1.7922862390896048, + "learning_rate": 0.00025886804593202114, + "loss_nats": 1.2423181533813477, + "step": 2340, + "unclipped_grad_norm": 0.6358290314674377 + }, + { + "bits_per_byte": 1.8506316973601145, + "learning_rate": 0.00025846617662105193, + "loss_nats": 1.2827601432800293, + "step": 2350, + "unclipped_grad_norm": 0.6418381333351135 + }, + { + "bits_per_byte": 1.858295588247863, + "learning_rate": 0.00025806271023100637, + "loss_nats": 1.2880723476409912, + "step": 2360, + "unclipped_grad_norm": 0.6279385089874268 + }, + { + "bits_per_byte": 1.8775554093700364, + "learning_rate": 0.00025765765365601165, + "loss_nats": 1.3014222383499146, + "step": 2370, + "unclipped_grad_norm": 0.6207043528556824 + }, + { + "bits_per_byte": 1.8383786214167428, + "learning_rate": 0.00025725101381736707, + "loss_nats": 1.2742669582366943, + "step": 2380, + "unclipped_grad_norm": 0.6515224575996399 + }, + { + "bits_per_byte": 1.8946633835646143, + "learning_rate": 0.00025684279766342567, + "loss_nats": 1.3132805824279785, + "step": 2390, + "unclipped_grad_norm": 0.7189726829528809 + }, + { + "bits_per_byte": 1.8317641686645592, + "learning_rate": 0.00025643301216947504, + "loss_nats": 1.2696821689605713, + "step": 2400, + "unclipped_grad_norm": 0.6402539014816284 + }, + { + "bits_per_byte": 1.8239792019906034, + "learning_rate": 0.00025602166433761866, + "loss_nats": 1.2642860412597656, + "step": 2410, + "unclipped_grad_norm": 0.656789243221283 + }, + { + "bits_per_byte": 1.8057149884342747, + "learning_rate": 0.0002556087611966561, + "loss_nats": 1.2516262531280518, + "step": 2420, + "unclipped_grad_norm": 0.6494315266609192 + }, + { + "bits_per_byte": 1.8177274606490894, + "learning_rate": 0.00025519430980196277, + "loss_nats": 1.2599526643753052, + "step": 2430, + "unclipped_grad_norm": 0.6307070851325989 + }, + { + "bits_per_byte": 1.8891331094434698, + "learning_rate": 0.0002547783172353695, + "loss_nats": 1.3094472885131836, + "step": 2440, + "unclipped_grad_norm": 0.62800532579422 + }, + { + "bits_per_byte": 1.8127318805894774, + "learning_rate": 0.0002543607906050415, + "loss_nats": 1.2564899921417236, + "step": 2450, + "unclipped_grad_norm": 0.5990567207336426 + }, + { + "bits_per_byte": 1.8415953849184807, + "learning_rate": 0.0002539417370453568, + "loss_nats": 1.2764966487884521, + "step": 2460, + "unclipped_grad_norm": 0.6279588341712952 + }, + { + "bits_per_byte": 1.8055199601081964, + "learning_rate": 0.00025352116371678464, + "loss_nats": 1.2514910697937012, + "step": 2470, + "unclipped_grad_norm": 0.6423900723457336 + }, + { + "bits_per_byte": 1.837602463713399, + "learning_rate": 0.0002530990778057627, + "loss_nats": 1.2737289667129517, + "step": 2480, + "unclipped_grad_norm": 0.6430677771568298 + }, + { + "bits_per_byte": 1.8147010819418565, + "learning_rate": 0.00025267548652457475, + "loss_nats": 1.25785493850708, + "step": 2490, + "unclipped_grad_norm": 0.6383248567581177 + }, + { + "bits_per_byte": 1.823237268834781, + "learning_rate": 0.00025225039711122696, + "loss_nats": 1.2637717723846436, + "step": 2500, + "unclipped_grad_norm": 0.6207119226455688 + }, + { + "bits_per_byte": 1.7868851239392585, + "learning_rate": 0.00025182381682932464, + "loss_nats": 1.2385743856430054, + "step": 2510, + "unclipped_grad_norm": 0.6664624810218811 + }, + { + "bits_per_byte": 1.8497139979351105, + "learning_rate": 0.0002513957529679477, + "loss_nats": 1.2821240425109863, + "step": 2520, + "unclipped_grad_norm": 0.6723573803901672 + }, + { + "bits_per_byte": 1.860183269823733, + "learning_rate": 0.00025096621284152656, + "loss_nats": 1.2893807888031006, + "step": 2530, + "unclipped_grad_norm": 0.6805514097213745 + }, + { + "bits_per_byte": 1.8088952916140297, + "learning_rate": 0.0002505352037897168, + "loss_nats": 1.2538306713104248, + "step": 2540, + "unclipped_grad_norm": 0.6090428233146667 + }, + { + "bits_per_byte": 1.807309095625122, + "learning_rate": 0.0002501027331772739, + "loss_nats": 1.252731204032898, + "step": 2550, + "unclipped_grad_norm": 0.6697689890861511 + }, + { + "bits_per_byte": 1.8339791332250213, + "learning_rate": 0.00024966880839392746, + "loss_nats": 1.2712174654006958, + "step": 2560, + "unclipped_grad_norm": 0.711447536945343 + }, + { + "bits_per_byte": 1.8207660501245335, + "learning_rate": 0.0002492334368542548, + "loss_nats": 1.2620588541030884, + "step": 2570, + "unclipped_grad_norm": 0.7240896224975586 + }, + { + "bits_per_byte": 1.8499687042410384, + "learning_rate": 0.00024879662599755416, + "loss_nats": 1.282300591468811, + "step": 2580, + "unclipped_grad_norm": 0.6432276964187622 + }, + { + "bits_per_byte": 1.799047908990927, + "learning_rate": 0.00024835838328771803, + "loss_nats": 1.2470049858093262, + "step": 2590, + "unclipped_grad_norm": 0.6087315082550049 + }, + { + "bits_per_byte": 1.8000796329134532, + "learning_rate": 0.0002479187162131051, + "loss_nats": 1.2477201223373413, + "step": 2600, + "unclipped_grad_norm": 0.6460713148117065 + }, + { + "bits_per_byte": 1.731975707050337, + "learning_rate": 0.00024747763228641264, + "loss_nats": 1.2005140781402588, + "step": 2610, + "unclipped_grad_norm": 0.6017140746116638 + }, + { + "bits_per_byte": 1.8080539524860089, + "learning_rate": 0.00024703513904454795, + "loss_nats": 1.2532474994659424, + "step": 2620, + "unclipped_grad_norm": 0.6441754102706909 + }, + { + "bits_per_byte": 1.816733916875055, + "learning_rate": 0.00024659124404849956, + "loss_nats": 1.2592639923095703, + "step": 2630, + "unclipped_grad_norm": 0.6813428401947021 + }, + { + "bits_per_byte": 1.678848030424354, + "learning_rate": 0.00024614595488320833, + "loss_nats": 1.1636887788772583, + "step": 2640, + "unclipped_grad_norm": 0.595856785774231 + }, + { + "bits_per_byte": 1.7752633963072042, + "learning_rate": 0.00024569927915743745, + "loss_nats": 1.2305188179016113, + "step": 2650, + "unclipped_grad_norm": 0.6481371521949768 + }, + { + "bits_per_byte": 1.8084792655815927, + "learning_rate": 0.0002452512245036427, + "loss_nats": 1.2535423040390015, + "step": 2660, + "unclipped_grad_norm": 0.6557333469390869 + }, + { + "bits_per_byte": 1.8080642714450608, + "learning_rate": 0.00024480179857784177, + "loss_nats": 1.2532546520233154, + "step": 2670, + "unclipped_grad_norm": 0.6596212387084961 + }, + { + "bits_per_byte": 1.742593399966654, + "learning_rate": 0.00024435100905948387, + "loss_nats": 1.2078737020492554, + "step": 2680, + "unclipped_grad_norm": 0.614648163318634 + }, + { + "bits_per_byte": 1.8988219240624769, + "learning_rate": 0.00024389886365131793, + "loss_nats": 1.3161630630493164, + "step": 2690, + "unclipped_grad_norm": 0.6085010170936584 + }, + { + "bits_per_byte": 1.8138036764696543, + "learning_rate": 0.00024344537007926163, + "loss_nats": 1.257232904434204, + "step": 2700, + "unclipped_grad_norm": 0.671608567237854 + }, + { + "bits_per_byte": 1.733393188058748, + "learning_rate": 0.00024299053609226878, + "loss_nats": 1.2014966011047363, + "step": 2710, + "unclipped_grad_norm": 0.6095597743988037 + }, + { + "bits_per_byte": 1.7407918816988661, + "learning_rate": 0.00024253436946219733, + "loss_nats": 1.206624984741211, + "step": 2720, + "unclipped_grad_norm": 0.618058443069458 + }, + { + "bits_per_byte": 1.7766681505994515, + "learning_rate": 0.0002420768779836764, + "loss_nats": 1.231492519378662, + "step": 2730, + "unclipped_grad_norm": 0.6472728848457336 + }, + { + "bits_per_byte": 1.7733017621914628, + "learning_rate": 0.00024161806947397305, + "loss_nats": 1.2291591167449951, + "step": 2740, + "unclipped_grad_norm": 0.6257908344268799 + }, + { + "bits_per_byte": 1.795363696644144, + "learning_rate": 0.00024115795177285886, + "loss_nats": 1.2444512844085693, + "step": 2750, + "unclipped_grad_norm": 0.6372259259223938 + }, + { + "bits_per_byte": 1.8274453403360924, + "learning_rate": 0.0002406965327424758, + "loss_nats": 1.266688585281372, + "step": 2760, + "unclipped_grad_norm": 0.6607920527458191 + }, + { + "bits_per_byte": 1.7009056653107637, + "learning_rate": 0.00024023382026720197, + "loss_nats": 1.1789779663085938, + "step": 2770, + "unclipped_grad_norm": 0.6046813726425171 + }, + { + "bits_per_byte": 1.7231344229347816, + "learning_rate": 0.00023976982225351698, + "loss_nats": 1.1943857669830322, + "step": 2780, + "unclipped_grad_norm": 0.5984002351760864 + }, + { + "bits_per_byte": 1.8003735512637777, + "learning_rate": 0.0002393045466298666, + "loss_nats": 1.2479238510131836, + "step": 2790, + "unclipped_grad_norm": 0.6106930375099182 + }, + { + "bits_per_byte": 1.8636562874752567, + "learning_rate": 0.00023883800134652763, + "loss_nats": 1.291788101196289, + "step": 2800, + "unclipped_grad_norm": 0.7129590511322021 + }, + { + "bits_per_byte": 1.7946411975278693, + "learning_rate": 0.0002383701943754715, + "loss_nats": 1.2439504861831665, + "step": 2810, + "unclipped_grad_norm": 0.6775277256965637 + }, + { + "bits_per_byte": 1.791927483279905, + "learning_rate": 0.00023790113371022884, + "loss_nats": 1.2420694828033447, + "step": 2820, + "unclipped_grad_norm": 0.6178675889968872 + }, + { + "bits_per_byte": 1.865970142059965, + "learning_rate": 0.0002374308273657521, + "loss_nats": 1.2933919429779053, + "step": 2830, + "unclipped_grad_norm": 0.670330822467804 + }, + { + "bits_per_byte": 1.7515937680342575, + "learning_rate": 0.00023695928337827911, + "loss_nats": 1.2141122817993164, + "step": 2840, + "unclipped_grad_norm": 0.6283618211746216 + }, + { + "bits_per_byte": 1.772156013771415, + "learning_rate": 0.00023648650980519564, + "loss_nats": 1.2283649444580078, + "step": 2850, + "unclipped_grad_norm": 0.6352250576019287 + }, + { + "bits_per_byte": 1.748516998410322, + "learning_rate": 0.00023601251472489757, + "loss_nats": 1.211979627609253, + "step": 2860, + "unclipped_grad_norm": 0.6485711932182312 + }, + { + "bits_per_byte": 1.8177360597816326, + "learning_rate": 0.00023553730623665305, + "loss_nats": 1.2599586248397827, + "step": 2870, + "unclipped_grad_norm": 0.6308175325393677 + }, + { + "bits_per_byte": 1.8428422591372362, + "learning_rate": 0.00023506089246046403, + "loss_nats": 1.2773609161376953, + "step": 2880, + "unclipped_grad_norm": 0.6145550608634949 + }, + { + "bits_per_byte": 1.736003540733544, + "learning_rate": 0.00023458328153692736, + "loss_nats": 1.203305959701538, + "step": 2890, + "unclipped_grad_norm": 0.6001958847045898 + }, + { + "bits_per_byte": 1.6967804894471683, + "learning_rate": 0.0002341044816270961, + "loss_nats": 1.1761186122894287, + "step": 2900, + "unclipped_grad_norm": 0.5927048921585083 + }, + { + "bits_per_byte": 1.7640587266034922, + "learning_rate": 0.00023362450091233953, + "loss_nats": 1.222752332687378, + "step": 2910, + "unclipped_grad_norm": 0.6438782811164856 + }, + { + "bits_per_byte": 1.7472266125808982, + "learning_rate": 0.00023314334759420388, + "loss_nats": 1.2110852003097534, + "step": 2920, + "unclipped_grad_norm": 0.6283231377601624 + }, + { + "bits_per_byte": 1.755904169212832, + "learning_rate": 0.0002326610298942717, + "loss_nats": 1.2171000242233276, + "step": 2930, + "unclipped_grad_norm": 0.6333559155464172 + }, + { + "bits_per_byte": 1.7778348809029048, + "learning_rate": 0.00023217755605402175, + "loss_nats": 1.2323012351989746, + "step": 2940, + "unclipped_grad_norm": 0.6731265187263489 + }, + { + "bits_per_byte": 1.7617189026385036, + "learning_rate": 0.00023169293433468802, + "loss_nats": 1.2211304903030396, + "step": 2950, + "unclipped_grad_norm": 0.6760640144348145 + }, + { + "bits_per_byte": 1.8026423463939598, + "learning_rate": 0.00023120717301711857, + "loss_nats": 1.2494964599609375, + "step": 2960, + "unclipped_grad_norm": 0.6630797386169434 + }, + { + "bits_per_byte": 1.7652381836231092, + "learning_rate": 0.00023072028040163408, + "loss_nats": 1.2235698699951172, + "step": 2970, + "unclipped_grad_norm": 0.6644895672798157 + }, + { + "bits_per_byte": 1.7522786029499933, + "learning_rate": 0.00023023226480788593, + "loss_nats": 1.2145869731903076, + "step": 2980, + "unclipped_grad_norm": 0.6685414910316467 + }, + { + "bits_per_byte": 1.7757676494395338, + "learning_rate": 0.00022974313457471416, + "loss_nats": 1.2308683395385742, + "step": 2990, + "unclipped_grad_norm": 0.6160974502563477 + }, + { + "bits_per_byte": 1.7724707420224939, + "learning_rate": 0.0002292528980600049, + "loss_nats": 1.2285830974578857, + "step": 3000, + "unclipped_grad_norm": 0.607780396938324 + }, + { + "bits_per_byte": 1.7762887568716481, + "learning_rate": 0.00022876156364054758, + "loss_nats": 1.231229543685913, + "step": 3010, + "unclipped_grad_norm": 0.6294593811035156 + }, + { + "bits_per_byte": 1.7199477563969447, + "learning_rate": 0.0002282691397118918, + "loss_nats": 1.1921769380569458, + "step": 3020, + "unclipped_grad_norm": 0.6106687188148499 + }, + { + "bits_per_byte": 1.8015117324471877, + "learning_rate": 0.00022777563468820376, + "loss_nats": 1.2487127780914307, + "step": 3030, + "unclipped_grad_norm": 0.666127622127533 + }, + { + "bits_per_byte": 1.7212353905039546, + "learning_rate": 0.00022728105700212282, + "loss_nats": 1.1930694580078125, + "step": 3040, + "unclipped_grad_norm": 0.6306331753730774 + }, + { + "bits_per_byte": 1.757228091659174, + "learning_rate": 0.00022678541510461698, + "loss_nats": 1.2180176973342896, + "step": 3050, + "unclipped_grad_norm": 0.6245158314704895 + }, + { + "bits_per_byte": 1.7509691270463237, + "learning_rate": 0.00022628871746483876, + "loss_nats": 1.213679313659668, + "step": 3060, + "unclipped_grad_norm": 0.639885425567627 + }, + { + "bits_per_byte": 1.691025605983997, + "learning_rate": 0.0002257909725699804, + "loss_nats": 1.1721296310424805, + "step": 3070, + "unclipped_grad_norm": 0.6122934818267822 + }, + { + "bits_per_byte": 1.7294964051554986, + "learning_rate": 0.00022529218892512886, + "loss_nats": 1.1987955570220947, + "step": 3080, + "unclipped_grad_norm": 0.610792875289917 + }, + { + "bits_per_byte": 1.7942898369721567, + "learning_rate": 0.00022479237505312034, + "loss_nats": 1.2437069416046143, + "step": 3090, + "unclipped_grad_norm": 0.6861922144889832 + }, + { + "bits_per_byte": 1.7002087916094675, + "learning_rate": 0.00022429153949439496, + "loss_nats": 1.178494930267334, + "step": 3100, + "unclipped_grad_norm": 0.6396893858909607 + }, + { + "bits_per_byte": 1.7804619158948343, + "learning_rate": 0.00022378969080685053, + "loss_nats": 1.2341221570968628, + "step": 3110, + "unclipped_grad_norm": 0.6085984110832214 + }, + { + "bits_per_byte": 1.7344268037904336, + "learning_rate": 0.0002232868375656965, + "loss_nats": 1.2022130489349365, + "step": 3120, + "unclipped_grad_norm": 0.6325599551200867 + }, + { + "bits_per_byte": 1.804337063435562, + "learning_rate": 0.00022278298836330733, + "loss_nats": 1.250671148300171, + "step": 3130, + "unclipped_grad_norm": 0.6261355876922607 + }, + { + "bits_per_byte": 1.7675470507109425, + "learning_rate": 0.00022227815180907572, + "loss_nats": 1.2251702547073364, + "step": 3140, + "unclipped_grad_norm": 0.6559895873069763 + }, + { + "bits_per_byte": 1.7346134049666198, + "learning_rate": 0.00022177233652926546, + "loss_nats": 1.2023423910140991, + "step": 3150, + "unclipped_grad_norm": 0.6373011469841003 + }, + { + "bits_per_byte": 1.7581727923603634, + "learning_rate": 0.00022126555116686416, + "loss_nats": 1.218672513961792, + "step": 3160, + "unclipped_grad_norm": 0.6461610198020935 + }, + { + "bits_per_byte": 1.6645058812209519, + "learning_rate": 0.00022075780438143532, + "loss_nats": 1.15374755859375, + "step": 3170, + "unclipped_grad_norm": 0.5917196869850159 + }, + { + "bits_per_byte": 1.7640267378304317, + "learning_rate": 0.00022024910484897065, + "loss_nats": 1.2227301597595215, + "step": 3180, + "unclipped_grad_norm": 0.6284464001655579 + }, + { + "bits_per_byte": 1.73202816175885, + "learning_rate": 0.00021973946126174158, + "loss_nats": 1.2005504369735718, + "step": 3190, + "unclipped_grad_norm": 0.5928532481193542 + }, + { + "bits_per_byte": 1.69034249089477, + "learning_rate": 0.00021922888232815086, + "loss_nats": 1.1716561317443848, + "step": 3200, + "unclipped_grad_norm": 0.5921319723129272 + }, + { + "bits_per_byte": 1.7046081078185382, + "learning_rate": 0.0002187173767725837, + "loss_nats": 1.181544303894043, + "step": 3210, + "unclipped_grad_norm": 0.6365057826042175 + }, + { + "bits_per_byte": 1.707051465339346, + "learning_rate": 0.0002182049533352588, + "loss_nats": 1.183237910270691, + "step": 3220, + "unclipped_grad_norm": 0.594910740852356 + }, + { + "bits_per_byte": 1.7874896429570413, + "learning_rate": 0.00021769162077207884, + "loss_nats": 1.2389934062957764, + "step": 3230, + "unclipped_grad_norm": 0.689224123954773 + }, + { + "bits_per_byte": 1.679454957199249, + "learning_rate": 0.00021717738785448102, + "loss_nats": 1.164109468460083, + "step": 3240, + "unclipped_grad_norm": 0.5985220074653625 + }, + { + "bits_per_byte": 1.8380165979366767, + "learning_rate": 0.00021666226336928708, + "loss_nats": 1.27401602268219, + "step": 3250, + "unclipped_grad_norm": 0.6826193928718567 + }, + { + "bits_per_byte": 1.68937973201524, + "learning_rate": 0.0002161462561185532, + "loss_nats": 1.1709887981414795, + "step": 3260, + "unclipped_grad_norm": 0.6134940385818481 + }, + { + "bits_per_byte": 1.7100723406017513, + "learning_rate": 0.00021562937491941957, + "loss_nats": 1.1853318214416504, + "step": 3270, + "unclipped_grad_norm": 0.6108191609382629 + }, + { + "bits_per_byte": 1.7702869063218378, + "learning_rate": 0.00021511162860395975, + "loss_nats": 1.22706937789917, + "step": 3280, + "unclipped_grad_norm": 0.6297270655632019 + }, + { + "bits_per_byte": 1.7746875983921155, + "learning_rate": 0.0002145930260190298, + "loss_nats": 1.2301197052001953, + "step": 3290, + "unclipped_grad_norm": 0.664413571357727 + }, + { + "bits_per_byte": 1.6921553600175148, + "learning_rate": 0.00021407357602611698, + "loss_nats": 1.1729127168655396, + "step": 3300, + "unclipped_grad_norm": 0.615717887878418 + }, + { + "bits_per_byte": 1.725326513802679, + "learning_rate": 0.0002135532875011885, + "loss_nats": 1.1959052085876465, + "step": 3310, + "unclipped_grad_norm": 0.6334171891212463 + }, + { + "bits_per_byte": 1.7268232788131381, + "learning_rate": 0.0002130321693345397, + "loss_nats": 1.196942687034607, + "step": 3320, + "unclipped_grad_norm": 0.6215294599533081 + }, + { + "bits_per_byte": 1.740506390498434, + "learning_rate": 0.00021251023043064224, + "loss_nats": 1.2064270973205566, + "step": 3330, + "unclipped_grad_norm": 0.6459984183311462 + }, + { + "bits_per_byte": 1.7147918885067286, + "learning_rate": 0.00021198747970799188, + "loss_nats": 1.188603162765503, + "step": 3340, + "unclipped_grad_norm": 0.6547986268997192 + }, + { + "bits_per_byte": 1.7022486778313513, + "learning_rate": 0.00021146392609895618, + "loss_nats": 1.1799088716506958, + "step": 3350, + "unclipped_grad_norm": 0.6313554048538208 + }, + { + "bits_per_byte": 1.7441553464017903, + "learning_rate": 0.00021093957854962178, + "loss_nats": 1.2089563608169556, + "step": 3360, + "unclipped_grad_norm": 0.6534625291824341 + }, + { + "bits_per_byte": 1.6944064409346582, + "learning_rate": 0.00021041444601964157, + "loss_nats": 1.1744730472564697, + "step": 3370, + "unclipped_grad_norm": 0.6572955846786499 + }, + { + "bits_per_byte": 1.7221503382065446, + "learning_rate": 0.00020988853748208164, + "loss_nats": 1.1937036514282227, + "step": 3380, + "unclipped_grad_norm": 0.5944768786430359 + }, + { + "bits_per_byte": 1.7498646544624827, + "learning_rate": 0.00020936186192326782, + "loss_nats": 1.2129137516021729, + "step": 3390, + "unclipped_grad_norm": 0.641020655632019 + }, + { + "bits_per_byte": 1.8062166618268416, + "learning_rate": 0.00020883442834263225, + "loss_nats": 1.2519739866256714, + "step": 3400, + "unclipped_grad_norm": 0.6903617978096008 + }, + { + "bits_per_byte": 1.8293177154560358, + "learning_rate": 0.0002083062457525596, + "loss_nats": 1.2679864168167114, + "step": 3410, + "unclipped_grad_norm": 0.6414300203323364 + }, + { + "bits_per_byte": 1.6874834513068266, + "learning_rate": 0.00020777732317823306, + "loss_nats": 1.1696743965148926, + "step": 3420, + "unclipped_grad_norm": 0.6029585599899292 + }, + { + "bits_per_byte": 1.6310922318980712, + "learning_rate": 0.00020724766965748005, + "loss_nats": 1.1305869817733765, + "step": 3430, + "unclipped_grad_norm": 0.6737542152404785 + }, + { + "bits_per_byte": 1.715020969397678, + "learning_rate": 0.00020671729424061788, + "loss_nats": 1.1887619495391846, + "step": 3440, + "unclipped_grad_norm": 0.6997514963150024 + }, + { + "bits_per_byte": 1.7837383573764216, + "learning_rate": 0.00020618620599029906, + "loss_nats": 1.2363932132720947, + "step": 3450, + "unclipped_grad_norm": 0.6508408784866333 + }, + { + "bits_per_byte": 1.7008217377771426, + "learning_rate": 0.00020565441398135649, + "loss_nats": 1.178919792175293, + "step": 3460, + "unclipped_grad_norm": 0.6544042229652405 + }, + { + "bits_per_byte": 1.698592498656659, + "learning_rate": 0.00020512192730064827, + "loss_nats": 1.1773746013641357, + "step": 3470, + "unclipped_grad_norm": 0.6608152985572815 + }, + { + "bits_per_byte": 1.637566002841849, + "learning_rate": 0.00020458875504690273, + "loss_nats": 1.135074257850647, + "step": 3480, + "unclipped_grad_norm": 0.604396641254425 + }, + { + "bits_per_byte": 1.8491990818784272, + "learning_rate": 0.00020405490633056245, + "loss_nats": 1.2817671298980713, + "step": 3490, + "unclipped_grad_norm": 0.6825912594795227 + }, + { + "bits_per_byte": 1.7082491525199548, + "learning_rate": 0.00020352039027362917, + "loss_nats": 1.1840680837631226, + "step": 3500, + "unclipped_grad_norm": 0.6790757775306702 + }, + { + "bits_per_byte": 1.7441649774302386, + "learning_rate": 0.0002029852160095075, + "loss_nats": 1.2089630365371704, + "step": 3510, + "unclipped_grad_norm": 0.6400102972984314 + }, + { + "bits_per_byte": 1.7249619105828498, + "learning_rate": 0.000202449392682849, + "loss_nats": 1.1956524848937988, + "step": 3520, + "unclipped_grad_norm": 0.6423133611679077 + }, + { + "bits_per_byte": 1.7157216267172928, + "learning_rate": 0.0002019129294493959, + "loss_nats": 1.1892476081848145, + "step": 3530, + "unclipped_grad_norm": 0.6613569855690002 + }, + { + "bits_per_byte": 1.7255580024507404, + "learning_rate": 0.00020137583547582478, + "loss_nats": 1.1960656642913818, + "step": 3540, + "unclipped_grad_norm": 0.6887786388397217 + }, + { + "bits_per_byte": 1.7043931295049597, + "learning_rate": 0.00020083811993958973, + "loss_nats": 1.1813952922821045, + "step": 3550, + "unclipped_grad_norm": 0.639394223690033 + }, + { + "bits_per_byte": 1.6599995918030446, + "learning_rate": 0.00020029979202876563, + "loss_nats": 1.1506240367889404, + "step": 3560, + "unclipped_grad_norm": 0.6266466379165649 + }, + { + "bits_per_byte": 1.6771077379802735, + "learning_rate": 0.00019976086094189118, + "loss_nats": 1.162482500076294, + "step": 3570, + "unclipped_grad_norm": 0.6500812768936157 + }, + { + "bits_per_byte": 1.70040863544977, + "learning_rate": 0.00019922133588781175, + "loss_nats": 1.178633451461792, + "step": 3580, + "unclipped_grad_norm": 0.644574761390686 + }, + { + "bits_per_byte": 1.7343306654886013, + "learning_rate": 0.0001986812260855219, + "loss_nats": 1.2021464109420776, + "step": 3590, + "unclipped_grad_norm": 0.671259880065918 + }, + { + "bits_per_byte": 1.747299705207515, + "learning_rate": 0.00019814054076400787, + "loss_nats": 1.2111358642578125, + "step": 3600, + "unclipped_grad_norm": 0.6472942233085632 + }, + { + "bits_per_byte": 1.6662810861431578, + "learning_rate": 0.00019759928916209004, + "loss_nats": 1.1549780368804932, + "step": 3610, + "unclipped_grad_norm": 0.608910322189331 + }, + { + "bits_per_byte": 1.64372349769069, + "learning_rate": 0.00019705748052826493, + "loss_nats": 1.1393423080444336, + "step": 3620, + "unclipped_grad_norm": 0.621802568435669 + }, + { + "bits_per_byte": 1.703449976647628, + "learning_rate": 0.00019651512412054723, + "loss_nats": 1.180741548538208, + "step": 3630, + "unclipped_grad_norm": 0.6392863392829895 + }, + { + "bits_per_byte": 1.6776752807281206, + "learning_rate": 0.00019597222920631144, + "loss_nats": 1.1628758907318115, + "step": 3640, + "unclipped_grad_norm": 0.5891351103782654 + }, + { + "bits_per_byte": 1.7084309381819167, + "learning_rate": 0.0001954288050621338, + "loss_nats": 1.1841940879821777, + "step": 3650, + "unclipped_grad_norm": 0.6256741285324097 + }, + { + "bits_per_byte": 1.630717481701841, + "learning_rate": 0.00019488486097363363, + "loss_nats": 1.1303272247314453, + "step": 3660, + "unclipped_grad_norm": 0.6102724075317383 + }, + { + "bits_per_byte": 1.7067716495663923, + "learning_rate": 0.00019434040623531452, + "loss_nats": 1.1830439567565918, + "step": 3670, + "unclipped_grad_norm": 0.6156262159347534 + }, + { + "bits_per_byte": 1.6514180014902922, + "learning_rate": 0.0001937954501504058, + "loss_nats": 1.1446757316589355, + "step": 3680, + "unclipped_grad_norm": 0.6071178913116455 + }, + { + "bits_per_byte": 1.6306356179600303, + "learning_rate": 0.00019325000203070333, + "loss_nats": 1.1302704811096191, + "step": 3690, + "unclipped_grad_norm": 0.6209743022918701 + }, + { + "bits_per_byte": 1.6608648365195353, + "learning_rate": 0.0001927040711964106, + "loss_nats": 1.1512237787246704, + "step": 3700, + "unclipped_grad_norm": 0.6676256656646729 + }, + { + "bits_per_byte": 1.7052348125982824, + "learning_rate": 0.00019215766697597923, + "loss_nats": 1.181978702545166, + "step": 3710, + "unclipped_grad_norm": 0.636160671710968 + }, + { + "bits_per_byte": 1.6914129109137401, + "learning_rate": 0.00019161079870594978, + "loss_nats": 1.1723980903625488, + "step": 3720, + "unclipped_grad_norm": 0.6435323357582092 + }, + { + "bits_per_byte": 1.6087709476078889, + "learning_rate": 0.0001910634757307922, + "loss_nats": 1.1151150465011597, + "step": 3730, + "unclipped_grad_norm": 0.6332689523696899 + }, + { + "bits_per_byte": 1.6706470377179612, + "learning_rate": 0.0001905157074027459, + "loss_nats": 1.1580042839050293, + "step": 3740, + "unclipped_grad_norm": 0.6307963728904724 + }, + { + "bits_per_byte": 1.6905688200633053, + "learning_rate": 0.00018996750308166037, + "loss_nats": 1.1718130111694336, + "step": 3750, + "unclipped_grad_norm": 0.6598763465881348 + }, + { + "bits_per_byte": 1.7722440688886567, + "learning_rate": 0.00018941887213483482, + "loss_nats": 1.2284259796142578, + "step": 3760, + "unclipped_grad_norm": 0.640985906124115 + }, + { + "bits_per_byte": 1.7387026364561848, + "learning_rate": 0.00018886982393685845, + "loss_nats": 1.205176830291748, + "step": 3770, + "unclipped_grad_norm": 0.6711131930351257 + }, + { + "bits_per_byte": 1.5617717007627514, + "learning_rate": 0.00018832036786945008, + "loss_nats": 1.0825376510620117, + "step": 3780, + "unclipped_grad_norm": 0.6163194179534912 + }, + { + "bits_per_byte": 1.6614848339758959, + "learning_rate": 0.0001877705133212979, + "loss_nats": 1.151653528213501, + "step": 3790, + "unclipped_grad_norm": 0.6834599375724792 + }, + { + "bits_per_byte": 1.656327418241822, + "learning_rate": 0.00018722026968789906, + "loss_nats": 1.1480786800384521, + "step": 3800, + "unclipped_grad_norm": 0.6011042594909668 + }, + { + "bits_per_byte": 1.6923225271541535, + "learning_rate": 0.00018666964637139907, + "loss_nats": 1.173028588294983, + "step": 3810, + "unclipped_grad_norm": 0.6273730397224426 + }, + { + "bits_per_byte": 1.6737349862142028, + "learning_rate": 0.00018611865278043115, + "loss_nats": 1.1601446866989136, + "step": 3820, + "unclipped_grad_norm": 0.6733741760253906 + }, + { + "bits_per_byte": 1.7115077078058523, + "learning_rate": 0.00018556729832995555, + "loss_nats": 1.1863267421722412, + "step": 3830, + "unclipped_grad_norm": 0.6350049376487732 + }, + { + "bits_per_byte": 1.6544032763439689, + "learning_rate": 0.00018501559244109864, + "loss_nats": 1.146744966506958, + "step": 3840, + "unclipped_grad_norm": 0.6300914287567139 + }, + { + "bits_per_byte": 1.5964234531891937, + "learning_rate": 0.00018446354454099178, + "loss_nats": 1.1065564155578613, + "step": 3850, + "unclipped_grad_norm": 0.6059905290603638 + }, + { + "bits_per_byte": 1.6516942056275778, + "learning_rate": 0.00018391116406261048, + "loss_nats": 1.144867181777954, + "step": 3860, + "unclipped_grad_norm": 0.6342678666114807 + }, + { + "bits_per_byte": 1.7301702331815791, + "learning_rate": 0.00018335846044461305, + "loss_nats": 1.1992626190185547, + "step": 3870, + "unclipped_grad_norm": 0.6215102076530457 + }, + { + "bits_per_byte": 1.6283571918014, + "learning_rate": 0.0001828054431311793, + "loss_nats": 1.1286911964416504, + "step": 3880, + "unclipped_grad_norm": 0.5960464477539062 + }, + { + "bits_per_byte": 1.6026741626348024, + "learning_rate": 0.00018225212157184934, + "loss_nats": 1.1108890771865845, + "step": 3890, + "unclipped_grad_norm": 0.6551031470298767 + }, + { + "bits_per_byte": 1.6802876971947267, + "learning_rate": 0.00018169850522136192, + "loss_nats": 1.164686679840088, + "step": 3900, + "unclipped_grad_norm": 0.6671215295791626 + }, + { + "bits_per_byte": 1.6830413114176912, + "learning_rate": 0.000181144603539493, + "loss_nats": 1.1665953397750854, + "step": 3910, + "unclipped_grad_norm": 0.6533258557319641 + }, + { + "bits_per_byte": 1.6358492720209363, + "learning_rate": 0.00018059042599089412, + "loss_nats": 1.133884310722351, + "step": 3920, + "unclipped_grad_norm": 0.6260588765144348 + }, + { + "bits_per_byte": 1.6750401625516007, + "learning_rate": 0.00018003598204493048, + "loss_nats": 1.1610493659973145, + "step": 3930, + "unclipped_grad_norm": 0.6286660432815552 + }, + { + "bits_per_byte": 1.690076777699187, + "learning_rate": 0.00017948128117551936, + "loss_nats": 1.1714719533920288, + "step": 3940, + "unclipped_grad_norm": 0.638534665107727 + }, + { + "bits_per_byte": 1.7253781085979378, + "learning_rate": 0.00017892633286096816, + "loss_nats": 1.1959409713745117, + "step": 3950, + "unclipped_grad_norm": 0.6417268514633179 + }, + { + "bits_per_byte": 1.7046425043487108, + "learning_rate": 0.00017837114658381247, + "loss_nats": 1.1815681457519531, + "step": 3960, + "unclipped_grad_norm": 0.6444203853607178 + }, + { + "bits_per_byte": 1.6336202048831037, + "learning_rate": 0.00017781573183065386, + "loss_nats": 1.1323392391204834, + "step": 3970, + "unclipped_grad_norm": 0.6553277373313904 + }, + { + "bits_per_byte": 1.6738094547020264, + "learning_rate": 0.0001772600980919981, + "loss_nats": 1.160196304321289, + "step": 3980, + "unclipped_grad_norm": 0.6473615765571594 + }, + { + "bits_per_byte": 1.6985402159307965, + "learning_rate": 0.0001767042548620927, + "loss_nats": 1.1773383617401123, + "step": 3990, + "unclipped_grad_norm": 0.6596603989601135 + }, + { + "bits_per_byte": 1.5694868424804571, + "learning_rate": 0.00017614821163876486, + "loss_nats": 1.0878853797912598, + "step": 4000, + "unclipped_grad_norm": 0.6214215755462646 + }, + { + "bits_per_byte": 1.6703202706813218, + "learning_rate": 0.00017559197792325906, + "loss_nats": 1.1577777862548828, + "step": 4010, + "unclipped_grad_norm": 0.6871203780174255 + }, + { + "bits_per_byte": 1.6408489796641692, + "learning_rate": 0.0001750355632200748, + "loss_nats": 1.1373498439788818, + "step": 4020, + "unclipped_grad_norm": 0.645101010799408 + }, + { + "bits_per_byte": 1.6883333895673907, + "learning_rate": 0.00017447897703680418, + "loss_nats": 1.1702635288238525, + "step": 4030, + "unclipped_grad_norm": 0.6595595479011536 + }, + { + "bits_per_byte": 1.60955054496425, + "learning_rate": 0.0001739222288839694, + "loss_nats": 1.1156554222106934, + "step": 4040, + "unclipped_grad_norm": 0.6268433332443237 + }, + { + "bits_per_byte": 1.647514339281008, + "learning_rate": 0.00017336532827486026, + "loss_nats": 1.141969919204712, + "step": 4050, + "unclipped_grad_norm": 0.6421452760696411 + }, + { + "bits_per_byte": 1.6559180995327685, + "learning_rate": 0.00017280828472537163, + "loss_nats": 1.1477949619293213, + "step": 4060, + "unclipped_grad_norm": 0.6443775296211243 + }, + { + "bits_per_byte": 1.6818680457735051, + "learning_rate": 0.0001722511077538408, + "loss_nats": 1.16578209400177, + "step": 4070, + "unclipped_grad_norm": 0.6187630891799927 + }, + { + "bits_per_byte": 1.7579626295610091, + "learning_rate": 0.00017169380688088497, + "loss_nats": 1.218526840209961, + "step": 4080, + "unclipped_grad_norm": 0.7221863865852356 + }, + { + "bits_per_byte": 1.645009927919144, + "learning_rate": 0.0001711363916292384, + "loss_nats": 1.1402339935302734, + "step": 4090, + "unclipped_grad_norm": 0.6955327391624451 + }, + { + "bits_per_byte": 1.6512618412433087, + "learning_rate": 0.00017057887152358974, + "loss_nats": 1.1445674896240234, + "step": 4100, + "unclipped_grad_norm": 0.650931179523468 + }, + { + "bits_per_byte": 1.63509650395811, + "learning_rate": 0.0001700212560904194, + "loss_nats": 1.1333625316619873, + "step": 4110, + "unclipped_grad_norm": 0.6691992878913879 + }, + { + "bits_per_byte": 1.6736204457687283, + "learning_rate": 0.00016946355485783656, + "loss_nats": 1.1600652933120728, + "step": 4120, + "unclipped_grad_norm": 0.6472700834274292 + }, + { + "bits_per_byte": 1.6208185042834795, + "learning_rate": 0.0001689057773554166, + "loss_nats": 1.1234657764434814, + "step": 4130, + "unclipped_grad_norm": 0.6534613370895386 + }, + { + "bits_per_byte": 1.6688040716313153, + "learning_rate": 0.00016834793311403806, + "loss_nats": 1.1567268371582031, + "step": 4140, + "unclipped_grad_norm": 0.6454835534095764 + }, + { + "bits_per_byte": 1.6179322914366998, + "learning_rate": 0.00016779003166571981, + "loss_nats": 1.1214652061462402, + "step": 4150, + "unclipped_grad_norm": 0.6616674065589905 + }, + { + "bits_per_byte": 1.6541207088486012, + "learning_rate": 0.00016723208254345838, + "loss_nats": 1.146549105644226, + "step": 4160, + "unclipped_grad_norm": 0.6816359758377075 + }, + { + "bits_per_byte": 1.6906610027641678, + "learning_rate": 0.00016667409528106481, + "loss_nats": 1.1718769073486328, + "step": 4170, + "unclipped_grad_norm": 0.6716027855873108 + }, + { + "bits_per_byte": 1.7341571349938807, + "learning_rate": 0.00016611607941300184, + "loss_nats": 1.202026128768921, + "step": 4180, + "unclipped_grad_norm": 0.7155779004096985 + }, + { + "bits_per_byte": 1.6401925218858258, + "learning_rate": 0.0001655580444742211, + "loss_nats": 1.1368948221206665, + "step": 4190, + "unclipped_grad_norm": 0.6572866439819336 + }, + { + "bits_per_byte": 1.68856247045834, + "learning_rate": 0.00016499999999999997, + "loss_nats": 1.1704223155975342, + "step": 4200, + "unclipped_grad_norm": 0.6852372884750366 + }, + { + "bits_per_byte": 1.666490217046607, + "learning_rate": 0.0001644419555257789, + "loss_nats": 1.155122995376587, + "step": 4210, + "unclipped_grad_norm": 0.645636796951294 + }, + { + "bits_per_byte": 1.6858561514643626, + "learning_rate": 0.0001638839205869981, + "loss_nats": 1.168546438217163, + "step": 4220, + "unclipped_grad_norm": 0.6557843089103699 + }, + { + "bits_per_byte": 1.6662501292660026, + "learning_rate": 0.0001633259047189352, + "loss_nats": 1.154956579208374, + "step": 4230, + "unclipped_grad_norm": 0.6790511012077332 + }, + { + "bits_per_byte": 1.609906549051536, + "learning_rate": 0.0001627679174565416, + "loss_nats": 1.1159021854400635, + "step": 4240, + "unclipped_grad_norm": 0.6366406083106995 + }, + { + "bits_per_byte": 1.649572971611836, + "learning_rate": 0.00016220996833428016, + "loss_nats": 1.1433968544006348, + "step": 4250, + "unclipped_grad_norm": 0.6794030070304871 + }, + { + "bits_per_byte": 1.628443527092133, + "learning_rate": 0.00016165206688596191, + "loss_nats": 1.1287510395050049, + "step": 4260, + "unclipped_grad_norm": 0.6431719064712524 + }, + { + "bits_per_byte": 1.6184941587570687, + "learning_rate": 0.00016109422264458335, + "loss_nats": 1.1218546628952026, + "step": 4270, + "unclipped_grad_norm": 0.6492335200309753 + }, + { + "bits_per_byte": 1.621275462186822, + "learning_rate": 0.00016053644514216336, + "loss_nats": 1.1237825155258179, + "step": 4280, + "unclipped_grad_norm": 0.6348761916160583 + }, + { + "bits_per_byte": 1.6786368357290946, + "learning_rate": 0.00015997874390958058, + "loss_nats": 1.16354238986969, + "step": 4290, + "unclipped_grad_norm": 0.6547877788543701 + }, + { + "bits_per_byte": 1.593915774156963, + "learning_rate": 0.00015942112847641018, + "loss_nats": 1.1048182249069214, + "step": 4300, + "unclipped_grad_norm": 0.6597434282302856 + }, + { + "bits_per_byte": 1.686149037918782, + "learning_rate": 0.00015886360837076156, + "loss_nats": 1.168749451637268, + "step": 4310, + "unclipped_grad_norm": 0.6848970055580139 + }, + { + "bits_per_byte": 1.6599087849633891, + "learning_rate": 0.00015830619311911495, + "loss_nats": 1.1505610942840576, + "step": 4320, + "unclipped_grad_norm": 0.6610612869262695 + }, + { + "bits_per_byte": 1.6178521475213978, + "learning_rate": 0.00015774889224615915, + "loss_nats": 1.1214096546173096, + "step": 4330, + "unclipped_grad_norm": 0.6605933904647827 + }, + { + "bits_per_byte": 1.6522273518452526, + "learning_rate": 0.00015719171527462834, + "loss_nats": 1.1452367305755615, + "step": 4340, + "unclipped_grad_norm": 0.6742057800292969 + }, + { + "bits_per_byte": 1.6295751729348102, + "learning_rate": 0.0001566346717251397, + "loss_nats": 1.129535436630249, + "step": 4350, + "unclipped_grad_norm": 0.6645446419715881 + }, + { + "bits_per_byte": 1.6214135642554648, + "learning_rate": 0.00015607777111603056, + "loss_nats": 1.1238782405853271, + "step": 4360, + "unclipped_grad_norm": 0.639927864074707 + }, + { + "bits_per_byte": 1.6188106068346562, + "learning_rate": 0.00015552102296319576, + "loss_nats": 1.122074007987976, + "step": 4370, + "unclipped_grad_norm": 0.6986302137374878 + }, + { + "bits_per_byte": 1.6898489006867936, + "learning_rate": 0.00015496443677992516, + "loss_nats": 1.171314001083374, + "step": 4380, + "unclipped_grad_norm": 0.6770191788673401 + }, + { + "bits_per_byte": 1.64916915634761, + "learning_rate": 0.0001544080220767409, + "loss_nats": 1.1431169509887695, + "step": 4390, + "unclipped_grad_norm": 0.6751537322998047 + }, + { + "bits_per_byte": 1.5996969709657163, + "learning_rate": 0.00015385178836123511, + "loss_nats": 1.108825445175171, + "step": 4400, + "unclipped_grad_norm": 0.6509587168693542 + }, + { + "bits_per_byte": 1.6680851841507087, + "learning_rate": 0.00015329574513790726, + "loss_nats": 1.1562285423278809, + "step": 4410, + "unclipped_grad_norm": 0.6488450169563293 + }, + { + "bits_per_byte": 1.6133293477690076, + "learning_rate": 0.00015273990190800186, + "loss_nats": 1.1182746887207031, + "step": 4420, + "unclipped_grad_norm": 0.6429536938667297 + }, + { + "bits_per_byte": 1.6717196935113925, + "learning_rate": 0.0001521842681693461, + "loss_nats": 1.1587477922439575, + "step": 4430, + "unclipped_grad_norm": 0.6755688190460205 + }, + { + "bits_per_byte": 1.6113522352146887, + "learning_rate": 0.00015162885341618752, + "loss_nats": 1.1169042587280273, + "step": 4440, + "unclipped_grad_norm": 0.6762297749519348 + }, + { + "bits_per_byte": 1.594701562888755, + "learning_rate": 0.00015107366713903178, + "loss_nats": 1.105362892150879, + "step": 4450, + "unclipped_grad_norm": 0.6539916396141052 + }, + { + "bits_per_byte": 1.574604014274229, + "learning_rate": 0.0001505187188244806, + "loss_nats": 1.0914323329925537, + "step": 4460, + "unclipped_grad_norm": 0.6519320011138916 + }, + { + "bits_per_byte": 1.6233662552733612, + "learning_rate": 0.00014996401795506952, + "loss_nats": 1.1252317428588867, + "step": 4470, + "unclipped_grad_norm": 0.6663927435874939 + }, + { + "bits_per_byte": 1.651233980053869, + "learning_rate": 0.00014940957400910585, + "loss_nats": 1.1445481777191162, + "step": 4480, + "unclipped_grad_norm": 0.6907956004142761 + }, + { + "bits_per_byte": 1.6254278113092537, + "learning_rate": 0.00014885539646050699, + "loss_nats": 1.126660704612732, + "step": 4490, + "unclipped_grad_norm": 0.6732251048088074 + }, + { + "bits_per_byte": 1.6504411400333914, + "learning_rate": 0.00014830149477863805, + "loss_nats": 1.143998622894287, + "step": 4500, + "unclipped_grad_norm": 0.6376641988754272 + }, + { + "bits_per_byte": 1.5969390571764803, + "learning_rate": 0.00014774787842815066, + "loss_nats": 1.1069138050079346, + "step": 4510, + "unclipped_grad_norm": 0.6637102961540222 + }, + { + "bits_per_byte": 1.6246974009910393, + "learning_rate": 0.00014719455686882066, + "loss_nats": 1.1261544227600098, + "step": 4520, + "unclipped_grad_norm": 0.6717531681060791 + }, + { + "bits_per_byte": 1.6581115662618726, + "learning_rate": 0.00014664153955538695, + "loss_nats": 1.149315357208252, + "step": 4530, + "unclipped_grad_norm": 0.6531894207000732 + }, + { + "bits_per_byte": 1.7167389040971464, + "learning_rate": 0.00014608883593738946, + "loss_nats": 1.1899527311325073, + "step": 4540, + "unclipped_grad_norm": 0.703811764717102 + }, + { + "bits_per_byte": 1.5547266034528073, + "learning_rate": 0.00014553645545900817, + "loss_nats": 1.0776543617248535, + "step": 4550, + "unclipped_grad_norm": 0.6300640106201172 + }, + { + "bits_per_byte": 1.6345066034656506, + "learning_rate": 0.0001449844075589013, + "loss_nats": 1.1329536437988281, + "step": 4560, + "unclipped_grad_norm": 0.723333477973938 + }, + { + "bits_per_byte": 1.6147346180092077, + "learning_rate": 0.0001444327016700444, + "loss_nats": 1.1192487478256226, + "step": 4570, + "unclipped_grad_norm": 0.6539926528930664 + }, + { + "bits_per_byte": 1.5679087296761398, + "learning_rate": 0.0001438813472195688, + "loss_nats": 1.0867915153503418, + "step": 4580, + "unclipped_grad_norm": 0.6434015035629272 + }, + { + "bits_per_byte": 1.6062840784764127, + "learning_rate": 0.00014333035362860093, + "loss_nats": 1.1133912801742554, + "step": 4590, + "unclipped_grad_norm": 0.6853119134902954 + }, + { + "bits_per_byte": 1.6093730588685595, + "learning_rate": 0.00014277973031210088, + "loss_nats": 1.115532398223877, + "step": 4600, + "unclipped_grad_norm": 0.670010507106781 + }, + { + "bits_per_byte": 1.6695516802146158, + "learning_rate": 0.00014222948667870207, + "loss_nats": 1.1572450399398804, + "step": 4610, + "unclipped_grad_norm": 0.7086074948310852 + }, + { + "bits_per_byte": 1.6683496934677358, + "learning_rate": 0.00014167963213054991, + "loss_nats": 1.15641188621521, + "step": 4620, + "unclipped_grad_norm": 0.6766663789749146 + }, + { + "bits_per_byte": 1.639615692074832, + "learning_rate": 0.00014113017606314152, + "loss_nats": 1.1364949941635132, + "step": 4630, + "unclipped_grad_norm": 0.6698880195617676 + }, + { + "bits_per_byte": 1.6970934978717387, + "learning_rate": 0.00014058112786516518, + "loss_nats": 1.1763355731964111, + "step": 4640, + "unclipped_grad_norm": 0.703077495098114 + }, + { + "bits_per_byte": 1.5958786121512603, + "learning_rate": 0.0001400324969183396, + "loss_nats": 1.1061787605285645, + "step": 4650, + "unclipped_grad_norm": 0.6709136962890625 + }, + { + "bits_per_byte": 1.5661840876532878, + "learning_rate": 0.00013948429259725406, + "loss_nats": 1.0855960845947266, + "step": 4660, + "unclipped_grad_norm": 0.6643497347831726 + }, + { + "bits_per_byte": 1.7242953058281056, + "learning_rate": 0.00013893652426920776, + "loss_nats": 1.1951904296875, + "step": 4670, + "unclipped_grad_norm": 0.7115020751953125 + }, + { + "bits_per_byte": 1.70608750258126, + "learning_rate": 0.00013838920129405016, + "loss_nats": 1.1825697422027588, + "step": 4680, + "unclipped_grad_norm": 0.7020435333251953 + }, + { + "bits_per_byte": 1.6853506944534768, + "learning_rate": 0.0001378423330240207, + "loss_nats": 1.1681960821151733, + "step": 4690, + "unclipped_grad_norm": 0.7577327489852905 + }, + { + "bits_per_byte": 1.5994532715494436, + "learning_rate": 0.00013729592880358936, + "loss_nats": 1.1086565256118774, + "step": 4700, + "unclipped_grad_norm": 0.6607687473297119 + }, + { + "bits_per_byte": 1.6667487069708538, + "learning_rate": 0.00013674999796929659, + "loss_nats": 1.1553021669387817, + "step": 4710, + "unclipped_grad_norm": 0.7177369594573975 + }, + { + "bits_per_byte": 1.6056974456543196, + "learning_rate": 0.00013620454984959417, + "loss_nats": 1.1129846572875977, + "step": 4720, + "unclipped_grad_norm": 0.6873557567596436 + }, + { + "bits_per_byte": 1.5233628153455605, + "learning_rate": 0.00013565959376468542, + "loss_nats": 1.0559146404266357, + "step": 4730, + "unclipped_grad_norm": 0.6391798853874207 + }, + { + "bits_per_byte": 1.5809453585768425, + "learning_rate": 0.00013511513902636634, + "loss_nats": 1.0958278179168701, + "step": 4740, + "unclipped_grad_norm": 0.6574237942695618 + }, + { + "bits_per_byte": 1.630707678690742, + "learning_rate": 0.0001345711949378662, + "loss_nats": 1.130320429801941, + "step": 4750, + "unclipped_grad_norm": 0.6879438161849976 + }, + { + "bits_per_byte": 1.6100228093235194, + "learning_rate": 0.00013402777079368853, + "loss_nats": 1.1159827709197998, + "step": 4760, + "unclipped_grad_norm": 0.6870896816253662 + }, + { + "bits_per_byte": 1.5769849420927737, + "learning_rate": 0.0001334848758794528, + "loss_nats": 1.0930826663970947, + "step": 4770, + "unclipped_grad_norm": 0.6955029368400574 + }, + { + "bits_per_byte": 1.626970495687493, + "learning_rate": 0.00013294251947173504, + "loss_nats": 1.1277300119400024, + "step": 4780, + "unclipped_grad_norm": 0.703714907169342 + }, + { + "bits_per_byte": 1.568998067786705, + "learning_rate": 0.00013240071083790996, + "loss_nats": 1.0875465869903564, + "step": 4790, + "unclipped_grad_norm": 0.6794602274894714 + }, + { + "bits_per_byte": 1.5737296544772426, + "learning_rate": 0.0001318594592359921, + "loss_nats": 1.0908262729644775, + "step": 4800, + "unclipped_grad_norm": 0.7120228409767151 + }, + { + "bits_per_byte": 1.6262536719986969, + "learning_rate": 0.0001313187739144781, + "loss_nats": 1.1272331476211548, + "step": 4810, + "unclipped_grad_norm": 0.7055649757385254 + }, + { + "bits_per_byte": 1.5716624230138716, + "learning_rate": 0.0001307786641121882, + "loss_nats": 1.0893933773040771, + "step": 4820, + "unclipped_grad_norm": 0.6697573065757751 + }, + { + "bits_per_byte": 1.6060875023064765, + "learning_rate": 0.00013023913905810876, + "loss_nats": 1.1132550239562988, + "step": 4830, + "unclipped_grad_norm": 0.710551381111145 + }, + { + "bits_per_byte": 1.5948934955271181, + "learning_rate": 0.00012970020797123432, + "loss_nats": 1.1054959297180176, + "step": 4840, + "unclipped_grad_norm": 0.7147494554519653 + }, + { + "bits_per_byte": 1.6218992432615014, + "learning_rate": 0.00012916188006041027, + "loss_nats": 1.1242148876190186, + "step": 4850, + "unclipped_grad_norm": 0.6700032353401184 + }, + { + "bits_per_byte": 1.5424350033956424, + "learning_rate": 0.00012862416452417516, + "loss_nats": 1.0691344738006592, + "step": 4860, + "unclipped_grad_norm": 0.6617462635040283 + }, + { + "bits_per_byte": 1.626479141253978, + "learning_rate": 0.00012808707055060405, + "loss_nats": 1.1273894309997559, + "step": 4870, + "unclipped_grad_norm": 0.7252988815307617 + }, + { + "bits_per_byte": 1.483838622437524, + "learning_rate": 0.000127550607317151, + "loss_nats": 1.028518557548523, + "step": 4880, + "unclipped_grad_norm": 0.6700296998023987 + }, + { + "bits_per_byte": 1.540047540236365, + "learning_rate": 0.00012701478399049248, + "loss_nats": 1.0674796104431152, + "step": 4890, + "unclipped_grad_norm": 0.6948246359825134 + }, + { + "bits_per_byte": 1.5882991647450855, + "learning_rate": 0.0001264796097263708, + "loss_nats": 1.100925087928772, + "step": 4900, + "unclipped_grad_norm": 0.7125248908996582 + }, + { + "bits_per_byte": 1.6945509063613828, + "learning_rate": 0.0001259450936694375, + "loss_nats": 1.1745731830596924, + "step": 4910, + "unclipped_grad_norm": 0.7282727956771851 + }, + { + "bits_per_byte": 1.620069347856321, + "learning_rate": 0.00012541124495309727, + "loss_nats": 1.1229465007781982, + "step": 4920, + "unclipped_grad_norm": 0.6988062858581543 + }, + { + "bits_per_byte": 1.577541305968315, + "learning_rate": 0.00012487807269935167, + "loss_nats": 1.0934683084487915, + "step": 4930, + "unclipped_grad_norm": 0.6639720797538757 + }, + { + "bits_per_byte": 1.5697768052298118, + "learning_rate": 0.00012434558601864349, + "loss_nats": 1.0880863666534424, + "step": 4940, + "unclipped_grad_norm": 0.6598480939865112 + }, + { + "bits_per_byte": 1.5230164422867227, + "learning_rate": 0.0001238137940097009, + "loss_nats": 1.0556745529174805, + "step": 4950, + "unclipped_grad_norm": 0.653609037399292 + }, + { + "bits_per_byte": 1.5246480416954582, + "learning_rate": 0.0001232827057593821, + "loss_nats": 1.0568054914474487, + "step": 4960, + "unclipped_grad_norm": 0.6958068013191223 + }, + { + "bits_per_byte": 1.5995143253905, + "learning_rate": 0.00012275233034251992, + "loss_nats": 1.108698844909668, + "step": 4970, + "unclipped_grad_norm": 0.7115789651870728 + }, + { + "bits_per_byte": 1.650496862412271, + "learning_rate": 0.0001222226768217669, + "loss_nats": 1.1440372467041016, + "step": 4980, + "unclipped_grad_norm": 0.7071532607078552 + }, + { + "bits_per_byte": 1.5967141038691517, + "learning_rate": 0.00012169375424744034, + "loss_nats": 1.1067578792572021, + "step": 4990, + "unclipped_grad_norm": 0.7384892106056213 + }, + { + "bits_per_byte": 1.5399667083904593, + "learning_rate": 0.00012116557165736772, + "loss_nats": 1.0674235820770264, + "step": 5000, + "unclipped_grad_norm": 0.6687040328979492 + }, + { + "bits_per_byte": 1.5343267093380644, + "learning_rate": 0.00012063813807673214, + "loss_nats": 1.063514232635498, + "step": 5010, + "unclipped_grad_norm": 0.6622800827026367 + }, + { + "bits_per_byte": 1.5808822409439758, + "learning_rate": 0.00012011146251791832, + "loss_nats": 1.095784068107605, + "step": 5020, + "unclipped_grad_norm": 0.6676583886146545 + }, + { + "bits_per_byte": 1.5838780067393552, + "learning_rate": 0.0001195855539803584, + "loss_nats": 1.09786057472229, + "step": 5030, + "unclipped_grad_norm": 0.7090701460838318 + }, + { + "bits_per_byte": 1.541603123313419, + "learning_rate": 0.00011906042145037817, + "loss_nats": 1.068557858467102, + "step": 5040, + "unclipped_grad_norm": 0.6795203685760498 + }, + { + "bits_per_byte": 1.507806296644416, + "learning_rate": 0.0001185360739010438, + "loss_nats": 1.0451316833496094, + "step": 5050, + "unclipped_grad_norm": 0.6525983214378357 + }, + { + "bits_per_byte": 1.5288581769885796, + "learning_rate": 0.00011801252029200808, + "loss_nats": 1.0597237348556519, + "step": 5060, + "unclipped_grad_norm": 0.651759684085846 + }, + { + "bits_per_byte": 1.6122224674280545, + "learning_rate": 0.00011748976956935777, + "loss_nats": 1.1175074577331543, + "step": 5070, + "unclipped_grad_norm": 0.6996030807495117 + }, + { + "bits_per_byte": 1.5728480714089197, + "learning_rate": 0.00011696783066546024, + "loss_nats": 1.0902152061462402, + "step": 5080, + "unclipped_grad_norm": 0.7345776557922363 + }, + { + "bits_per_byte": 1.5561607667783524, + "learning_rate": 0.00011644671249881147, + "loss_nats": 1.0786484479904175, + "step": 5090, + "unclipped_grad_norm": 0.6930875182151794 + }, + { + "bits_per_byte": 1.5598473868822476, + "learning_rate": 0.00011592642397388293, + "loss_nats": 1.081203818321228, + "step": 5100, + "unclipped_grad_norm": 0.6548681259155273 + }, + { + "bits_per_byte": 1.4730458511826285, + "learning_rate": 0.00011540697398097015, + "loss_nats": 1.0210375785827637, + "step": 5110, + "unclipped_grad_norm": 0.6644712686538696 + }, + { + "bits_per_byte": 1.5811238765684381, + "learning_rate": 0.00011488837139604016, + "loss_nats": 1.0959515571594238, + "step": 5120, + "unclipped_grad_norm": 0.6931653022766113 + }, + { + "bits_per_byte": 1.5844427977647886, + "learning_rate": 0.00011437062508058039, + "loss_nats": 1.0982520580291748, + "step": 5130, + "unclipped_grad_norm": 0.686448872089386 + }, + { + "bits_per_byte": 1.5481269412085978, + "learning_rate": 0.00011385374388144673, + "loss_nats": 1.0730798244476318, + "step": 5140, + "unclipped_grad_norm": 0.679078996181488 + }, + { + "bits_per_byte": 1.6151162475114722, + "learning_rate": 0.00011333773663071286, + "loss_nats": 1.1195132732391357, + "step": 5150, + "unclipped_grad_norm": 0.7298036813735962 + }, + { + "bits_per_byte": 1.5006640291367344, + "learning_rate": 0.00011282261214551897, + "loss_nats": 1.040181040763855, + "step": 5160, + "unclipped_grad_norm": 0.7023690342903137 + }, + { + "bits_per_byte": 1.6822286933923642, + "learning_rate": 0.00011230837922792112, + "loss_nats": 1.166032075881958, + "step": 5170, + "unclipped_grad_norm": 0.7808181643486023 + }, + { + "bits_per_byte": 1.5830977214523907, + "learning_rate": 0.0001117950466647412, + "loss_nats": 1.0973197221755981, + "step": 5180, + "unclipped_grad_norm": 0.7051005363464355 + }, + { + "bits_per_byte": 1.5177551490315278, + "learning_rate": 0.00011128262322741627, + "loss_nats": 1.052027702331543, + "step": 5190, + "unclipped_grad_norm": 0.6638994216918945 + }, + { + "bits_per_byte": 1.5519999905060284, + "learning_rate": 0.00011077111767184913, + "loss_nats": 1.0757644176483154, + "step": 5200, + "unclipped_grad_norm": 0.7110773921012878 + }, + { + "bits_per_byte": 1.553785514387286, + "learning_rate": 0.00011026053873825837, + "loss_nats": 1.0770020484924316, + "step": 5210, + "unclipped_grad_norm": 0.6727936863899231 + }, + { + "bits_per_byte": 1.6304519404889088, + "learning_rate": 0.0001097508951510293, + "loss_nats": 1.130143165588379, + "step": 5220, + "unclipped_grad_norm": 0.737336277961731 + }, + { + "bits_per_byte": 1.5485336801778884, + "learning_rate": 0.0001092421956185646, + "loss_nats": 1.0733617544174194, + "step": 5230, + "unclipped_grad_norm": 0.7087617516517639 + }, + { + "bits_per_byte": 1.5433107390538359, + "learning_rate": 0.00010873444883313581, + "loss_nats": 1.0697414875030518, + "step": 5240, + "unclipped_grad_norm": 0.6938673257827759 + }, + { + "bits_per_byte": 1.5172201110046935, + "learning_rate": 0.00010822766347073447, + "loss_nats": 1.0516568422317505, + "step": 5250, + "unclipped_grad_norm": 0.677476167678833 + }, + { + "bits_per_byte": 1.5285298621080827, + "learning_rate": 0.00010772184819092424, + "loss_nats": 1.0594961643218994, + "step": 5260, + "unclipped_grad_norm": 0.7000418305397034 + }, + { + "bits_per_byte": 1.530004785321882, + "learning_rate": 0.0001072170116366926, + "loss_nats": 1.060518503189087, + "step": 5270, + "unclipped_grad_norm": 0.6852406859397888 + }, + { + "bits_per_byte": 1.5473502675573014, + "learning_rate": 0.00010671316243430345, + "loss_nats": 1.0725414752960205, + "step": 5280, + "unclipped_grad_norm": 0.7066041231155396 + }, + { + "bits_per_byte": 1.5522775705045209, + "learning_rate": 0.00010621030919314944, + "loss_nats": 1.0759568214416504, + "step": 5290, + "unclipped_grad_norm": 0.7406570315361023 + }, + { + "bits_per_byte": 1.5518569009405105, + "learning_rate": 0.000105708460505605, + "loss_nats": 1.0756652355194092, + "step": 5300, + "unclipped_grad_norm": 0.7126148343086243 + }, + { + "bits_per_byte": 1.5478675913710969, + "learning_rate": 0.00010520762494687964, + "loss_nats": 1.0729000568389893, + "step": 5310, + "unclipped_grad_norm": 0.6942803859710693 + }, + { + "bits_per_byte": 1.550509932818953, + "learning_rate": 0.00010470781107487111, + "loss_nats": 1.0747315883636475, + "step": 5320, + "unclipped_grad_norm": 0.6907455325126648 + }, + { + "bits_per_byte": 1.544284676805672, + "learning_rate": 0.00010420902743001955, + "loss_nats": 1.0704165697097778, + "step": 5330, + "unclipped_grad_norm": 0.7187883853912354 + }, + { + "bits_per_byte": 1.4951578325867105, + "learning_rate": 0.00010371128253516117, + "loss_nats": 1.0363644361495972, + "step": 5340, + "unclipped_grad_norm": 0.7044789791107178 + }, + { + "bits_per_byte": 1.580473266200224, + "learning_rate": 0.000103214584895383, + "loss_nats": 1.0955005884170532, + "step": 5350, + "unclipped_grad_norm": 0.726965606212616 + }, + { + "bits_per_byte": 1.5009686104114124, + "learning_rate": 0.00010271894299787715, + "loss_nats": 1.0403921604156494, + "step": 5360, + "unclipped_grad_norm": 0.7176401615142822 + }, + { + "bits_per_byte": 1.4711079506727063, + "learning_rate": 0.00010222436531179622, + "loss_nats": 1.0196943283081055, + "step": 5370, + "unclipped_grad_norm": 0.7109264135360718 + }, + { + "bits_per_byte": 1.5906493076691257, + "learning_rate": 0.0001017308602881082, + "loss_nats": 1.1025540828704834, + "step": 5380, + "unclipped_grad_norm": 0.7163623571395874 + }, + { + "bits_per_byte": 1.5261117860369515, + "learning_rate": 0.00010123843635945242, + "loss_nats": 1.0578200817108154, + "step": 5390, + "unclipped_grad_norm": 0.7170036435127258 + }, + { + "bits_per_byte": 1.5557079364586306, + "learning_rate": 0.00010074710193999505, + "loss_nats": 1.0783345699310303, + "step": 5400, + "unclipped_grad_norm": 0.7109150290489197 + }, + { + "bits_per_byte": 1.5523785243205774, + "learning_rate": 0.0001002568654252858, + "loss_nats": 1.0760267972946167, + "step": 5410, + "unclipped_grad_norm": 0.6841652989387512 + }, + { + "bits_per_byte": 1.5366292130678156, + "learning_rate": 9.976773519211404e-05, + "loss_nats": 1.065110206604004, + "step": 5420, + "unclipped_grad_norm": 0.7674379348754883 + }, + { + "bits_per_byte": 1.570237718734124, + "learning_rate": 9.92797195983659e-05, + "loss_nats": 1.0884058475494385, + "step": 5430, + "unclipped_grad_norm": 0.7482572793960571 + }, + { + "bits_per_byte": 1.5099595194332183, + "learning_rate": 9.879282698288142e-05, + "loss_nats": 1.0466241836547852, + "step": 5440, + "unclipped_grad_norm": 0.7148683667182922 + }, + { + "bits_per_byte": 1.5041766027979564, + "learning_rate": 9.830706566531197e-05, + "loss_nats": 1.0426157712936401, + "step": 5450, + "unclipped_grad_norm": 0.7205177545547485 + }, + { + "bits_per_byte": 1.4857185647941054, + "learning_rate": 9.782244394597826e-05, + "loss_nats": 1.0298216342926025, + "step": 5460, + "unclipped_grad_norm": 0.699296236038208 + }, + { + "bits_per_byte": 1.5412328446661114, + "learning_rate": 9.733897010572829e-05, + "loss_nats": 1.0683012008666992, + "step": 5470, + "unclipped_grad_norm": 0.7379934787750244 + }, + { + "bits_per_byte": 1.553612499840518, + "learning_rate": 9.685665240579612e-05, + "loss_nats": 1.0768821239471436, + "step": 5480, + "unclipped_grad_norm": 0.7247312664985657 + }, + { + "bits_per_byte": 1.5568433659196268, + "learning_rate": 9.637549908766041e-05, + "loss_nats": 1.0791215896606445, + "step": 5490, + "unclipped_grad_norm": 0.7230545282363892 + }, + { + "bits_per_byte": 1.5454637898599874, + "learning_rate": 9.589551837290389e-05, + "loss_nats": 1.071233868598938, + "step": 5500, + "unclipped_grad_norm": 0.7159186005592346 + }, + { + "bits_per_byte": 1.6399625810816223, + "learning_rate": 9.541671846307257e-05, + "loss_nats": 1.136735439300537, + "step": 5510, + "unclipped_grad_norm": 0.7487865686416626 + }, + { + "bits_per_byte": 1.5508425472657217, + "learning_rate": 9.493910753953597e-05, + "loss_nats": 1.0749621391296387, + "step": 5520, + "unclipped_grad_norm": 0.7227869629859924 + }, + { + "bits_per_byte": 1.507233250451741, + "learning_rate": 9.446269376334689e-05, + "loss_nats": 1.0447344779968262, + "step": 5530, + "unclipped_grad_norm": 0.7033931612968445 + }, + { + "bits_per_byte": 1.597289213853637, + "learning_rate": 9.39874852751024e-05, + "loss_nats": 1.10715651512146, + "step": 5540, + "unclipped_grad_norm": 0.7876261472702026 + }, + { + "bits_per_byte": 1.512559897114264, + "learning_rate": 9.351349019480433e-05, + "loss_nats": 1.048426628112793, + "step": 5550, + "unclipped_grad_norm": 0.7062801122665405 + }, + { + "bits_per_byte": 1.5152815225641678, + "learning_rate": 9.304071662172086e-05, + "loss_nats": 1.050313115119934, + "step": 5560, + "unclipped_grad_norm": 0.754061222076416 + }, + { + "bits_per_byte": 1.4831498319208185, + "learning_rate": 9.256917263424787e-05, + "loss_nats": 1.028041124343872, + "step": 5570, + "unclipped_grad_norm": 0.7236031293869019 + }, + { + "bits_per_byte": 1.5690197376007136, + "learning_rate": 9.209886628977114e-05, + "loss_nats": 1.0875616073608398, + "step": 5580, + "unclipped_grad_norm": 0.7374637722969055 + }, + { + "bits_per_byte": 1.5019833080515028, + "learning_rate": 9.162980562452843e-05, + "loss_nats": 1.041095495223999, + "step": 5590, + "unclipped_grad_norm": 0.7137759923934937 + }, + { + "bits_per_byte": 1.54801171283252, + "learning_rate": 9.116199865347238e-05, + "loss_nats": 1.0729999542236328, + "step": 5600, + "unclipped_grad_norm": 0.7259743809700012 + }, + { + "bits_per_byte": 1.5450353810766881, + "learning_rate": 9.069545337013333e-05, + "loss_nats": 1.070936918258667, + "step": 5610, + "unclipped_grad_norm": 0.6988793611526489 + }, + { + "bits_per_byte": 1.5537045105587297, + "learning_rate": 9.0230177746483e-05, + "loss_nats": 1.0769459009170532, + "step": 5620, + "unclipped_grad_norm": 0.72163325548172 + }, + { + "bits_per_byte": 1.5318818039733988, + "learning_rate": 8.976617973279797e-05, + "loss_nats": 1.0618195533752441, + "step": 5630, + "unclipped_grad_norm": 0.7131519317626953 + }, + { + "bits_per_byte": 1.5027740842801702, + "learning_rate": 8.930346725752419e-05, + "loss_nats": 1.0416436195373535, + "step": 5640, + "unclipped_grad_norm": 0.7095826864242554 + }, + { + "bits_per_byte": 1.5224679896131212, + "learning_rate": 8.88420482271411e-05, + "loss_nats": 1.055294394493103, + "step": 5650, + "unclipped_grad_norm": 0.7380594611167908 + }, + { + "bits_per_byte": 1.590806843777316, + "learning_rate": 8.838193052602691e-05, + "loss_nats": 1.102663278579712, + "step": 5660, + "unclipped_grad_norm": 0.7485193610191345 + }, + { + "bits_per_byte": 1.5054829830139103, + "learning_rate": 8.792312201632352e-05, + "loss_nats": 1.0435212850570679, + "step": 5670, + "unclipped_grad_norm": 0.7387163639068604 + }, + { + "bits_per_byte": 1.5299944663628302, + "learning_rate": 8.746563053780261e-05, + "loss_nats": 1.0605113506317139, + "step": 5680, + "unclipped_grad_norm": 0.7047367691993713 + }, + { + "bits_per_byte": 1.5861086217210458, + "learning_rate": 8.700946390773114e-05, + "loss_nats": 1.0994067192077637, + "step": 5690, + "unclipped_grad_norm": 0.7179912328720093 + }, + { + "bits_per_byte": 1.515916826476455, + "learning_rate": 8.655462992073833e-05, + "loss_nats": 1.0507534742355347, + "step": 5700, + "unclipped_grad_norm": 0.7373331785202026 + }, + { + "bits_per_byte": 1.527769698791269, + "learning_rate": 8.610113634868199e-05, + "loss_nats": 1.058969259262085, + "step": 5710, + "unclipped_grad_norm": 0.7480198740959167 + }, + { + "bits_per_byte": 1.5396189594704148, + "learning_rate": 8.564899094051613e-05, + "loss_nats": 1.0671825408935547, + "step": 5720, + "unclipped_grad_norm": 0.6888155341148376 + }, + { + "bits_per_byte": 1.5500142788191664, + "learning_rate": 8.519820142215816e-05, + "loss_nats": 1.074388027191162, + "step": 5730, + "unclipped_grad_norm": 0.7189749479293823 + }, + { + "bits_per_byte": 1.5236374716389882, + "learning_rate": 8.474877549635728e-05, + "loss_nats": 1.0561050176620483, + "step": 5740, + "unclipped_grad_norm": 0.7438164949417114 + }, + { + "bits_per_byte": 1.5411162404288263, + "learning_rate": 8.430072084256247e-05, + "loss_nats": 1.0682203769683838, + "step": 5750, + "unclipped_grad_norm": 0.794143795967102 + }, + { + "bits_per_byte": 1.5646985015151347, + "learning_rate": 8.385404511679161e-05, + "loss_nats": 1.084566354751587, + "step": 5760, + "unclipped_grad_norm": 0.7443080544471741 + }, + { + "bits_per_byte": 1.5111640859198614, + "learning_rate": 8.340875595150034e-05, + "loss_nats": 1.0474591255187988, + "step": 5770, + "unclipped_grad_norm": 0.7111071944236755 + }, + { + "bits_per_byte": 1.4933853793869185, + "learning_rate": 8.296486095545203e-05, + "loss_nats": 1.0351358652114868, + "step": 5780, + "unclipped_grad_norm": 0.7597989439964294 + }, + { + "bits_per_byte": 1.51288804001211, + "learning_rate": 8.252236771358729e-05, + "loss_nats": 1.0486540794372559, + "step": 5790, + "unclipped_grad_norm": 0.7139179110527039 + }, + { + "bits_per_byte": 1.48582313024583, + "learning_rate": 8.208128378689484e-05, + "loss_nats": 1.0298941135406494, + "step": 5800, + "unclipped_grad_norm": 0.7527741193771362 + }, + { + "bits_per_byte": 1.569064281107287, + "learning_rate": 8.16416167122819e-05, + "loss_nats": 1.0875924825668335, + "step": 5810, + "unclipped_grad_norm": 0.7431255578994751 + }, + { + "bits_per_byte": 1.4993827583878063, + "learning_rate": 8.120337400244578e-05, + "loss_nats": 1.0392929315567017, + "step": 5820, + "unclipped_grad_norm": 0.7169746160507202 + }, + { + "bits_per_byte": 1.504188125635564, + "learning_rate": 8.076656314574521e-05, + "loss_nats": 1.04262375831604, + "step": 5830, + "unclipped_grad_norm": 0.7262318134307861 + }, + { + "bits_per_byte": 1.5003941883575305, + "learning_rate": 8.03311916060725e-05, + "loss_nats": 1.0399940013885498, + "step": 5840, + "unclipped_grad_norm": 0.726586639881134 + }, + { + "bits_per_byte": 1.4721355470116115, + "learning_rate": 7.989726682272606e-05, + "loss_nats": 1.0204066038131714, + "step": 5850, + "unclipped_grad_norm": 0.7553265690803528 + }, + { + "bits_per_byte": 1.5378941454649118, + "learning_rate": 7.946479621028318e-05, + "loss_nats": 1.06598699092865, + "step": 5860, + "unclipped_grad_norm": 0.7790977358818054 + }, + { + "bits_per_byte": 1.5240519498275678, + "learning_rate": 7.903378715847343e-05, + "loss_nats": 1.0563923120498657, + "step": 5870, + "unclipped_grad_norm": 0.7502213716506958 + }, + { + "bits_per_byte": 1.5468687161348855, + "learning_rate": 7.860424703205224e-05, + "loss_nats": 1.0722076892852783, + "step": 5880, + "unclipped_grad_norm": 0.767290472984314 + }, + { + "bits_per_byte": 1.5495691877187334, + "learning_rate": 7.817618317067534e-05, + "loss_nats": 1.0740795135498047, + "step": 5890, + "unclipped_grad_norm": 0.7468048930168152 + }, + { + "bits_per_byte": 1.5445479822441428, + "learning_rate": 7.774960288877295e-05, + "loss_nats": 1.07059907913208, + "step": 5900, + "unclipped_grad_norm": 0.7380504012107849 + }, + { + "bits_per_byte": 1.5492126676834947, + "learning_rate": 7.732451347542522e-05, + "loss_nats": 1.073832392692566, + "step": 5910, + "unclipped_grad_norm": 0.7607431411743164 + }, + { + "bits_per_byte": 1.4937749200911228, + "learning_rate": 7.690092219423722e-05, + "loss_nats": 1.0354058742523193, + "step": 5920, + "unclipped_grad_norm": 0.7359480261802673 + }, + { + "bits_per_byte": 1.5842945487197448, + "learning_rate": 7.647883628321536e-05, + "loss_nats": 1.098149299621582, + "step": 5930, + "unclipped_grad_norm": 0.7712169289588928 + }, + { + "bits_per_byte": 1.449506241793734, + "learning_rate": 7.605826295464316e-05, + "loss_nats": 1.0047211647033691, + "step": 5940, + "unclipped_grad_norm": 0.7011276483535767 + }, + { + "bits_per_byte": 1.504274804891599, + "learning_rate": 7.563920939495851e-05, + "loss_nats": 1.0426838397979736, + "step": 5950, + "unclipped_grad_norm": 0.7238391041755676 + }, + { + "bits_per_byte": 1.5397549977472473, + "learning_rate": 7.522168276463047e-05, + "loss_nats": 1.0672768354415894, + "step": 5960, + "unclipped_grad_norm": 0.7776085734367371 + }, + { + "bits_per_byte": 1.5223966168130132, + "learning_rate": 7.480569019803719e-05, + "loss_nats": 1.0552449226379395, + "step": 5970, + "unclipped_grad_norm": 0.8004531264305115 + }, + { + "bits_per_byte": 1.5577928821350404, + "learning_rate": 7.439123880334388e-05, + "loss_nats": 1.0797797441482544, + "step": 5980, + "unclipped_grad_norm": 0.7434797883033752 + }, + { + "bits_per_byte": 1.5245451960702423, + "learning_rate": 7.39783356623813e-05, + "loss_nats": 1.0567342042922974, + "step": 5990, + "unclipped_grad_norm": 0.7621127367019653 + }, + { + "bits_per_byte": 1.4749999180617317, + "learning_rate": 7.356698783052497e-05, + "loss_nats": 1.0223920345306396, + "step": 6000, + "unclipped_grad_norm": 0.7376543283462524 + }, + { + "bits_per_byte": 1.5041539010880425, + "learning_rate": 7.315720233657431e-05, + "loss_nats": 1.0426000356674194, + "step": 6010, + "unclipped_grad_norm": 0.7570760846138 + }, + { + "bits_per_byte": 1.4634781123498286, + "learning_rate": 7.274898618263286e-05, + "loss_nats": 1.0144057273864746, + "step": 6020, + "unclipped_grad_norm": 0.7092445492744446 + }, + { + "bits_per_byte": 1.5053727421347072, + "learning_rate": 7.234234634398831e-05, + "loss_nats": 1.0434448719024658, + "step": 6030, + "unclipped_grad_norm": 0.7244347333908081 + }, + { + "bits_per_byte": 1.5688415635744197, + "learning_rate": 7.193728976899362e-05, + "loss_nats": 1.0874381065368652, + "step": 6040, + "unclipped_grad_norm": 0.7976362705230713 + }, + { + "bits_per_byte": 1.4774671811710096, + "learning_rate": 7.153382337894797e-05, + "loss_nats": 1.0241022109985352, + "step": 6050, + "unclipped_grad_norm": 0.7659919857978821 + }, + { + "bits_per_byte": 1.4645516280565143, + "learning_rate": 7.113195406797887e-05, + "loss_nats": 1.0151498317718506, + "step": 6060, + "unclipped_grad_norm": 0.7441014051437378 + }, + { + "bits_per_byte": 1.5433717928948922, + "learning_rate": 7.073168870292397e-05, + "loss_nats": 1.0697838068008423, + "step": 6070, + "unclipped_grad_norm": 0.7805195450782776 + }, + { + "bits_per_byte": 1.4776137103895446, + "learning_rate": 7.033303412321404e-05, + "loss_nats": 1.0242037773132324, + "step": 6080, + "unclipped_grad_norm": 0.7491018772125244 + }, + { + "bits_per_byte": 1.486596708209411, + "learning_rate": 6.993599714075586e-05, + "loss_nats": 1.0304303169250488, + "step": 6090, + "unclipped_grad_norm": 0.7174160480499268 + }, + { + "bits_per_byte": 1.47886454020927, + "learning_rate": 6.954058453981609e-05, + "loss_nats": 1.0250707864761353, + "step": 6100, + "unclipped_grad_norm": 0.7355735301971436 + }, + { + "bits_per_byte": 1.5482112127075207, + "learning_rate": 6.914680307690504e-05, + "loss_nats": 1.0731382369995117, + "step": 6110, + "unclipped_grad_norm": 0.7645819187164307 + }, + { + "bits_per_byte": 1.5315517692663931, + "learning_rate": 6.875465948066139e-05, + "loss_nats": 1.0615907907485962, + "step": 6120, + "unclipped_grad_norm": 0.7616561651229858 + }, + { + "bits_per_byte": 1.4671989570012456, + "learning_rate": 6.836416045173733e-05, + "loss_nats": 1.0169848203659058, + "step": 6130, + "unclipped_grad_norm": 0.7611680030822754 + }, + { + "bits_per_byte": 1.4875635946725616, + "learning_rate": 6.797531266268377e-05, + "loss_nats": 1.0311005115509033, + "step": 6140, + "unclipped_grad_norm": 0.7648262977600098 + }, + { + "bits_per_byte": 1.5445696520581516, + "learning_rate": 6.758812275783663e-05, + "loss_nats": 1.0706140995025635, + "step": 6150, + "unclipped_grad_norm": 0.7702162861824036 + }, + { + "bits_per_byte": 1.4100173053291247, + "learning_rate": 6.720259735320298e-05, + "loss_nats": 0.9773495197296143, + "step": 6160, + "unclipped_grad_norm": 0.7114855051040649 + }, + { + "bits_per_byte": 1.5173619966916552, + "learning_rate": 6.681874303634842e-05, + "loss_nats": 1.0517551898956299, + "step": 6170, + "unclipped_grad_norm": 0.7402294874191284 + }, + { + "bits_per_byte": 1.4772606300073232, + "learning_rate": 6.643656636628402e-05, + "loss_nats": 1.0239590406417847, + "step": 6180, + "unclipped_grad_norm": 0.7306315302848816 + }, + { + "bits_per_byte": 1.495765275309558, + "learning_rate": 6.605607387335466e-05, + "loss_nats": 1.0367854833602905, + "step": 6190, + "unclipped_grad_norm": 0.7585544586181641 + }, + { + "bits_per_byte": 1.5043707712107803, + "learning_rate": 6.567727205912724e-05, + "loss_nats": 1.042750358581543, + "step": 6200, + "unclipped_grad_norm": 0.747658908367157 + }, + { + "bits_per_byte": 1.5010081664211108, + "learning_rate": 6.530016739627959e-05, + "loss_nats": 1.040419578552246, + "step": 6210, + "unclipped_grad_norm": 0.7751714587211609 + }, + { + "bits_per_byte": 1.5035775872250012, + "learning_rate": 6.492476632848993e-05, + "loss_nats": 1.0422005653381348, + "step": 6220, + "unclipped_grad_norm": 0.7300167083740234 + }, + { + "bits_per_byte": 1.4580931635686634, + "learning_rate": 6.455107527032678e-05, + "loss_nats": 1.01067316532135, + "step": 6230, + "unclipped_grad_norm": 0.7508504390716553 + }, + { + "bits_per_byte": 1.4949628042606322, + "learning_rate": 6.417910060713926e-05, + "loss_nats": 1.0362292528152466, + "step": 6240, + "unclipped_grad_norm": 0.8087684512138367 + }, + { + "bits_per_byte": 1.5367451293744974, + "learning_rate": 6.380884869494799e-05, + "loss_nats": 1.0651905536651611, + "step": 6250, + "unclipped_grad_norm": 0.7689018845558167 + }, + { + "bits_per_byte": 1.4083784826490529, + "learning_rate": 6.34403258603367e-05, + "loss_nats": 0.9762135744094849, + "step": 6260, + "unclipped_grad_norm": 0.7644893527030945 + }, + { + "bits_per_byte": 1.5135271275427165, + "learning_rate": 6.30735384003437e-05, + "loss_nats": 1.0490970611572266, + "step": 6270, + "unclipped_grad_norm": 0.7905248999595642 + }, + { + "bits_per_byte": 1.5059574831476408, + "learning_rate": 6.270849258235484e-05, + "loss_nats": 1.0438501834869385, + "step": 6280, + "unclipped_grad_norm": 0.8002632856369019 + }, + { + "bits_per_byte": 1.4386743444857673, + "learning_rate": 6.234519464399581e-05, + "loss_nats": 0.9972130656242371, + "step": 6290, + "unclipped_grad_norm": 0.7424378395080566 + }, + { + "bits_per_byte": 1.5108663839512178, + "learning_rate": 6.198365079302614e-05, + "loss_nats": 1.0472527742385864, + "step": 6300, + "unclipped_grad_norm": 0.7815948724746704 + }, + { + "bits_per_byte": 1.4647595550814074, + "learning_rate": 6.16238672072326e-05, + "loss_nats": 1.0152939558029175, + "step": 6310, + "unclipped_grad_norm": 0.7460298538208008 + }, + { + "bits_per_byte": 1.420051375119739, + "learning_rate": 6.126585003432406e-05, + "loss_nats": 0.9843046069145203, + "step": 6320, + "unclipped_grad_norm": 0.7480984330177307 + }, + { + "bits_per_byte": 1.511718901951545, + "learning_rate": 6.090960539182614e-05, + "loss_nats": 1.0478436946868896, + "step": 6330, + "unclipped_grad_norm": 0.7749253511428833 + }, + { + "bits_per_byte": 1.4587121291291185, + "learning_rate": 6.0555139366976935e-05, + "loss_nats": 1.0111021995544434, + "step": 6340, + "unclipped_grad_norm": 0.750465989112854 + }, + { + "bits_per_byte": 1.5740031068921143, + "learning_rate": 6.020245801662274e-05, + "loss_nats": 1.0910158157348633, + "step": 6350, + "unclipped_grad_norm": 0.8142428994178772 + }, + { + "bits_per_byte": 1.4492702815967504, + "learning_rate": 5.985156736711483e-05, + "loss_nats": 1.0045576095581055, + "step": 6360, + "unclipped_grad_norm": 0.804401695728302 + }, + { + "bits_per_byte": 1.4448909153751799, + "learning_rate": 5.950247341420618e-05, + "loss_nats": 1.0015220642089844, + "step": 6370, + "unclipped_grad_norm": 0.7500267028808594 + }, + { + "bits_per_byte": 1.4666350258890664, + "learning_rate": 5.915518212294935e-05, + "loss_nats": 1.0165939331054688, + "step": 6380, + "unclipped_grad_norm": 0.7499048709869385 + }, + { + "bits_per_byte": 1.5287405408553896, + "learning_rate": 5.880969942759425e-05, + "loss_nats": 1.0596421957015991, + "step": 6390, + "unclipped_grad_norm": 0.7983035445213318 + }, + { + "bits_per_byte": 1.5132952949293532, + "learning_rate": 5.846603123148688e-05, + "loss_nats": 1.048936367034912, + "step": 6400, + "unclipped_grad_norm": 0.7945475578308105 + }, + { + "bits_per_byte": 1.4842119967725471, + "learning_rate": 5.812418340696858e-05, + "loss_nats": 1.0287773609161377, + "step": 6410, + "unclipped_grad_norm": 0.8107759952545166 + }, + { + "bits_per_byte": 1.4670899200005985, + "learning_rate": 5.778416179527541e-05, + "loss_nats": 1.0169092416763306, + "step": 6420, + "unclipped_grad_norm": 0.791615903377533 + }, + { + "bits_per_byte": 1.5136435597973505, + "learning_rate": 5.7445972206438635e-05, + "loss_nats": 1.0491777658462524, + "step": 6430, + "unclipped_grad_norm": 0.7751617431640625 + }, + { + "bits_per_byte": 1.4515793206672345, + "learning_rate": 5.710962041918516e-05, + "loss_nats": 1.0061581134796143, + "step": 6440, + "unclipped_grad_norm": 0.7813049554824829 + }, + { + "bits_per_byte": 1.523892521910218, + "learning_rate": 5.677511218083908e-05, + "loss_nats": 1.0562818050384521, + "step": 6450, + "unclipped_grad_norm": 0.8193799257278442 + }, + { + "bits_per_byte": 1.5827926242297599, + "learning_rate": 5.644245320722316e-05, + "loss_nats": 1.097108244895935, + "step": 6460, + "unclipped_grad_norm": 0.815384566783905 + }, + { + "bits_per_byte": 1.5468171213396267, + "learning_rate": 5.61116491825615e-05, + "loss_nats": 1.072171926498413, + "step": 6470, + "unclipped_grad_norm": 0.8593401312828064 + }, + { + "bits_per_byte": 1.4332476039204423, + "learning_rate": 5.5782705759382104e-05, + "loss_nats": 0.9934515357017517, + "step": 6480, + "unclipped_grad_norm": 0.7557542324066162 + }, + { + "bits_per_byte": 1.538845381506834, + "learning_rate": 5.5455628558420546e-05, + "loss_nats": 1.0666463375091553, + "step": 6490, + "unclipped_grad_norm": 0.802010178565979 + }, + { + "bits_per_byte": 1.512460835107367, + "learning_rate": 5.513042316852368e-05, + "loss_nats": 1.0483579635620117, + "step": 6500, + "unclipped_grad_norm": 0.811919093132019 + }, + { + "bits_per_byte": 1.4542873594877201, + "learning_rate": 5.480709514655443e-05, + "loss_nats": 1.0080351829528809, + "step": 6510, + "unclipped_grad_norm": 0.7658334374427795 + }, + { + "bits_per_byte": 1.413757584020089, + "learning_rate": 5.448565001729653e-05, + "loss_nats": 0.9799420833587646, + "step": 6520, + "unclipped_grad_norm": 0.7362275719642639 + }, + { + "bits_per_byte": 1.458498526676747, + "learning_rate": 5.41660932733604e-05, + "loss_nats": 1.0109541416168213, + "step": 6530, + "unclipped_grad_norm": 0.7702703475952148 + }, + { + "bits_per_byte": 1.4985250809079536, + "learning_rate": 5.384843037508902e-05, + "loss_nats": 1.038698434829712, + "step": 6540, + "unclipped_grad_norm": 0.8070250749588013 + }, + { + "bits_per_byte": 1.4283477322060354, + "learning_rate": 5.353266675046498e-05, + "loss_nats": 0.9900552034378052, + "step": 6550, + "unclipped_grad_norm": 0.7630698680877686 + }, + { + "bits_per_byte": 1.5059423486743648, + "learning_rate": 5.321880779501729e-05, + "loss_nats": 1.043839693069458, + "step": 6560, + "unclipped_grad_norm": 0.7847185730934143 + }, + { + "bits_per_byte": 1.4655090554738677, + "learning_rate": 5.290685887172962e-05, + "loss_nats": 1.0158134698867798, + "step": 6570, + "unclipped_grad_norm": 0.7837479710578918 + }, + { + "bits_per_byte": 1.5058788870761963, + "learning_rate": 5.2596825310948295e-05, + "loss_nats": 1.0437957048416138, + "step": 6580, + "unclipped_grad_norm": 0.8260042667388916 + }, + { + "bits_per_byte": 1.4850650307208266, + "learning_rate": 5.228871241029154e-05, + "loss_nats": 1.0293686389923096, + "step": 6590, + "unclipped_grad_norm": 0.7737330794334412 + }, + { + "bits_per_byte": 1.4176852378091684, + "learning_rate": 5.198252543455865e-05, + "loss_nats": 0.9826645255088806, + "step": 6600, + "unclipped_grad_norm": 0.7685086131095886 + }, + { + "bits_per_byte": 1.4428018421151492, + "learning_rate": 5.167826961564027e-05, + "loss_nats": 1.000074028968811, + "step": 6610, + "unclipped_grad_norm": 0.7548773884773254 + }, + { + "bits_per_byte": 1.4245123471091439, + "learning_rate": 5.137595015242884e-05, + "loss_nats": 0.9873967170715332, + "step": 6620, + "unclipped_grad_norm": 0.7893918752670288 + }, + { + "bits_per_byte": 1.463975486176124, + "learning_rate": 5.1075572210729925e-05, + "loss_nats": 1.0147504806518555, + "step": 6630, + "unclipped_grad_norm": 0.7861996293067932 + }, + { + "bits_per_byte": 1.5099068927420543, + "learning_rate": 5.07771409231737e-05, + "loss_nats": 1.0465877056121826, + "step": 6640, + "unclipped_grad_norm": 0.8437765836715698 + }, + { + "bits_per_byte": 1.5082221506942022, + "learning_rate": 5.048066138912756e-05, + "loss_nats": 1.0454199314117432, + "step": 6650, + "unclipped_grad_norm": 0.7631747126579285 + }, + { + "bits_per_byte": 1.5093254193994872, + "learning_rate": 5.018613867460863e-05, + "loss_nats": 1.0461846590042114, + "step": 6660, + "unclipped_grad_norm": 0.7733337879180908 + }, + { + "bits_per_byte": 1.4808910117843863, + "learning_rate": 4.989357781219755e-05, + "loss_nats": 1.026475429534912, + "step": 6670, + "unclipped_grad_norm": 0.8145992159843445 + }, + { + "bits_per_byte": 1.4583311875574574, + "learning_rate": 4.960298380095222e-05, + "loss_nats": 1.0108381509780884, + "step": 6680, + "unclipped_grad_norm": 0.7763416767120361 + }, + { + "bits_per_byte": 1.48509357984087, + "learning_rate": 4.931436160632256e-05, + "loss_nats": 1.029388427734375, + "step": 6690, + "unclipped_grad_norm": 0.8123944401741028 + }, + { + "bits_per_byte": 1.540551449403393, + "learning_rate": 4.9027716160065485e-05, + "loss_nats": 1.067828893661499, + "step": 6700, + "unclipped_grad_norm": 0.8398496508598328 + }, + { + "bits_per_byte": 1.5131737031951933, + "learning_rate": 4.874305236016084e-05, + "loss_nats": 1.0488520860671997, + "step": 6710, + "unclipped_grad_norm": 0.782952606678009 + }, + { + "bits_per_byte": 1.525510706672186, + "learning_rate": 4.846037507072752e-05, + "loss_nats": 1.0574034452438354, + "step": 6720, + "unclipped_grad_norm": 0.8299216628074646 + }, + { + "bits_per_byte": 1.4785902278811438, + "learning_rate": 4.817968912194049e-05, + "loss_nats": 1.0248806476593018, + "step": 6730, + "unclipped_grad_norm": 0.7975350618362427 + }, + { + "bits_per_byte": 1.4183937203393977, + "learning_rate": 4.790099930994813e-05, + "loss_nats": 0.9831556081771851, + "step": 6740, + "unclipped_grad_norm": 0.7755722999572754 + }, + { + "bits_per_byte": 1.4976699831678637, + "learning_rate": 4.762431039679047e-05, + "loss_nats": 1.0381057262420654, + "step": 6750, + "unclipped_grad_norm": 0.8062617182731628 + }, + { + "bits_per_byte": 1.4670971432719349, + "learning_rate": 4.734962711031755e-05, + "loss_nats": 1.0169142484664917, + "step": 6760, + "unclipped_grad_norm": 0.785228431224823 + }, + { + "bits_per_byte": 1.3873008049072597, + "learning_rate": 4.707695414410892e-05, + "loss_nats": 0.9616036415100098, + "step": 6770, + "unclipped_grad_norm": 0.7726664543151855 + }, + { + "bits_per_byte": 1.4705109988915614, + "learning_rate": 4.680629615739322e-05, + "loss_nats": 1.0192805528640747, + "step": 6780, + "unclipped_grad_norm": 0.8086632490158081 + }, + { + "bits_per_byte": 1.4429045157577143, + "learning_rate": 4.653765777496863e-05, + "loss_nats": 1.0001451969146729, + "step": 6790, + "unclipped_grad_norm": 0.818213701248169 + }, + { + "bits_per_byte": 1.5045286512842726, + "learning_rate": 4.6271043587123986e-05, + "loss_nats": 1.0428597927093506, + "step": 6800, + "unclipped_grad_norm": 0.8267188668251038 + }, + { + "bits_per_byte": 1.501266140397405, + "learning_rate": 4.600645814956006e-05, + "loss_nats": 1.0405983924865723, + "step": 6810, + "unclipped_grad_norm": 0.8420217633247375 + }, + { + "bits_per_byte": 1.4878318876079075, + "learning_rate": 4.574390598331204e-05, + "loss_nats": 1.0312864780426025, + "step": 6820, + "unclipped_grad_norm": 0.8419885039329529 + }, + { + "bits_per_byte": 1.4652020664420775, + "learning_rate": 4.548339157467199e-05, + "loss_nats": 1.0156006813049316, + "step": 6830, + "unclipped_grad_norm": 0.782660722732544 + }, + { + "bits_per_byte": 1.4358041260255179, + "learning_rate": 4.52249193751124e-05, + "loss_nats": 0.9952235817909241, + "step": 6840, + "unclipped_grad_norm": 0.8103845119476318 + }, + { + "bits_per_byte": 1.4906290134415403, + "learning_rate": 4.4968493801209985e-05, + "loss_nats": 1.0332252979278564, + "step": 6850, + "unclipped_grad_norm": 0.8234423995018005 + }, + { + "bits_per_byte": 1.4584503715345054, + "learning_rate": 4.471411923457034e-05, + "loss_nats": 1.010920763015747, + "step": 6860, + "unclipped_grad_norm": 0.7921655178070068 + }, + { + "bits_per_byte": 1.4079278881037924, + "learning_rate": 4.446180002175291e-05, + "loss_nats": 0.9759012460708618, + "step": 6870, + "unclipped_grad_norm": 0.7879679799079895 + }, + { + "bits_per_byte": 1.4277146640682095, + "learning_rate": 4.421154047419693e-05, + "loss_nats": 0.9896163940429688, + "step": 6880, + "unclipped_grad_norm": 0.7540534734725952 + }, + { + "bits_per_byte": 1.4341213617781507, + "learning_rate": 4.396334486814753e-05, + "loss_nats": 0.9940571784973145, + "step": 6890, + "unclipped_grad_norm": 0.7987163662910461 + }, + { + "bits_per_byte": 1.5343765843068147, + "learning_rate": 4.371721744458285e-05, + "loss_nats": 1.0635488033294678, + "step": 6900, + "unclipped_grad_norm": 0.8171630501747131 + }, + { + "bits_per_byte": 1.492606469961161, + "learning_rate": 4.347316240914145e-05, + "loss_nats": 1.0345959663391113, + "step": 6910, + "unclipped_grad_norm": 0.8321383595466614 + }, + { + "bits_per_byte": 1.4313504632987746, + "learning_rate": 4.3231183932050546e-05, + "loss_nats": 0.992136538028717, + "step": 6920, + "unclipped_grad_norm": 0.8009306192398071 + }, + { + "bits_per_byte": 1.4367294786784852, + "learning_rate": 4.299128614805464e-05, + "loss_nats": 0.995864987373352, + "step": 6930, + "unclipped_grad_norm": 0.8468543291091919 + }, + { + "bits_per_byte": 1.4102267801978756, + "learning_rate": 4.2753473156344965e-05, + "loss_nats": 0.9774947166442871, + "step": 6940, + "unclipped_grad_norm": 0.7895923256874084 + }, + { + "bits_per_byte": 1.4339222058684515, + "learning_rate": 4.251774902048942e-05, + "loss_nats": 0.9939191341400146, + "step": 6950, + "unclipped_grad_norm": 0.8116638660430908 + }, + { + "bits_per_byte": 1.4886653155339888, + "learning_rate": 4.228411776836306e-05, + "loss_nats": 1.0318641662597656, + "step": 6960, + "unclipped_grad_norm": 0.8071072101593018 + }, + { + "bits_per_byte": 1.4701075275926374, + "learning_rate": 4.2052583392079427e-05, + "loss_nats": 1.0190008878707886, + "step": 6970, + "unclipped_grad_norm": 0.8008604645729065 + }, + { + "bits_per_byte": 1.4455920886427476, + "learning_rate": 4.182314984792214e-05, + "loss_nats": 1.002008080482483, + "step": 6980, + "unclipped_grad_norm": 0.7700145840644836 + }, + { + "bits_per_byte": 1.484166077404767, + "learning_rate": 4.159582105627748e-05, + "loss_nats": 1.0287455320358276, + "step": 6990, + "unclipped_grad_norm": 0.8664272427558899 + }, + { + "bits_per_byte": 1.5201646259701156, + "learning_rate": 4.137060090156724e-05, + "loss_nats": 1.0536978244781494, + "step": 7000, + "unclipped_grad_norm": 0.823028564453125 + }, + { + "bits_per_byte": 1.512097435766094, + "learning_rate": 4.114749323218249e-05, + "loss_nats": 1.048106074333191, + "step": 7010, + "unclipped_grad_norm": 0.8564846515655518 + }, + { + "bits_per_byte": 1.4570136284691975, + "learning_rate": 4.0926501860417706e-05, + "loss_nats": 1.0099248886108398, + "step": 7020, + "unclipped_grad_norm": 0.8378328084945679 + }, + { + "bits_per_byte": 1.5138229377022003, + "learning_rate": 4.070763056240571e-05, + "loss_nats": 1.049302101135254, + "step": 7030, + "unclipped_grad_norm": 0.8506705164909363 + }, + { + "bits_per_byte": 1.4941986853428488, + "learning_rate": 4.049088307805306e-05, + "loss_nats": 1.0356996059417725, + "step": 7040, + "unclipped_grad_norm": 0.8149515986442566 + }, + { + "bits_per_byte": 1.485162544883866, + "learning_rate": 4.027626311097629e-05, + "loss_nats": 1.0294362306594849, + "step": 7050, + "unclipped_grad_norm": 0.7959237694740295 + }, + { + "bits_per_byte": 1.4833622304946341, + "learning_rate": 4.006377432843846e-05, + "loss_nats": 1.0281883478164673, + "step": 7060, + "unclipped_grad_norm": 0.846172571182251 + }, + { + "bits_per_byte": 1.452753274242024, + "learning_rate": 3.985342036128656e-05, + "loss_nats": 1.006971836090088, + "step": 7070, + "unclipped_grad_norm": 0.8459955453872681 + }, + { + "bits_per_byte": 1.4353539614368844, + "learning_rate": 3.964520480388956e-05, + "loss_nats": 0.9949115514755249, + "step": 7080, + "unclipped_grad_norm": 0.8262940049171448 + }, + { + "bits_per_byte": 1.458162472576961, + "learning_rate": 3.943913121407678e-05, + "loss_nats": 1.010721206665039, + "step": 7090, + "unclipped_grad_norm": 0.8544578552246094 + }, + { + "bits_per_byte": 1.4575949298291138, + "learning_rate": 3.923520311307735e-05, + "loss_nats": 1.0103278160095215, + "step": 7100, + "unclipped_grad_norm": 0.7949163913726807 + }, + { + "bits_per_byte": 1.4806316619468853, + "learning_rate": 3.9033423985459816e-05, + "loss_nats": 1.0262956619262695, + "step": 7110, + "unclipped_grad_norm": 0.8059237599372864 + }, + { + "bits_per_byte": 1.5197814486239933, + "learning_rate": 3.88337972790728e-05, + "loss_nats": 1.0534322261810303, + "step": 7120, + "unclipped_grad_norm": 0.8576845526695251 + }, + { + "bits_per_byte": 1.4784170413517248, + "learning_rate": 3.8636326404985845e-05, + "loss_nats": 1.0247606039047241, + "step": 7130, + "unclipped_grad_norm": 0.8865160942077637 + }, + { + "bits_per_byte": 1.4469725933812232, + "learning_rate": 3.844101473743143e-05, + "loss_nats": 1.002964973449707, + "step": 7140, + "unclipped_grad_norm": 0.8292717933654785 + }, + { + "bits_per_byte": 1.431326987666932, + "learning_rate": 3.824786561374701e-05, + "loss_nats": 0.9921202659606934, + "step": 7150, + "unclipped_grad_norm": 0.7803593873977661 + }, + { + "bits_per_byte": 1.508955484717481, + "learning_rate": 3.805688233431824e-05, + "loss_nats": 1.0459282398223877, + "step": 7160, + "unclipped_grad_norm": 0.8625690340995789 + }, + { + "bits_per_byte": 1.4504712364477252, + "learning_rate": 3.786806816252236e-05, + "loss_nats": 1.0053900480270386, + "step": 7170, + "unclipped_grad_norm": 0.8165751695632935 + }, + { + "bits_per_byte": 1.471559577113872, + "learning_rate": 3.768142632467269e-05, + "loss_nats": 1.0200073719024658, + "step": 7180, + "unclipped_grad_norm": 0.8274741768836975 + }, + { + "bits_per_byte": 1.4350309780185642, + "learning_rate": 3.7496960009963236e-05, + "loss_nats": 0.9946876764297485, + "step": 7190, + "unclipped_grad_norm": 0.8526366949081421 + }, + { + "bits_per_byte": 1.4706673311211957, + "learning_rate": 3.7314672370414325e-05, + "loss_nats": 1.0193889141082764, + "step": 7200, + "unclipped_grad_norm": 0.8346464037895203 + }, + { + "bits_per_byte": 1.5350235830393606, + "learning_rate": 3.713456652081882e-05, + "loss_nats": 1.0639972686767578, + "step": 7210, + "unclipped_grad_norm": 0.8620064854621887 + }, + { + "bits_per_byte": 1.504267237654961, + "learning_rate": 3.695664553868873e-05, + "loss_nats": 1.0426785945892334, + "step": 7220, + "unclipped_grad_norm": 0.8516167998313904 + }, + { + "bits_per_byte": 1.4407633317544724, + "learning_rate": 3.678091246420277e-05, + "loss_nats": 0.9986610412597656, + "step": 7230, + "unclipped_grad_norm": 0.8479061126708984 + }, + { + "bits_per_byte": 1.4553653467433283, + "learning_rate": 3.660737030015427e-05, + "loss_nats": 1.0087823867797852, + "step": 7240, + "unclipped_grad_norm": 0.816892147064209 + }, + { + "bits_per_byte": 1.4837560707651098, + "learning_rate": 3.6436022011900056e-05, + "loss_nats": 1.0284613370895386, + "step": 7250, + "unclipped_grad_norm": 0.8647069931030273 + }, + { + "bits_per_byte": 1.3830619485114442, + "learning_rate": 3.626687052730954e-05, + "loss_nats": 0.9586654901504517, + "step": 7260, + "unclipped_grad_norm": 0.8161391615867615 + }, + { + "bits_per_byte": 1.4094007475257815, + "learning_rate": 3.6099918736714955e-05, + "loss_nats": 0.9769221544265747, + "step": 7270, + "unclipped_grad_norm": 0.8319637179374695 + }, + { + "bits_per_byte": 1.4853560253660867, + "learning_rate": 3.5935169492861716e-05, + "loss_nats": 1.0295703411102295, + "step": 7280, + "unclipped_grad_norm": 0.8720204830169678 + }, + { + "bits_per_byte": 1.4114874130286998, + "learning_rate": 3.5772625610859886e-05, + "loss_nats": 0.9783685207366943, + "step": 7290, + "unclipped_grad_norm": 0.80472332239151 + }, + { + "bits_per_byte": 1.4451105372203317, + "learning_rate": 3.5612289868135887e-05, + "loss_nats": 1.0016742944717407, + "step": 7300, + "unclipped_grad_norm": 0.8547036647796631 + }, + { + "bits_per_byte": 1.4482086326929742, + "learning_rate": 3.545416500438522e-05, + "loss_nats": 1.0038217306137085, + "step": 7310, + "unclipped_grad_norm": 0.8266134262084961 + }, + { + "bits_per_byte": 1.3891315602256944, + "learning_rate": 3.52982537215255e-05, + "loss_nats": 0.9628726243972778, + "step": 7320, + "unclipped_grad_norm": 0.8016625046730042 + }, + { + "bits_per_byte": 1.4291524390294224, + "learning_rate": 3.5144558683650396e-05, + "loss_nats": 0.9906129837036133, + "step": 7330, + "unclipped_grad_norm": 0.867045521736145 + }, + { + "bits_per_byte": 1.3836929528574597, + "learning_rate": 3.499308251698404e-05, + "loss_nats": 0.9591028690338135, + "step": 7340, + "unclipped_grad_norm": 0.8164173364639282 + }, + { + "bits_per_byte": 1.467159572974198, + "learning_rate": 3.484382780983616e-05, + "loss_nats": 1.0169575214385986, + "step": 7350, + "unclipped_grad_norm": 0.8525064587593079 + }, + { + "bits_per_byte": 1.433820048173839, + "learning_rate": 3.469679711255795e-05, + "loss_nats": 0.9938483238220215, + "step": 7360, + "unclipped_grad_norm": 0.8385956883430481 + }, + { + "bits_per_byte": 1.4710990075748616, + "learning_rate": 3.4551992937498284e-05, + "loss_nats": 1.0196881294250488, + "step": 7370, + "unclipped_grad_norm": 0.8249295353889465 + }, + { + "bits_per_byte": 1.3941142415864916, + "learning_rate": 3.440941775896104e-05, + "loss_nats": 0.9663263559341431, + "step": 7380, + "unclipped_grad_norm": 0.8101018667221069 + }, + { + "bits_per_byte": 1.4706592479366052, + "learning_rate": 3.4269074013162596e-05, + "loss_nats": 1.0193833112716675, + "step": 7390, + "unclipped_grad_norm": 0.8176234364509583 + }, + { + "bits_per_byte": 1.410369611789417, + "learning_rate": 3.41309640981904e-05, + "loss_nats": 0.977593719959259, + "step": 7400, + "unclipped_grad_norm": 0.8377200961112976 + }, + { + "bits_per_byte": 1.4092478549491645, + "learning_rate": 3.3995090373961795e-05, + "loss_nats": 0.9768161773681641, + "step": 7410, + "unclipped_grad_norm": 0.81974858045578 + }, + { + "bits_per_byte": 1.474973260750848, + "learning_rate": 3.3861455162183876e-05, + "loss_nats": 1.0223735570907593, + "step": 7420, + "unclipped_grad_norm": 0.8284385204315186 + }, + { + "bits_per_byte": 1.4820893868955984, + "learning_rate": 3.3730060746313664e-05, + "loss_nats": 1.027306079864502, + "step": 7430, + "unclipped_grad_norm": 0.845361590385437 + }, + { + "bits_per_byte": 1.502685341232325, + "learning_rate": 3.3600909371519225e-05, + "loss_nats": 1.0415821075439453, + "step": 7440, + "unclipped_grad_norm": 0.8964833617210388 + }, + { + "bits_per_byte": 1.4439314241660162, + "learning_rate": 3.3474003244641194e-05, + "loss_nats": 1.0008569955825806, + "step": 7450, + "unclipped_grad_norm": 0.8369156718254089 + }, + { + "bits_per_byte": 1.4378882977799987, + "learning_rate": 3.334934453415514e-05, + "loss_nats": 0.9966682195663452, + "step": 7460, + "unclipped_grad_norm": 0.8354071378707886 + }, + { + "bits_per_byte": 1.4212995392083758, + "learning_rate": 3.3226935370134486e-05, + "loss_nats": 0.9851697683334351, + "step": 7470, + "unclipped_grad_norm": 0.8366437554359436 + }, + { + "bits_per_byte": 1.4686721603885364, + "learning_rate": 3.31067778442141e-05, + "loss_nats": 1.0180059671401978, + "step": 7480, + "unclipped_grad_norm": 0.8683294653892517 + }, + { + "bits_per_byte": 1.5098209014166228, + "learning_rate": 3.298887400955457e-05, + "loss_nats": 1.0465281009674072, + "step": 7490, + "unclipped_grad_norm": 0.8586563467979431 + }, + { + "bits_per_byte": 1.4743733852646386, + "learning_rate": 3.287322588080714e-05, + "loss_nats": 1.0219577550888062, + "step": 7500, + "unclipped_grad_norm": 0.8332951068878174 + }, + { + "bits_per_byte": 1.4255870666943855, + "learning_rate": 3.275983543407924e-05, + "loss_nats": 0.988141655921936, + "step": 7510, + "unclipped_grad_norm": 0.8199127912521362 + }, + { + "bits_per_byte": 1.4619254529778392, + "learning_rate": 3.2648704606900735e-05, + "loss_nats": 1.0133295059204102, + "step": 7520, + "unclipped_grad_norm": 0.8662075400352478 + }, + { + "bits_per_byte": 1.4951213722647276, + "learning_rate": 3.253983529819086e-05, + "loss_nats": 1.0363391637802124, + "step": 7530, + "unclipped_grad_norm": 0.868276834487915 + }, + { + "bits_per_byte": 1.4492965949423324, + "learning_rate": 3.243322936822571e-05, + "loss_nats": 1.0045758485794067, + "step": 7540, + "unclipped_grad_norm": 0.8661652207374573 + }, + { + "bits_per_byte": 1.4698851540250717, + "learning_rate": 3.232888863860648e-05, + "loss_nats": 1.0188467502593994, + "step": 7550, + "unclipped_grad_norm": 0.8560738563537598 + }, + { + "bits_per_byte": 1.4346130601769675, + "learning_rate": 3.222681489222838e-05, + "loss_nats": 0.9943979978561401, + "step": 7560, + "unclipped_grad_norm": 0.8252226710319519 + }, + { + "bits_per_byte": 1.3999351663875943, + "learning_rate": 3.212700987325009e-05, + "loss_nats": 0.9703611135482788, + "step": 7570, + "unclipped_grad_norm": 0.8430092930793762 + }, + { + "bits_per_byte": 1.563467105734957, + "learning_rate": 3.202947528706403e-05, + "loss_nats": 1.0837128162384033, + "step": 7580, + "unclipped_grad_norm": 0.8905554413795471 + }, + { + "bits_per_byte": 1.4270704170580772, + "learning_rate": 3.1934212800267116e-05, + "loss_nats": 0.9891698360443115, + "step": 7590, + "unclipped_grad_norm": 0.8126006722450256 + }, + { + "bits_per_byte": 1.4057284019819078, + "learning_rate": 3.1841224040632484e-05, + "loss_nats": 0.9743766784667969, + "step": 7600, + "unclipped_grad_norm": 0.882316529750824 + }, + { + "bits_per_byte": 1.4318451713939815, + "learning_rate": 3.175051059708143e-05, + "loss_nats": 0.9924794435501099, + "step": 7610, + "unclipped_grad_norm": 0.8412167429924011 + }, + { + "bits_per_byte": 1.3879486635530598, + "learning_rate": 3.1662074019656414e-05, + "loss_nats": 0.9620527029037476, + "step": 7620, + "unclipped_grad_norm": 0.8003552556037903 + }, + { + "bits_per_byte": 1.3963014309488393, + "learning_rate": 3.157591581949455e-05, + "loss_nats": 0.9678424000740051, + "step": 7630, + "unclipped_grad_norm": 0.852692186832428 + }, + { + "bits_per_byte": 1.4094292106544992, + "learning_rate": 3.149203746880175e-05, + "loss_nats": 0.9769418835639954, + "step": 7640, + "unclipped_grad_norm": 0.8325322270393372 + }, + { + "bits_per_byte": 1.5484170759406035, + "learning_rate": 3.141044040082763e-05, + "loss_nats": 1.073280930519104, + "step": 7650, + "unclipped_grad_norm": 0.8584242463111877 + }, + { + "bits_per_byte": 1.4568894569952746, + "learning_rate": 3.133112600984091e-05, + "loss_nats": 1.0098388195037842, + "step": 7660, + "unclipped_grad_norm": 0.8722478151321411 + }, + { + "bits_per_byte": 1.4809008147954856, + "learning_rate": 3.125409565110571e-05, + "loss_nats": 1.0264822244644165, + "step": 7670, + "unclipped_grad_norm": 0.8899861574172974 + }, + { + "bits_per_byte": 1.4283591690523179, + "learning_rate": 3.117935064085835e-05, + "loss_nats": 0.9900631308555603, + "step": 7680, + "unclipped_grad_norm": 0.8703398108482361 + }, + { + "bits_per_byte": 1.448618811315282, + "learning_rate": 3.110689225628479e-05, + "loss_nats": 1.004106044769287, + "step": 7690, + "unclipped_grad_norm": 0.8683562874794006 + }, + { + "bits_per_byte": 1.3795722485427868, + "learning_rate": 3.1036721735498926e-05, + "loss_nats": 0.9562466144561768, + "step": 7700, + "unclipped_grad_norm": 0.8153799176216125 + }, + { + "bits_per_byte": 1.4603757173109144, + "learning_rate": 3.096884027752132e-05, + "loss_nats": 1.012255311012268, + "step": 7710, + "unclipped_grad_norm": 0.8314683437347412 + }, + { + "bits_per_byte": 1.475302951492552, + "learning_rate": 3.090324904225885e-05, + "loss_nats": 1.0226020812988281, + "step": 7720, + "unclipped_grad_norm": 0.8867014646530151 + }, + { + "bits_per_byte": 1.4762574552048406, + "learning_rate": 3.083994915048472e-05, + "loss_nats": 1.023263692855835, + "step": 7730, + "unclipped_grad_norm": 0.8719123005867004 + }, + { + "bits_per_byte": 1.4191212929438728, + "learning_rate": 3.077894168381942e-05, + "loss_nats": 0.9836599230766296, + "step": 7740, + "unclipped_grad_norm": 0.8960307240486145 + }, + { + "bits_per_byte": 1.432920836883803, + "learning_rate": 3.0720227684712254e-05, + "loss_nats": 0.9932250380516052, + "step": 7750, + "unclipped_grad_norm": 0.8596091270446777 + }, + { + "bits_per_byte": 1.4627151973106012, + "learning_rate": 3.0663808156423456e-05, + "loss_nats": 1.0138769149780273, + "step": 7760, + "unclipped_grad_norm": 0.8388862609863281 + }, + { + "bits_per_byte": 1.3769384202361483, + "learning_rate": 3.060968406300711e-05, + "loss_nats": 0.9544209837913513, + "step": 7770, + "unclipped_grad_norm": 0.8569913506507874 + }, + { + "bits_per_byte": 1.4625328957006867, + "learning_rate": 3.055785632929463e-05, + "loss_nats": 1.0137505531311035, + "step": 7780, + "unclipped_grad_norm": 0.862941563129425 + }, + { + "bits_per_byte": 1.410140186933166, + "learning_rate": 3.0508325840878984e-05, + "loss_nats": 0.9774346947669983, + "step": 7790, + "unclipped_grad_norm": 0.8385294079780579 + }, + { + "bits_per_byte": 1.4256966196429852, + "learning_rate": 3.046109344409957e-05, + "loss_nats": 0.9882175922393799, + "step": 7800, + "unclipped_grad_norm": 0.8127948641777039 + }, + { + "bits_per_byte": 1.4127408225881881, + "learning_rate": 3.0416159946027727e-05, + "loss_nats": 0.9792373180389404, + "step": 7810, + "unclipped_grad_norm": 0.8709434866905212 + }, + { + "bits_per_byte": 1.3889279327670727, + "learning_rate": 3.0373526114452946e-05, + "loss_nats": 0.9627314805984497, + "step": 7820, + "unclipped_grad_norm": 0.861073911190033 + }, + { + "bits_per_byte": 1.430519529121131, + "learning_rate": 3.0333192677869803e-05, + "loss_nats": 0.9915605783462524, + "step": 7830, + "unclipped_grad_norm": 0.8894413709640503 + }, + { + "bits_per_byte": 1.4341919606563298, + "learning_rate": 3.029516032546544e-05, + "loss_nats": 0.994106113910675, + "step": 7840, + "unclipped_grad_norm": 0.8452141880989075 + }, + { + "bits_per_byte": 1.3809828502451635, + "learning_rate": 3.02594297071078e-05, + "loss_nats": 0.9572243690490723, + "step": 7850, + "unclipped_grad_norm": 0.8403438925743103 + }, + { + "bits_per_byte": 1.4340615118156506, + "learning_rate": 3.0226001433334603e-05, + "loss_nats": 0.9940156936645508, + "step": 7860, + "unclipped_grad_norm": 0.8349435329437256 + }, + { + "bits_per_byte": 1.4495812262295102, + "learning_rate": 3.019487607534278e-05, + "loss_nats": 1.0047731399536133, + "step": 7870, + "unclipped_grad_norm": 0.8659698367118835 + }, + { + "bits_per_byte": 1.3787726152076, + "learning_rate": 3.016605416497886e-05, + "loss_nats": 0.9556923508644104, + "step": 7880, + "unclipped_grad_norm": 0.8691674470901489 + }, + { + "bits_per_byte": 1.4674514275327122, + "learning_rate": 3.0139536194729734e-05, + "loss_nats": 1.0171598196029663, + "step": 7890, + "unclipped_grad_norm": 0.8606283664703369 + }, + { + "bits_per_byte": 1.4335653418679113, + "learning_rate": 3.0115322617714362e-05, + "loss_nats": 0.9936717748641968, + "step": 7900, + "unclipped_grad_norm": 0.864492654800415 + }, + { + "bits_per_byte": 1.4437606453937095, + "learning_rate": 3.0093413847675965e-05, + "loss_nats": 1.0007386207580566, + "step": 7910, + "unclipped_grad_norm": 0.8537375330924988 + }, + { + "bits_per_byte": 1.459385957155199, + "learning_rate": 3.0073810258974985e-05, + "loss_nats": 1.0115692615509033, + "step": 7920, + "unclipped_grad_norm": 0.8682068586349487 + }, + { + "bits_per_byte": 1.4180243016053444, + "learning_rate": 3.0056512186582658e-05, + "loss_nats": 0.98289954662323, + "step": 7930, + "unclipped_grad_norm": 0.841934323310852 + }, + { + "bits_per_byte": 1.4232665047862938, + "learning_rate": 3.0041519926075316e-05, + "loss_nats": 0.9865331649780273, + "step": 7940, + "unclipped_grad_norm": 0.886671245098114 + }, + { + "bits_per_byte": 1.480160601466172, + "learning_rate": 3.0028833733629313e-05, + "loss_nats": 1.02596914768219, + "step": 7950, + "unclipped_grad_norm": 0.906080961227417 + }, + { + "bits_per_byte": 1.4180263653971548, + "learning_rate": 3.0018453826016686e-05, + "loss_nats": 0.9829009771347046, + "step": 7960, + "unclipped_grad_norm": 0.846083402633667 + }, + { + "bits_per_byte": 1.4403725011803867, + "learning_rate": 3.001038038060143e-05, + "loss_nats": 0.9983901381492615, + "step": 7970, + "unclipped_grad_norm": 0.8702674508094788 + }, + { + "bits_per_byte": 1.4775808617032298, + "learning_rate": 3.0004613535336433e-05, + "loss_nats": 1.0241810083389282, + "step": 7980, + "unclipped_grad_norm": 0.9070017337799072 + }, + { + "bits_per_byte": 1.5088130830825668, + "learning_rate": 3.000115338876117e-05, + "loss_nats": 1.0458295345306396, + "step": 7990, + "unclipped_grad_norm": 0.943093478679657 + }, + { + "bits_per_byte": 1.4082921473583199, + "learning_rate": 3e-05, + "loss_nats": 0.9761537313461304, + "step": 8000, + "unclipped_grad_norm": 0.8546885848045349 + } + ], + "variant": "uniform_groups_6_7_forward" +} diff --git a/experiments/k3/attnres_forward/run_matrix.py b/experiments/k3/attnres_forward/run_matrix.py index 0106cbc..36b7691 100644 --- a/experiments/k3/attnres_forward/run_matrix.py +++ b/experiments/k3/attnres_forward/run_matrix.py @@ -84,6 +84,34 @@ def validate_manifest(args: argparse.Namespace) -> None: raise ValueError("the frozen protocol permits one or two processes") +def stop_processes(items: list[dict[str, Any]]) -> None: + for item in items: + if item["process"].poll() is None: + item["process"].terminate() + for item in items: + process = item["process"] + if process.poll() is not None: + continue + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + process.wait() + + +def quarantine_failed_output( + output_dir: Path, variant: str, seed: int, run_kind: str +) -> str | None: + output = cell_output(output_dir, variant, seed, run_kind) + if not output.exists(): + return None + failed = output.with_suffix(".failed.json") + if failed.exists(): + failed = output.with_suffix(f".failed-{time.time_ns()}.json") + output.replace(failed) + return str(failed) + + def run_cells( args: argparse.Namespace, cells: list[tuple[str, int, str]], @@ -138,13 +166,15 @@ def run_cells( variant, seed, run_kind = item["identity"] elapsed = time.monotonic() - item["started"] if return_code != 0: - for survivor in survivors: - survivor["process"].terminate() - for survivor in running: - if survivor is not item and survivor not in survivors: - survivor["process"].terminate() + stop_processes( + [candidate for candidate in running if candidate is not item] + ) + quarantined = quarantine_failed_output( + args.output_dir, variant, seed, run_kind + ) raise RuntimeError( - f"cell failed: {variant}/{seed}/{run_kind}: {return_code}" + f"cell failed: {variant}/{seed}/{run_kind}: {return_code}; " + f"quarantined_output={quarantined}" ) completed += 1 print( diff --git a/experiments/k3/attnres_forward/train.py b/experiments/k3/attnres_forward/train.py index b43a545..0757e47 100644 --- a/experiments/k3/attnres_forward/train.py +++ b/experiments/k3/attnres_forward/train.py @@ -170,9 +170,6 @@ class ForwardInterventionLanguageModel(parent.GradientLanguageModel): def forward( self, input_ids: torch.Tensor, capture: bool = False ) -> tuple[torch.Tensor, parent.ActivationTrace | None]: - if not self.selected_indices: - return super().forward(input_ids, capture) - self.forward_calls += 1 embedded = self.embed(input_ids) trace = parent.ActivationTrace([], [], [], [], []) if capture else None @@ -276,11 +273,13 @@ def build_intervention_audit( str(index): ( source_counts[str(index)] == [study_manifest["selected_source_counts"][str(index)]] + == [EXPECTED_SOURCE_COUNTS[index]] ) for index in selected } visit_gate = ( - all(value == model.forward_calls for value in model.depth_visits) + model.forward_calls > 0 + and all(value == model.forward_calls for value in model.depth_visits) and model.output_visits == model.forward_calls ) selected_parameter_gate = all( @@ -415,6 +414,14 @@ def main() -> None: ) if expected != VARIANTS[variant]: raise ValueError("study manifest selector drift") + for index, source_count in EXPECTED_SOURCE_COUNTS.items(): + if ( + study_manifest["selected_source_counts"].get(str(index)) + != source_count + ): + raise ValueError( + f"study manifest source-count drift at depth index {index}" + ) output_value = argument_value("--output") if output_value is None: diff --git a/experiments/k3/attnres_forward/verify.py b/experiments/k3/attnres_forward/verify.py index c9efb02..dc0a2e3 100644 --- a/experiments/k3/attnres_forward/verify.py +++ b/experiments/k3/attnres_forward/verify.py @@ -92,10 +92,14 @@ def smoke_compare(args: argparse.Namespace) -> None: "gradient_gate", "environment", ) - checks = { - field: parent_result[field] == wrapper_result[field] - for field in fields - } + checks = {} + for field in fields: + parent_value = exact_structure(parent_result[field]) + wrapper_value = exact_structure(wrapper_result[field]) + if field == "manifest": + parent_value.pop("path", None) + wrapper_value.pop("path", None) + checks[field] = parent_value == wrapper_value wrapper_identity = { "protocol": wrapper_result["protocol_id"] == runner.PROTOCOL_ID, "parent_protocol": ( @@ -185,6 +189,7 @@ def step_zero(args: argparse.Namespace) -> None: "initial_mixer_hash": initial_mixer, "logits_sha256": tensor_sha256(logits), "loss_nats": loss.detach().cpu().item(), + "loss_tensor_sha256": tensor_sha256(loss), "evaluation": exact_structure(evaluation), "diagnostic": exact_structure(diagnostic), } diff --git a/package.json b/package.json index e69ea6a..fc09f59 100644 --- a/package.json +++ b/package.json @@ -23,6 +23,7 @@ "check:data:k3-attnres-gradient": "node scripts/check-k3-attnres-gradient-data.mjs", "check:data:k3-attnres-spike": "node scripts/check-k3-attnres-spike-data.mjs", "check:data:k3-attnres-local-path": "node scripts/check-k3-attnres-local-path-data.mjs", + "check:data:k3-attnres-forward": "node scripts/check-k3-attnres-forward-data.mjs", "check:site": "node scripts/check-site.mjs", "check:moe-browser": "node scripts/check-moe-browser.mjs", "check:reasoning-browser": "node scripts/check-reasoning-browser.mjs", @@ -46,6 +47,7 @@ "check:k3-attnres-gradient-browser": "node scripts/check-k3-attnres-gradient-browser.mjs", "check:k3-attnres-spike-browser": "node scripts/check-k3-attnres-spike-browser.mjs", "check:k3-attnres-local-path-browser": "node scripts/check-k3-attnres-local-path-browser.mjs", + "check:k3-attnres-forward-browser": "node scripts/check-k3-attnres-forward-browser.mjs", "check:k3-browser": "node scripts/check-k3-browser.mjs" }, "dependencies": { diff --git a/research/K3_ATTNRES_FORWARD_TRAINING_AUDIT.md b/research/K3_ATTNRES_FORWARD_TRAINING_AUDIT.md new file mode 100644 index 0000000..217dd42 --- /dev/null +++ b/research/K3_ATTNRES_FORWARD_TRAINING_AUDIT.md @@ -0,0 +1,112 @@ +# Round 08 AttnRes 训练期前向干预：结果审计 + +审计日期：2026-07-30 +协议：`llm-atlas-k3-attnres-forward-training-v1` +结果后 Grok 会话：`019fb28b-a9e1-7643-8e43-06f5e16a2077` + +## 1. 一句话结论 + +冻结 analyzer 的唯一主状态为： + +```text +forward_training_attenuation_established_within_reduced_protocol +``` + +联合 `groups 6+7` 的训练期 uniform-forward 消融在三个预注册 seed 上同时通过 +`spike contrast` 与 `peak / mean` 的 20% attenuation 门；逐 seed 与三 seed 平均 +validation BPC 也全部通过质量护栏。结果后独立只读复算得到： + +```text +blocking_errors = 0 +status_confirmed = true +replay_confirmed = true +``` + +这只是在固定 depth-32 缩小 Block AttnRes、固定数据与 8,000-step 预算中的训练期 +架构消融；**不是真实 Kimi-K3 / 2.8T checkpoint 结果，也不是 Figure 5(c) 未公开 +telemetry 的复现。** + +## 2. 主门复算 + +冻结定义： + +```text +S = layers 21–25 +R = other 27 layers +C = mean(g[S]) / mean(g[R]) +P = max(g) / mean(g) +D = (X_reference - X_variant) / X_reference +``` + +主变体为 `uniform_groups_6_7_forward`，只读复算如下： + +| seed | C reference | C variant | C drop | P reference | P variant | P drop | ΔBPC | +|---|---:|---:|---:|---:|---:|---:|---:| +| 2026073001 | 3.046269 | 0.806011 | 73.54% | 3.219067 | 1.435618 | 55.40% | +0.005696 | +| 2026073002 | 3.332848 | 0.758201 | 77.25% | 3.618924 | 1.375016 | 62.00% | +0.009598 | +| 2026073003 | 1.881492 | 0.712882 | 62.11% | 2.288267 | 1.548141 | 32.34% | +0.004441 | + +因此 attenuation 为 `6/6`；最小 contrast drop 为 `62.11%`，最小 peak drop 为 +`32.34%`，都高于冻结的 `20%` 门槛。BPC 三格都低于 `+0.05`，平均 +`+0.006578` 低于 `+0.03`，质量门为 `4/4`。 + +这里的 contrast 下降不等价于“尖峰层被关闭”：它可能由 spike-window 分子下降、 +27 层 reference 分母上升，或两者共同造成；被联合消融覆盖的 layers 26–28 仍属于 +这个分母。 + +## 3. 完整性与复现 + +- 13 个新 raw 文件完整：12 formal + 1 primary replay； +- 每格 `65,536,000` target bytes，新处理总量 `851,968,000`； +- 三个历史 paired reference 合计 `196,608,000` bytes，未在 Round 08 重跑； +- 13/13 raw canonical self-hash、aggregate 与 reproduction self-hash 自洽； +- architecture / depth / steps / batch 固定为 Block / 32 / 8,000 / 32； +- initial public/mixer state、schedule、validation、diagnostic、input-gate、model、 + optimizer 与 scientific environment 对同 seed historical reference 配对 exact； +- primary seed 1 的 formal / replay scientific payload exact： + `b85563ca5cb53e60b39c3801d372376206105b8a089a8633b3e81973a7f0c051`。 + +四个正式 selector 的 visit、source count、uniform arithmetic 与 reachability 均通过。 +被选 mixer 的 query / key norm 留在 optimizer param groups，但 forward 不再调用它们： +gradient hook 为 0、Adam state 不存在、最终 tensor 与初始值 byte-exact。未选 mixer 的 +optimizer-state 检查是比科学协议更强的实现审计，不参与主 status。 + +## 4. 描述性 non-additivity + +冻结的 bookkeeping residual 为： + +```text +I67 = ln(Xref / X67) - ln(Xref / X6) - ln(Xref / X7) +``` + +step 8,000 的三 seed 平均为： + +- spike contrast：`-0.367038` +- peak / mean：`-0.170448` + +它来自三套独立训练，只能描述 joint run 与两个 single runs 的 log-effect 残差；不能 +写成因果 interaction、Shapley contribution 或“group 6/7 互相抑制”的机制结论。 + +## 5. 两阶段独立审阅 + +结果前 Grok 实现审阅指出 smoke-only empty selector 的 visit census 可空真。正式 +4×3 路径全部是非空 selector，因此不影响 raw formal 数值；矩阵结束后已删除 early +return、加入 `forward_calls > 0`，并重跑 step-zero、parent smoke、wrapper smoke +与 equivalence。修补后的 learned wrapper 实际执行 39 次 forward，父/包装器 15 组 +科学字段仍全部 exact。 + +结果后 Grok 在只读 sandbox 中从 13 个 raw 与三个 historical references 独立复算 +identity、自哈希、selector、pairing、两项主指标、BPC、replay 与 `I67`。它报告 +`0 mismatch`、`blocking_errors=0`，确认 analyzer status 与 claim boundary。 + +## 6. 最终 claim boundary + +可以说：在这一固定缩小协议内，联合 group 6+7 的 train-time uniform-forward +architecture ablation 相对历史同 seed reference 达到预注册 attenuation + BPC 门控。 + +不能说： + +- 已定位真实 Kimi-K3 的训练尖峰； +- 已复现 K3 报告 Figure 5(c)； +- 已把 forward、natural backward 与 optimizer update 分离成纯因果效应； +- 已证明下游能力等价、总体统计显著性、可加性或因果 interaction。 diff --git a/research/K3_ATTNRES_FORWARD_TRAINING_IMPLEMENTATION_REVIEW.md b/research/K3_ATTNRES_FORWARD_TRAINING_IMPLEMENTATION_REVIEW.md new file mode 100644 index 0000000..2d363dd --- /dev/null +++ b/research/K3_ATTNRES_FORWARD_TRAINING_IMPLEMENTATION_REVIEW.md @@ -0,0 +1,89 @@ +# Round 08 runner / analyzer 实现审阅与处置 + +审阅日期：2026-07-30 +Grok 会话：`019fb1f0-bb24-7202-8123-295edda7518f` +身份：**正式文件完成前的外部模型只读实现审计，不是结果或论文证据** + +## 1. 审阅边界 + +Grok Headless 只读检查： + +- 冻结协议与 manifest； +- `train.py` / `verify.py` / `analyze.py` / `run_matrix.py`； +- 父 runner 的 salt、schema 与 DepthMixer 算术路径。 + +明确禁止读取 Round 08 raw formal results、运行训练、修改文件、web search 与 subagents。 + +## 2. 对正式 4×3 路径的确认 + +审阅确认： + +- 新 protocol ID 没有进入父 `window_start` salt； +- selected 路径用 constant-zero FP32 logits 和父 `softmax+einsum` kernel； +- 非空 selector 的 query / key norm hook、AdamW state 与 final=initial gate 自洽； +- historical pairing 比较的字段在父 JSON 中真实存在； +- replay payload 正确排除 run kind、timing、path、parent self hash 与 GPU/version metadata， + 同时保留模型、optimizer、diagnostics、history、selector 与 reachability； +- analyzer 强制 4×3 identity、step 8,000 主判定、1-based layers 21–25、finite/positive、 + `D=(ref-variant)/ref`、BPC `variant-ref` 与 descriptive `I67`； +- 12 formal + 1 replay、65,536,000 bytes/cell 与最大两进程算术正确。 + +没有发现会改变正在运行的四个非空 formal variants 数值语义的 blocking error。 + +## 3. Blocking finding：smoke-only empty selector 的空审计 + +`learned_reference` 直接调用父 `forward`，没有递增 wrapper 的 visit counters。因此： + +```text +forward_calls = 0 +all depth/output visits = 0 +visit_gate = all(0 == 0) = true +selected reachability = all([]) = true +``` + +这不会影响四个正式变体，它们全部是非空 selector；而 empty selector 另有 20-step +parent-equivalence gate，模型/optimizer/evaluations/diagnostics/history/hash 已 exact。 +但 `forward_intervention.passed` 本身不能在修复前被当作 learned census 证据。 + +处置： + +- formal matrix 完成后，删除 `selected_indices` 为空时的父路径 early return； +- 让 empty selector 也走同一 copied forward，其中 `mix` 对 64 个节点逐个调用父 + `DepthMixer.forward`； +- 加入 `forward_calls > 0`； +- 重跑 step-0、parent smoke、wrapper smoke 与 parent equivalence； +- 只有 copied path 仍逐字段 exact 才保留。 + +这项修复只强化 smoke audit，不更改任何正式 variant 的 selector 或 forward。 + +## 4. Non-blocking findings 与处置 + +| finding | 处置 | +|---|---| +| `smoke_compare` 整体比较 `manifest.path` | 规范化 path 后比较 scientific manifest fields | +| unselected gate 额外要求 optimizer state | 保留为强实现 gate，但在文档中标成 protocol 之外的额外审计，不用于科学 status | +| GPU/version 差异只有 `metadata_equal`，没有 warnings 数组 | aggregate 增加显式 metadata warnings | +| `run_matrix` 失败后只 terminate、不 wait/kill，invalid file 会阻塞重跑 | 加入 terminate→wait→kill 清理，并在 cell failure 时标明 exact invalid target | +| step-0 CE 只比较 Python float | 追加 scalar tensor SHA-256 | +| `EXPECTED_SOURCE_COUNTS` 未使用 | 用于 runner↔manifest 交叉校验 | + +上述修订不读取结果、不改变冻结阈值或主公式。 + +## 5. 矩阵结束后的处置结果 + +13 个单元全部退出后才应用上述修订；四个正式非空 selector 的 raw 文件未被重写。 +修补后的前置闸门结果： + +- step-zero 五个 variant 的 logits、loss tensor、evaluation 与 diagnostic exact； +- smoke-only learned wrapper 实际执行 `39` 次 forward，64 个 depth mixer 与 + output mixer 的 visit census 全部非零且 exact； +- parent 与 wrapper 的 architecture、seed、schedule、model、optimizer、hash、 + evaluation、diagnostic、history、gradient gate 与 scientific environment 共 + 15 组字段全部 exact； +- primary smoke 的 selected 参数仍为 0 hook、无 optimizer state、final=initial； +- unselected optimizer-state 条件继续作为额外实现闸门，不进入科学 status。 + +结果后 Grok 会话 `019fb28b-a9e1-7643-8e43-06f5e16a2077` 在只读 sandbox 中独立 +复算 13 个 raw、三个 historical references 与 aggregate，报告 +`blocking_errors=0`、`status_confirmed=true`、`replay_confirmed=true`。详细数字与 +claim boundary 见 `research/K3_ATTNRES_FORWARD_TRAINING_AUDIT.md`。 diff --git a/scripts/check-agents-browser.mjs b/scripts/check-agents-browser.mjs index 1794758..43d92ee 100644 --- a/scripts/check-agents-browser.mjs +++ b/scripts/check-agents-browser.mjs @@ -228,7 +228,7 @@ if (numeric(reliability.initial.passAt) <= numeric(reliability.k2.passAt) || num if (reliability.nonIdempotent.sideRisk === "LOW") failures.push("非幂等写操作风险没有提升"); if (numeric(rl.wait.utilization) >= numeric(rl.full.utilization) || numeric(rl.wait.lostWork) <= numeric(rl.full.lostWork)) failures.push("wait-all 长尾/重算方向异常"); if (!rl.wait.takeaway.includes("wait-all") || rl.keyboardSelected !== "rl" || rl.keyboardVisible !== "rl") failures.push("长程 RL 解释或键盘导航异常"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); if (home.paperCount !== "486" || papers.total !== 486 || !papers.hasAgentFilter || papers.agentVisible < 52) failures.push("论文库 Agent 标签或论文总数异常"); if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常"); if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`); diff --git a/scripts/check-alignment-browser.mjs b/scripts/check-alignment-browser.mjs index 15f95b1..07269a8 100644 --- a/scripts/check-alignment-browser.mjs +++ b/scripts/check-alignment-browser.mjs @@ -226,7 +226,7 @@ if (!update.steps[0].includes("Fixed preference")) failures.push("DPO 更新流 if (!recipe.family.includes("Multi-effort") || !recipe.regime.includes("9 RL experts") || !recipe.constraints.includes("verbosity")) failures.push("K3 配方合同异常"); if (!recipe.path.some((step) => step.includes("3 domains × 3 efforts")) || !recipe.path.some((step) => step.includes("MOPD"))) failures.push("K3 配方路径异常"); if (recipe.keyboardSelected !== "recipe" || recipe.keyboardVisible !== "recipe") failures.push("实验 tab 键盘导航异常"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); if (home.paperCount !== "486" || papers.total !== 486 || !papers.hasAlignmentFilter || papers.alignmentVisible < 35) failures.push("论文库后训练标签或论文总数异常"); if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常"); if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`); diff --git a/scripts/check-data-browser.mjs b/scripts/check-data-browser.mjs index c09739c..76bb783 100644 --- a/scripts/check-data-browser.mjs +++ b/scripts/check-data-browser.mjs @@ -234,7 +234,7 @@ if (layout.navLinks !== 20 || mobile.mobileLinks !== 20 || home.navLinks !== 20) if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出"); if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`); if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强") || home.firstHref !== "/k3/") { +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后") || home.firstHref !== "/k3/") { failures.push("首页 Transformer 新章入口异常"); } if (home.paperCount !== "486") failures.push(`首页论文总数异常：${home.paperCount}`); diff --git a/scripts/check-deepseek-browser.mjs b/scripts/check-deepseek-browser.mjs index 4a7121a..94066dd 100644 --- a/scripts/check-deepseek-browser.mjs +++ b/scripts/check-deepseek-browser.mjs @@ -1318,7 +1318,7 @@ if (completionDepth.tasks.panel !== "tasks" || completionDepth.tasks.mathCards ! if (completionDepth.hidden.panel !== "hidden" || completionDepth.hidden.stages !== 29 || completionDepth.hidden.selected !== "layer_07" || completionDepth.hidden.points !== 29 || completionDepth.hidden.exact === "1,537 / 1,537" || numeric(completionDepth.hidden.relative) <= 0) failures.push("29 阶段隐藏状态曲线或交互异常"); if (completionDepth.router.panel !== "router" || completionDepth.router.layers !== 26 || completionDepth.router.selected !== "layer 24" || completionDepth.router.points !== 26 || !completionDepth.router.ordered.includes("%") || !completionDepth.router.setExact.includes("%") || numeric(completionDepth.router.tv) <= 0 || completionDepth.router.reproCards !== 4 || completionDepth.router.links !== 4) failures.push("26 层 MoE 路由曲线或复跑证据异常"); if (completionDepth.keyboardSelected !== "tasks" || completionDepth.keyboardVisible !== "tasks") failures.push("完成度与全深度实验键盘 tab 导航异常"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强") || home.firstHref !== "/k3/" || home.paperCount !== "486") failures.push("首页 K3 首发入口或论文数异常"); +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后") || home.firstHref !== "/k3/" || home.paperCount !== "486") failures.push("首页 K3 首发入口或论文数异常"); if (papers.total !== 486 || !papers.hasFilter || papers.visible < 20 || !papers.hasCoder || !papers.hasEngram) failures.push("论文库 DeepSeek 聚光异常"); if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 13 || mobile.behaviorTabs !== 4 || mobile.behaviorSources !== 16 || mobile.behaviorEdges !== 10 || mobile.behaviorDeviceCells !== 29 || mobile.completionDepthTabs !== 4 || mobile.completionDepthHiddenStages !== 29 || mobile.completionDepthRouterLayers !== 26 || mobile.artifactHeatCells !== 64 || mobile.corpusCohorts !== 3 || mobile.lengthDeltaCards !== 4 || mobile.templateLayers !== 6 || mobile.templateScopes !== 2 || mobile.templateModes !== 2 || mobile.templateDomainCards !== 4 || mobile.templateDepthCells !== 24 || mobile.historyLayers !== 6 || mobile.historyScopes !== 2 || mobile.historyModes !== 2 || mobile.historyEffects !== 3 || mobile.historyDomainCards !== 4 || mobile.historyDepthCells !== 24 || mobile.distanceLayers !== 6 || mobile.distanceScopes !== 2 || mobile.distanceModes !== 2 || mobile.distanceContrasts !== 2 || mobile.distanceDomainCards !== 4 || mobile.distanceDepthCells !== 24 || mobile.boundaryLayers !== 6 || mobile.boundaryScopes !== 2 || mobile.boundaryModes !== 2 || mobile.boundaryContrasts !== 3 || mobile.boundaryTokenCards !== 4 || mobile.boundaryDomainCards !== 4 || mobile.boundaryDepthCells !== 24 || mobile.roleLayers !== 6 || mobile.roleScopes !== 2 || mobile.roleModes !== 2 || mobile.roleContrasts !== 3 || mobile.roleLevelCards !== 4 || mobile.roleDomainCards !== 4 || mobile.roleDepthCells !== 24 || mobile.specialLayers !== 6 || mobile.specialScopes !== 2 || mobile.specialModes !== 2 || mobile.specialContrasts !== 4 || mobile.specialTokenCards !== 4 || mobile.specialDomainCards !== 4 || mobile.specialDepthCells !== 24 || mobile.roleBlockLayers !== 6 || mobile.roleBlockScopes !== 2 || mobile.roleBlockModes !== 2 || mobile.roleBlockEffects !== 3 || mobile.roleBlockMatrixCards !== 4 || mobile.roleBlockDomainCards !== 4 || mobile.roleBlockDepthCells !== 24) failures.push("移动端导航或实验异常"); if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`); diff --git a/scripts/check-evaluation-browser.mjs b/scripts/check-evaluation-browser.mjs index 71ca2c7..06f7768 100644 --- a/scripts/check-evaluation-browser.mjs +++ b/scripts/check-evaluation-browser.mjs @@ -277,7 +277,7 @@ if (numeric(system.initial.success) <= numeric(system.initial.model) || numeric( if (numeric(system.cheap.success) >= numeric(system.initial.success) || numeric(system.cheap.cost) !== 4) failures.push("低预算没有降低成功率 / 成本"); if (numeric(system.locked.unsafe) !== 0 || numeric(system.locked.overrefusal) <= numeric(system.initial.overrefusal)) failures.push("安全壳没有展现危险服从 / 过拒权衡"); if (system.keyboardSelected !== "judge" || system.keyboardVisible !== "judge") failures.push("实验键盘 tab 导航异常"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); if (home.paperCount !== "486" || home.topicCount !== "17" || papers.total !== 486 || !papers.hasFilter || papers.visible < 80) failures.push("首页 / 论文库评测索引异常"); if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常"); if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`); diff --git a/scripts/check-inference-serving-browser.mjs b/scripts/check-inference-serving-browser.mjs index ddbf408..26fea9e 100644 --- a/scripts/check-inference-serving-browser.mjs +++ b/scripts/check-inference-serving-browser.mjs @@ -264,7 +264,7 @@ if (!fleet.k3.avoided.includes("320K") || fleet.k3.shortSlo !== "PROTECTED") fai if (!fleet.failed.state.includes("SECONDARY RE-PREFILL") || !fleet.failed.recompute.includes("FAILED PRIMARY")) failures.push("缓存故障没有触发原子失效后的重算"); if (fleet.bursty.shortSlo !== "VIOLATED") failures.push("平均并发阈值没有暴露长请求突发"); if (fleet.keyboardSelected !== "phase" || fleet.keyboardVisible !== "phase") failures.push("实验键盘 tab 导航异常"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); if (home.paperCount !== "486" || papers.total !== 486 || !papers.hasFilter || papers.visible !== 46) failures.push("论文库推理服务标签或总数异常"); if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常"); if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`); diff --git a/scripts/check-k3-attnres-forward-browser.mjs b/scripts/check-k3-attnres-forward-browser.mjs new file mode 100644 index 0000000..181a6ed --- /dev/null +++ b/scripts/check-k3-attnres-forward-browser.mjs @@ -0,0 +1,257 @@ +import { writeFileSync } from "node:fs"; + +const cdpPort = process.env.CDP_PORT ?? "9231"; +const baseUrl = process.env.SITE_URL ?? "http://127.0.0.1:4330"; +const pages = await fetch(`http://127.0.0.1:${cdpPort}/json/list`).then((response) => response.json()); +const page = pages.find((entry) => entry.type === "page"); +if (!page) throw new Error(`CDP ${cdpPort} 没有可用页面`); + +const socket = new WebSocket(page.webSocketDebuggerUrl); +await new Promise((resolve, reject) => { + socket.addEventListener("open", resolve, { once: true }); + socket.addEventListener("error", reject, { once: true }); +}); + +let nextId = 0; +const pending = new Map(); +const exceptions = []; +socket.addEventListener("message", (event) => { + const message = JSON.parse(event.data); + if (message.id && pending.has(message.id)) { + const { resolve, reject } = pending.get(message.id); + pending.delete(message.id); + if (message.error) reject(new Error(message.error.message)); + else resolve(message.result); + } + if (message.method === "Runtime.exceptionThrown") { + exceptions.push(message.params.exceptionDetails.exception?.description ?? message.params.exceptionDetails.text); + } +}); + +const command = (method, params = {}) => new Promise((resolve, reject) => { + const id = ++nextId; + pending.set(id, { resolve, reject }); + socket.send(JSON.stringify({ id, method, params })); +}); +const pause = (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds)); +const evaluate = async (expression) => { + const result = await command("Runtime.evaluate", { expression, returnByValue: true, awaitPromise: true }); + if (result.exceptionDetails) throw new Error(result.exceptionDetails.exception?.description ?? result.exceptionDetails.text); + return result.result.value; +}; +const navigate = async (path) => { + await command("Page.navigate", { url: `${baseUrl}${path}` }); + for (let attempt = 0; attempt < 100; attempt += 1) { + await pause(100); + if (await evaluate("document.readyState === 'complete'")) return; + } + throw new Error(`${path} 加载超时`); +}; +const screenshot = async (path) => { + const result = await command("Page.captureScreenshot", { format: "png", captureBeyondViewport: false }); + writeFileSync(path, Buffer.from(result.data, "base64")); +}; + +await command("Page.enable"); +await command("Runtime.enable"); +await command("Emulation.setDeviceMetricsOverride", { + width: 1440, + height: 1100, + deviceScaleFactor: 1, + mobile: false, +}); +await navigate("/k3/"); + +const desktop = await evaluate(`(() => { + const root = document.querySelector("[data-forward-lab]"); + root.scrollIntoView({ block: "start", behavior: "instant" }); + window.scrollBy(0, -78); + const text = (selector) => root.querySelector(selector)?.textContent.replace(/\\s+/g, " ").trim(); + const panel = () => root.querySelector("[data-forward-panel]:not([hidden])")?.dataset.forwardPanel; + const setSelect = (selector, value) => { + const node = root.querySelector(selector); + node.value = value; + node.dispatchEvent(new Event("change", { bubbles: true })); + }; + + const initial = { + panel: panel(), + tabs: root.querySelectorAll("[data-forward-tab]").length, + panels: root.querySelectorAll("[data-forward-panel]").length, + ledger: root.querySelectorAll(".forward-ledger article").length, + targetGroups: root.querySelectorAll(".group-map article.target").length, + spikeLayers: root.querySelectorAll(".group-map i.spike").length, + variants: root.querySelectorAll(".variant-grid article").length, + boundary: root.textContent.includes("K3 的训练尖峰已被定位") && + root.textContent.includes("Figure 5(c)") && + root.textContent.includes("训练期架构消融"), + }; + + root.querySelector('[data-forward-tab="trajectory"]').click(); + const trajectoryInitial = { + panel: panel(), + state: text("[data-forward-trajectory-state]"), + points: root.querySelectorAll("[data-forward-trajectory-series] circle").length, + lines: root.querySelectorAll("[data-forward-trajectory-series] polyline").length, + readouts: [...root.querySelectorAll("[data-forward-trajectory-readout] article")] + .map((node) => node.textContent.replace(/\\s+/g, " ").trim()), + }; + setSelect("[data-forward-trajectory-seed]", "2026073002"); + root.querySelector('[data-forward-trajectory-metric="peak_normalized"]').click(); + const trajectoryChanged = { + state: text("[data-forward-trajectory-state]"), + readouts: [...root.querySelectorAll("[data-forward-trajectory-readout] article")] + .map((node) => node.textContent.replace(/\\s+/g, " ").trim()), + }; + + root.querySelector('[data-forward-tab="gate"]').click(); + const gate = { + panel: panel(), + status: text(".status-banner"), + rows: root.querySelectorAll(".gate-table tbody tr").length, + passedRows: root.querySelectorAll(".gate-table tbody td.good:last-child").length, + quality: text(".gate-layout aside"), + claims: root.querySelectorAll(".claim-pair article").length, + }; + + root.querySelector('[data-forward-tab="interaction"]').click(); + const interactionInitial = { + panel: panel(), + state: text("[data-forward-interaction-state]"), + cells: root.querySelectorAll("[data-forward-interaction-cells] article").length, + summary: text("[data-forward-interaction-summary]"), + }; + root.querySelector('[data-forward-interaction-metric="peak_normalized"]').click(); + const interactionChanged = { + state: text("[data-forward-interaction-state]"), + summary: text("[data-forward-interaction-summary]"), + }; + + root.querySelector('[data-forward-tab="spectrum"]').click(); + const spectrumInitial = { + panel: panel(), + state: text("[data-forward-spectrum-state]"), + points: root.querySelectorAll("[data-forward-spectrum-points] circle").length, + contrast: text("[data-forward-spectrum-contrast]"), + peak: text("[data-forward-spectrum-peak]"), + layer: text("[data-forward-spectrum-layer]"), + audits: root.querySelectorAll(".audit-grid article").length, + replay: root.textContent.includes("scientific exact"), + }; + setSelect("[data-forward-spectrum-seed]", "2026073002"); + setSelect("[data-forward-spectrum-variant]", "uniform_groups_6_7_forward"); + const spectrumChanged = { + state: text("[data-forward-spectrum-state]"), + points: root.querySelectorAll("[data-forward-spectrum-points] circle").length, + contrast: text("[data-forward-spectrum-contrast]"), + peak: text("[data-forward-spectrum-peak]"), + layer: text("[data-forward-spectrum-layer]"), + }; + + const first = root.querySelector('[data-forward-tab="contract"]'); + first.focus(); + first.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowRight", bubbles: true })); + const keyboard = { + selected: root.querySelector('[data-forward-tab][aria-selected="true"]').dataset.forwardTab, + panel: panel(), + }; + + return { + initial, trajectoryInitial, trajectoryChanged, gate, + interactionInitial, interactionChanged, spectrumInitial, spectrumChanged, keyboard, + documentOverflow: document.documentElement.scrollWidth - document.documentElement.clientWidth, + rootOverflow: root.scrollWidth - root.clientWidth, + }; +})()`); +await pause(180); +await screenshot("/tmp/llm-atlas-k3-attnres-forward-desktop.png"); + +await command("Emulation.setDeviceMetricsOverride", { + width: 390, + height: 844, + deviceScaleFactor: 1, + mobile: true, +}); +await navigate("/k3/"); +const mobile = await evaluate(`(() => { + const root = document.querySelector("[data-forward-lab]"); + root.scrollIntoView({ block: "start", behavior: "instant" }); + window.scrollBy(0, -64); + root.querySelector('[data-forward-tab="gate"]').click(); + const table = root.querySelector(".gate-table-wrap"); + const gateTableScrolls = table.scrollWidth > table.clientWidth; + root.querySelector('[data-forward-tab="spectrum"]').click(); + return { + tabs: root.querySelectorAll("[data-forward-tab]").length, + visiblePanel: root.querySelector("[data-forward-panel]:not([hidden])")?.dataset.forwardPanel, + points: root.querySelectorAll("[data-forward-spectrum-points] circle").length, + audits: root.querySelectorAll(".audit-grid article").length, + gateTableScrolls, + documentOverflow: document.documentElement.scrollWidth - document.documentElement.clientWidth, + rootOverflow: root.scrollWidth - root.clientWidth, + }; +})()`); +await pause(180); +await screenshot("/tmp/llm-atlas-k3-attnres-forward-mobile.png"); + +const failures = []; +if (desktop.initial.panel !== "contract" || desktop.initial.tabs !== 5 || desktop.initial.panels !== 5 || + desktop.initial.ledger !== 6 || desktop.initial.targetGroups !== 2 || + desktop.initial.spikeLayers !== 5 || desktop.initial.variants !== 4) { + failures.push("五视图、账本、group 或 selector map 结构异常"); +} +if (!desktop.initial.boundary) failures.push("reduced-model / K3 / Figure 5(c) claim boundary 缺失"); +if (desktop.trajectoryInitial.panel !== "trajectory" || desktop.trajectoryInitial.points !== 24 || + desktop.trajectoryInitial.lines !== 4 || desktop.trajectoryInitial.readouts.length !== 4 || + !desktop.trajectoryInitial.state.includes("2026073001") || + !desktop.trajectoryInitial.readouts.some((value) => value.includes("+73.5%"))) { + failures.push("seed 1 contrast 训练轨迹异常"); +} +if (!desktop.trajectoryChanged.state.includes("2026073002") || + !desktop.trajectoryChanged.state.includes("PEAK / MEAN") || + !desktop.trajectoryChanged.readouts.some((value) => value.includes("+62.0%"))) { + failures.push("trajectory seed / metric 切换异常"); +} +if (desktop.gate.panel !== "gate" || desktop.gate.rows !== 6 || desktop.gate.passedRows !== 6 || + desktop.gate.claims !== 2 || !desktop.gate.status.includes("ATTENUATION ESTABLISHED") || + !desktop.gate.status.includes("6 / 6") || !desktop.gate.status.includes("4 / 4") || + !desktop.gate.quality.includes("+0.0066")) { + failures.push("冻结主门或 BPC quality readout 异常"); +} +if (desktop.interactionInitial.panel !== "interaction" || desktop.interactionInitial.cells !== 3 || + !desktop.interactionInitial.state.includes("8,000") || + !desktop.interactionInitial.summary.includes("-0.367") || + !desktop.interactionChanged.state.includes("PEAK / MEAN") || + !desktop.interactionChanged.summary.includes("-0.170")) { + failures.push("I67 描述性 residual 切换异常"); +} +if (desktop.spectrumInitial.panel !== "spectrum" || desktop.spectrumInitial.points !== 32 || + desktop.spectrumInitial.contrast !== "3.046×" || desktop.spectrumInitial.peak !== "3.219×" || + desktop.spectrumInitial.layer !== "L21" || desktop.spectrumInitial.audits !== 5 || + !desktop.spectrumInitial.replay) { + failures.push("reference 32 层谱或 replay 审计异常"); +} +if (desktop.spectrumChanged.points !== 32 || !desktop.spectrumChanged.state.includes("2026073002") || + !desktop.spectrumChanged.state.includes("GROUPS 6+7") || + desktop.spectrumChanged.contrast !== "0.758×" || desktop.spectrumChanged.peak !== "1.375×" || + desktop.spectrumChanged.layer !== "L9") { + failures.push("joint variant spectrum 切换异常"); +} +if (desktop.keyboard.selected !== "trajectory" || desktop.keyboard.panel !== "trajectory") { + failures.push("键盘 tab 切换异常"); +} +if (desktop.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("页面出现文档级横向溢出"); +if (desktop.rootOverflow > 1 || mobile.rootOverflow > 1) failures.push("Round 08 实验室出现横向溢出"); +if (mobile.tabs !== 5 || mobile.visiblePanel !== "spectrum" || mobile.points !== 32 || + mobile.audits !== 5 || !mobile.gateTableScrolls) { + failures.push("390px 移动端布局或局部表格滚动异常"); +} +if (exceptions.length) failures.push(`运行时异常：${exceptions.join(" | ")}`); + +console.log(JSON.stringify({ desktop, mobile, exceptions }, null, 2)); +socket.close(); +if (failures.length) { + console.error(`FAIL K3 AttnRes forward-training browser\n- ${failures.join("\n- ")}`); + process.exit(1); +} +console.log("PASS K3 AttnRes forward-training browser interactions"); diff --git a/scripts/check-k3-attnres-forward-data.mjs b/scripts/check-k3-attnres-forward-data.mjs new file mode 100644 index 0000000..e142292 --- /dev/null +++ b/scripts/check-k3-attnres-forward-data.mjs @@ -0,0 +1,106 @@ +import { createHash } from "node:crypto"; +import { readdirSync, readFileSync } from "node:fs"; + +const hash = (bytes) => createHash("sha256").update(bytes).digest("hex"); +const read = (path) => { + const bytes = readFileSync(new URL(path, import.meta.url)); + return { bytes, json: JSON.parse(bytes), sha256: hash(bytes) }; +}; +const aggregate = read("../src/data/k3-attnres-forward.json"); +const compact = read("../src/data/k3-attnres-forward-compact.json"); +const reproduction = read("../experiments/k3/attnres_forward/reproduction.json"); +const manifest = read("../experiments/k3/attnres_forward/manifest.json"); +const rawDirectory = new URL("../experiments/k3/attnres_forward/results/raw/", import.meta.url); +const failures = []; +const expect = (condition, message) => { + if (!condition) failures.push(message); +}; +const close = (actual, expected, tolerance = 1e-15) => + Math.abs(actual - expected) <= tolerance; + +expect(aggregate.sha256 === "f8df928adb8a563d851bb3c1abbf40bcada33626d9177821e4341d854980a097", "aggregate physical SHA-256 changed"); +expect(compact.sha256 === "664f6d6226df7c0c9aba6314d54a1cb8ae90f6922016dfa7c7a823729606f0c1", "compact physical SHA-256 changed"); +expect(reproduction.sha256 === "b029c333596dd1de957efc2b42f7ebfc1d32f241c687f06021c4d08ffa27b1d0", "reproduction physical SHA-256 changed"); +expect(manifest.sha256 === "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712", "manifest physical SHA-256 changed"); + +expect(aggregate.json.canonical_sha256_without_self === "eecf05c623e473ec5eba8afa54d555d2733a0af2fc5488398f1da095206d50c8", "aggregate canonical SHA-256 changed"); +expect(compact.json.canonical_sha256_without_self === "c3e672adb6879f99efccf3bd2e0fabaab1a1159225a8ddaf4f5f9ea7a002784b", "compact canonical SHA-256 changed"); +expect(reproduction.json.canonical_sha256_without_self === "57346df80c0d76bd1d306d5fa213feed74ac2094237c22ebcb49f16ea16437e0", "reproduction canonical SHA-256 changed"); + +expect(compact.json.protocol_id === "llm-atlas-k3-attnres-forward-training-v1", "protocol identity mismatch"); +expect(compact.json.status === "forward_training_attenuation_established_within_reduced_protocol", "frozen status changed"); +expect(compact.json.primary_step === 8000, "primary step changed"); +expect(compact.json.spike_layers_1based.join(",") === "21,22,23,24,25", "spike window changed"); +expect(compact.json.thresholds.material_relative_drop === 0.2, "attenuation threshold changed"); +expect(compact.json.processed_target_bytes.formal_12_cells === 786432000, "formal target bytes changed"); +expect(compact.json.processed_target_bytes.primary_replay === 65536000, "replay target bytes changed"); +expect(compact.json.processed_target_bytes.total === 851968000, "new target bytes changed"); +expect(compact.json.aggregate_sha256 === aggregate.json.canonical_sha256_without_self, "compact→aggregate canonical link mismatch"); +expect(reproduction.json.scientific_payload_sha256 === compact.json.replay.scientific_payload_sha256, "reproduction→replay hash link mismatch"); +expect(reproduction.json.passed && compact.json.replay.passed, "full replay is not exact"); +expect(reproduction.json.post_result_grok_review.blocking_errors === 0, "post-result audit reports a blocking error"); +expect(reproduction.json.post_result_grok_review.status_confirmed, "post-result audit did not confirm status"); +expect(reproduction.json.post_result_grok_review.replay_confirmed, "post-result audit did not confirm replay"); +expect(compact.json.metadata_warnings.length === 0, "historical pairing metadata warning appeared"); + +const rawNames = readdirSync(rawDirectory).filter((name) => name.endsWith(".json")).sort(); +expect(rawNames.length === 13, "raw matrix is not 12 formal + 1 replay"); +const expectedRaw = [ + ...aggregate.json.input_files.formal.map((item) => item), + { ...aggregate.json.input_files.replay, variant: "uniform_groups_6_7_forward", seed: 2026073001 }, +]; +for (const item of expectedRaw) { + const name = item.path.split("/").at(-1); + const raw = read(`../experiments/k3/attnres_forward/results/raw/${name}`); + expect(raw.sha256 === item.sha256, `${name} physical SHA-256 mismatch`); + expect(/^[0-9a-f]{64}$/.test(raw.json.canonical_sha256_without_self), `${name} canonical self-hash missing`); + expect(raw.json.forward_intervention.passed, `${name} forward audit failed`); + expect(raw.json.forward_intervention.forward_calls === 8054, `${name} forward census changed`); + expect(raw.json.target_bytes_seen === 65536000, `${name} target bytes changed`); +} + +const primary = compact.json.primary; +expect(primary.material_response_passed, "primary material response failed"); +expect(primary.passed_cells === 6 && primary.required_cells === 6, "primary attenuation is not 6/6"); +expect(primary.cells.every((cell) => cell.relative_drop >= 0.2 && cell.passed), "a primary attenuation cell fell below 20%"); +expect(primary.quality.passed && primary.quality.passed_checks === 4, "BPC quality gate is not 4/4"); +expect(close(primary.quality.mean_delta_bpc, 0.0065784582165467525), "mean BPC delta changed"); +expect(close(Math.min(...primary.cells.filter((cell) => cell.metric === "spike_contrast").map((cell) => cell.relative_drop)), 0.6211080443849011), "minimum contrast drop changed"); +expect(close(Math.min(...primary.cells.filter((cell) => cell.metric === "peak_normalized").map((cell) => cell.relative_drop)), 0.3234439150908039), "minimum peak drop changed"); +expect(Object.values(compact.json.secondary_status).every((value) => value === "secondary_material_response"), "secondary status changed"); +expect(compact.json.trajectories.length === 90, "trajectory cell count changed"); +expect(compact.json.final_spectra.length === 15, "final spectrum count changed"); +expect(compact.json.final_spectra.every((item) => item.normalized.length === 32), "a final spectrum is not 32 layers"); +expect(compact.json.interaction.cells.length === 36, "interaction cell count changed"); +expect(compact.json.interaction.summaries.length === 12, "interaction summary count changed"); +expect(close( + compact.json.interaction.summaries.find((item) => + item.step === 8000 && item.metric === "spike_contrast").mean_interaction_residual, + -0.3670379672721313, +), "final contrast I67 changed"); +expect(close( + compact.json.interaction.summaries.find((item) => + item.step === 8000 && item.metric === "peak_normalized").mean_interaction_residual, + -0.17044758609080035, +), "final peak I67 changed"); + +if (failures.length) { + console.error(`FAIL K3 AttnRes forward-training data\n- ${failures.join("\n- ")}`); + process.exit(1); +} + +console.log(JSON.stringify({ + protocol: compact.json.protocol_id, + status: compact.json.status, + rawRuns: rawNames.length, + attenuation: `${primary.passed_cells}/${primary.required_cells}`, + quality: `${primary.quality.passed_checks}/${primary.quality.required_checks}`, + replayExact: compact.json.replay.passed, + bytes: compact.json.processed_target_bytes, + hashes: { + aggregate: aggregate.sha256, + compact: compact.sha256, + reproduction: reproduction.sha256, + }, +}, null, 2)); +console.log("PASS K3 AttnRes forward-training frozen data"); diff --git a/scripts/check-k3-browser.mjs b/scripts/check-k3-browser.mjs index 72e9d9a..5f7b27d 100644 --- a/scripts/check-k3-browser.mjs +++ b/scripts/check-k3-browser.mjs @@ -88,6 +88,10 @@ const overview = await evaluate(`(() => ({ localPathTabs: document.querySelectorAll("[data-local-tab]").length, localPathPanels: document.querySelectorAll("[data-local-panel]").length, localPathVerdict: document.querySelector("#attnres-local-path")?.textContent.includes("localization 未建立"), + forwardTabs: document.querySelectorAll("[data-forward-tab]").length, + forwardPanels: document.querySelectorAll("[data-forward-panel]").length, + forwardVerdict: document.querySelector("#attnres-forward")?.textContent.includes("6 / 6 PASS") && + document.querySelector("#attnres-forward")?.textContent.includes("reduced protocol only"), nativeVisionCorrected: document.body.textContent.includes("MoonViT‑V2 从头训练") && document.body.textContent.includes("同一个 next-token prediction objective"), staleVisionClaim: document.body.textContent.includes("先固定语言模型训练视觉组件"), @@ -300,8 +304,9 @@ const mobile = await evaluate(`(() => { gradientTabs: document.querySelectorAll("[data-gradient-tab]").length, spikeTabs: document.querySelectorAll("[data-spike-tab]").length, localPathTabs: document.querySelectorAll("[data-local-tab]").length, + forwardTabs: document.querySelectorAll("[data-forward-tab]").length, offenders: [...document.querySelectorAll("body *")] - .filter((node) => !node.closest(".paper-chain, .spec-table-wrap, .cache-strip, .architecture-explorer, [data-k3-lab], [data-k3-artifact-lab], [data-attnres-lab], [data-gradient-lab], [data-spike-lab], [data-local-path-lab]")) + .filter((node) => !node.closest(".paper-chain, .spec-table-wrap, .cache-strip, .architecture-explorer, [data-k3-lab], [data-k3-artifact-lab], [data-attnres-lab], [data-gradient-lab], [data-spike-lab], [data-local-path-lab], [data-forward-lab]")) .filter((node) => node.getBoundingClientRect().right > document.documentElement.clientWidth + 1) .slice(0, 15) .map((node) => ({ @@ -332,7 +337,7 @@ console.log(JSON.stringify(report, null, 2)); const numeric = (text) => Number.parseFloat(text.replaceAll(",", "").replace("−", "-")); const failures = []; if (!overview.title.includes("因果环节")) failures.push("K3 二轮标题异常"); -if (overview.sections !== 36 || overview.tocLinks !== 36) failures.push("35 个编号专题加阅读链的目录结构异常"); +if (overview.sections !== 37 || overview.tocLinks !== 37) failures.push("36 个编号专题加阅读链的目录结构异常"); if (overview.ledgers !== 32 || overview.reportMap !== 9) failures.push("32 张问题账或报告地图异常"); if (overview.figureAtlas !== 21 || overview.paperLinks !== 100 || overview.paperGroups < 12) failures.push("图表审计或 100 节点阅读链异常"); if (overview.labTabs !== 8 || overview.labPanels !== 8) failures.push("八联实验结构异常"); @@ -341,6 +346,7 @@ if (overview.attnresTabs !== 5 || overview.attnresPanels !== 5) failures.push("A if (overview.gradientTabs !== 5 || overview.gradientPanels !== 5) failures.push("AttnRes 梯度定义扩展五视图异常"); if (overview.spikeTabs !== 5 || overview.spikePanels !== 5) failures.push("AttnRes 尖峰路径五视图异常"); if (overview.localPathTabs !== 5 || overview.localPathPanels !== 5 || !overview.localPathVerdict) failures.push("AttnRes 局部路径五视图或冻结判定异常"); +if (overview.forwardTabs !== 5 || overview.forwardPanels !== 5 || !overview.forwardVerdict) failures.push("AttnRes 训练期前向五视图或冻结判定异常"); if (!overview.nativeVisionCorrected || overview.staleVisionClaim) failures.push("原生多模态纠错未生效或旧错误残留"); if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出"); if (labs.memoryInitial.panel !== "memory" || numeric(labs.memoryInitial.additiveError) <= numeric(labs.memoryInitial.deltaError)) failures.push("Delta memory 初始递推异常"); @@ -365,7 +371,7 @@ if (artifacts.parameterChanged.shape !== "[96,128] F32" || !artifacts.parameterC if (artifacts.reproductionInitial.panel !== "reproduction" || numeric(artifacts.reproductionInitial.speedup) !== 1.85 || numeric(artifacts.reproductionInitial.localMean) < 2.6 || !artifacts.reproductionInitial.exactSuite || numeric(artifacts.reproductionInitial.cv) < 2) failures.push("FlashKDA H20、本机 exact suite 或 router 初始探针异常"); if (numeric(artifacts.reproductionChanged.speedup) !== 3.27 || numeric(artifacts.reproductionChanged.flash) !== 0.7064 || numeric(artifacts.reproductionChanged.localMean) >= numeric(artifacts.reproductionInitial.localMean) || !artifacts.reproductionChanged.localMode.includes("FP32 state") || numeric(artifacts.reproductionChanged.cv) <= numeric(artifacts.reproductionInitial.cv) || numeric(artifacts.reproductionChanged.zero) <= numeric(artifacts.reproductionInitial.zero)) failures.push("GB200 benchmark、本机 varlen/state 或 synthetic router counterexample 未更新"); if (artifacts.keyboardSelected !== "tensors" || artifacts.keyboardVisible !== "tensors") failures.push("开放工件键盘 tab 导航异常"); -if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 8 || mobile.artifactTabs !== 4 || mobile.artifactLayers !== 93 || mobile.attnresTabs !== 5 || mobile.gradientTabs !== 5 || mobile.spikeTabs !== 5 || mobile.localPathTabs !== 5) failures.push("移动端导航或实验异常"); +if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 8 || mobile.artifactTabs !== 4 || mobile.artifactLayers !== 93 || mobile.attnresTabs !== 5 || mobile.gradientTabs !== 5 || mobile.spikeTabs !== 5 || mobile.localPathTabs !== 5 || mobile.forwardTabs !== 5) failures.push("移动端导航或实验异常"); if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`); if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`); diff --git a/scripts/check-multimodal-browser.mjs b/scripts/check-multimodal-browser.mjs index a11fcfa..6124465 100644 --- a/scripts/check-multimodal-browser.mjs +++ b/scripts/check-multimodal-browser.mjs @@ -246,7 +246,7 @@ if (ocr.unreported.status !== "OUT OF EVIDENCE" || ocr.unreported.accuracy !== " if (loop.toolsStart.state !== "OPEN" || loop.toolsEnd.state !== "VERIFIED" || loop.toolsEnd.evidence !== "97%" || loop.toolsEnd.tools !== "3") failures.push("vision-in-the-loop 终局异常"); if (loop.cotEnd.state !== "FAILED" || !loop.cotEnd.takeaway.includes("不能凭空增加")) failures.push("文字 CoT 与新观察没有分开"); if (loop.keyboardSelected !== "connector" || loop.keyboardVisible !== "connector") failures.push("实验键盘 tab 导航异常"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); if (home.paperCount !== "486" || papers.total !== 486 || !papers.hasFilter || papers.multimodalVisible < 59) failures.push("论文库多模态标签或总数异常"); if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常"); if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`); diff --git a/scripts/check-numerics-browser.mjs b/scripts/check-numerics-browser.mjs index 9256462..c35eb5c 100644 --- a/scripts/check-numerics-browser.mjs +++ b/scripts/check-numerics-browser.mjs @@ -277,7 +277,7 @@ if (layout.navLinks !== 20 || mobile.mobileLinks !== 20 || home.navLinks !== 20) if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出"); if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`); if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强") || home.firstHref !== "/k3/") { +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后") || home.firstHref !== "/k3/") { failures.push("首页 Transformer 新章入口异常"); } if (home.paperCount !== "486") failures.push(`首页论文总数异常：${home.paperCount}`); diff --git a/scripts/check-reasoning-browser.mjs b/scripts/check-reasoning-browser.mjs index dcfcd1b..17d5691 100644 --- a/scripts/check-reasoning-browser.mjs +++ b/scripts/check-reasoning-browser.mjs @@ -289,7 +289,7 @@ if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentO } if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`); if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强")) failures.push("首页 K3 首发入口异常"); +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后")) failures.push("首页 K3 首发入口异常"); if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`); socket.close(); diff --git a/scripts/check-representation-browser.mjs b/scripts/check-representation-browser.mjs index da1e911..081b01e 100644 --- a/scripts/check-representation-browser.mjs +++ b/scripts/check-representation-browser.mjs @@ -288,7 +288,7 @@ if (numeric(residual.attnres.states) !== 9 || !residual.attnres.routeExplain.inc if (!residual.clamp.activation.includes("V4") || !residual.clamp.bound.includes("100")) failures.push("DeepSeek-V4 clamp 展示异常"); if (!residual.situ.activation.includes("KIMI") || !residual.situ.bound.includes("100")) failures.push("K3 SiTU 上界展示异常"); if (residual.keyboardSelected !== "position" || residual.keyboardVisible !== "position") failures.push("实验键盘 tab 导航异常"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); if (home.paperCount !== "486" || home.topicCount !== "17" || papers.total !== 486 || !papers.hasFilter || papers.visible < 30) failures.push("首页 / 论文库表示索引异常"); if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常"); if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`); diff --git a/scripts/check-scaling-browser.mjs b/scripts/check-scaling-browser.mjs index af88a6f..07aa3e1 100644 --- a/scripts/check-scaling-browser.mjs +++ b/scripts/check-scaling-browser.mjs @@ -273,7 +273,7 @@ if (layout.navLinks !== 20 || mobile.mobileLinks !== 20 || home.navLinks !== 20) if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出"); if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`); if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强") || home.firstHref !== "/k3/") { +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后") || home.firstHref !== "/k3/") { failures.push("首页 Transformer 新章入口异常"); } if (home.paperCount !== "486") failures.push(`首页论文总数异常：${home.paperCount}`); diff --git a/scripts/check-training-systems-browser.mjs b/scripts/check-training-systems-browser.mjs index 855406f..bc7ee48 100644 --- a/scripts/check-training-systems-browser.mjs +++ b/scripts/check-training-systems-browser.mjs @@ -233,7 +233,7 @@ if (layout.articleSections !== 16 || layout.paperLinks !== 37 || layout.labTabs if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出"); if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`); if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强")) failures.push("首页 K3 首发入口异常"); +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后")) failures.push("首页 K3 首发入口异常"); if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`); socket.close(); diff --git a/scripts/check-transformer-browser.mjs b/scripts/check-transformer-browser.mjs index ce6698b..0b0f9e7 100644 --- a/scripts/check-transformer-browser.mjs +++ b/scripts/check-transformer-browser.mjs @@ -236,7 +236,7 @@ if (block.family.trim() !== "Hybrid MoE" || !block.kv.includes("3 KDA : 1 Gated if (!block.path.some((step) => step.includes("KDA × 3")) || !block.note.includes("AttnRes")) failures.push("K3 Block 路径异常"); if (block.context.trim() !== "128K" || numeric(block.mha) !== 400 || numeric(block.kda) !== 1) failures.push("KV 成本缩放异常"); if (block.keyboardSelected !== "block" || block.keyboardVisible !== "block") failures.push("实验 tab 键盘导航异常"); -if (home.releaseCards !== 17 || !home.firstRelease.includes("16 个局部 mixer 单侧证据很强") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); +if (home.releaseCards !== 17 || !home.firstRelease.includes("局部 uniform routing 进入完整训练后") || home.firstHref !== "/k3/") failures.push("首页 K3 首发入口异常"); if (home.paperCount !== "486" || papers.total !== 486 || papers.transformerVisible < 30) failures.push("论文库或首页论文数量异常"); if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常"); if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`); diff --git a/src/components/K3AttnResForwardLab.astro b/src/components/K3AttnResForwardLab.astro new file mode 100644 index 0000000..91d0fa9 --- /dev/null +++ b/src/components/K3AttnResForwardLab.astro @@ -0,0 +1,703 @@ +--- +import rawLab from "@/data/k3-attnres-forward-compact.json"; + +const lab = rawLab as any; +const json = JSON.stringify(lab).replaceAll("<", "\\u003c"); +const seeds = [...new Set(lab.final_spectra.map((item: any) => item.seed))] as number[]; +const variants = [ + "uniform_group_6_forward", + "uniform_group_7_forward", + "uniform_groups_6_7_forward", + "uniform_group_7_mlp_forward", +]; +const variantLabels: Record = { + learned_reference: "HISTORICAL LEARNED REFERENCE", + uniform_group_6_forward: "GROUP 6 · UNIFORM FORWARD", + uniform_group_7_forward: "GROUP 7 · UNIFORM FORWARD", + uniform_groups_6_7_forward: "GROUPS 6+7 · UNIFORM FORWARD", + uniform_group_7_mlp_forward: "GROUP 7 MLP · UNIFORM FORWARD", +}; +const statusLabels: Record = { + forward_training_attenuation_established_within_reduced_protocol: + "ATTENUATION ESTABLISHED · REDUCED PROTOCOL", + attenuation_not_established: "ATTENUATION NOT ESTABLISHED", + quality_guard_failed: "QUALITY GUARD FAILED", + attenuation_and_quality_failed: "ATTENUATION + QUALITY FAILED", +}; +const statusLabel = statusLabels[lab.status] ?? lab.status; +const primary = lab.primary; +const quality = primary.quality; +const meanDrop = (effect: any, metric: string) => { + const cells = effect.cells.filter((cell: any) => cell.metric === metric); + return cells.reduce((sum: number, cell: any) => sum + cell.relative_drop, 0) / cells.length; +}; +const percent = (value: number) => `${value >= 0 ? "+" : "−"}${Math.abs(value * 100).toFixed(1)}%`; +const shortHash = (value: string) => `${value.slice(0, 10)}…${value.slice(-8)}`; +--- + +
+
+ ROUND 08 / TRAIN-TIME FORWARD +
+

不再只改 diagnostic backward：让局部 uniform routing 真正进入 8,000-step 训练

+

4 variants × 3 seeds · 1 historical paired reference · 1 full replay · frozen analyzer

+
+ REDUCED-MODEL ARCHITECTURE ABLATION +
+ +
+
NEW BYTES851.968M

12 formal + 1 replay

+
PRIMARY CELLS{primary.passed_cells} / {primary.required_cells}

contrast + peak

+
QUALITY{quality.passed_checks} / {quality.required_checks}

BPC degradation screen

+
REPLAY{lab.replay.passed ? "EXACT" : "FAILED"}

scientific payload

+
SELECTED PARAMSUNREACHABLE

0 hooks · 0 optimizer state

+
+ STATUS{primary.material_response_passed ? "PASS" : "NOT ESTABLISHED"}

reduced protocol only

+
+
+ +
+ + + + + +
+ +
+
+
I / WHAT ACTUALLY CHANGED

这是训练期架构消融，不是“只改变 forward”的纯因果实验

+

选中 mixer 每一次 train / eval / diagnostic 都改成参数无关的均匀读取；新的表示自然改变 backward 与后续更新。

+
+
+
+ LEARNED DEPTH MIXER + keys = RMSNorm(sources) + w = softmax(query · keys) + output = Σ wᵢ · sourceᵢ +
+ → +
+ SELECTED UNIFORM MIXER + logits = zeros(N, B, T) + w = softmax(logits) = 1 / N + output = Σ sourceᵢ / N +
+
+
+ {[1,2,3,4,5,6,7,8].map((group) => ( +
+ GROUP {group} + L{(group - 1) * 4 + 1}–{group * 4} +
+ {[0,1,2,3].map((offset) => { + const layer = (group - 1) * 4 + offset + 1; + return = 21 && layer <= 25 }}>{layer}; + })} +
+ {group === 6 ? "#40–47" : group === 7 ? "#48–55" : "LEARNED"} +
+ ))} +
+
+ {variants.map((variant) => { + const effect = lab.effects[variant]; + return ( +
+ {variant === "uniform_groups_6_7_forward" ? "PRIMARY" : "SECONDARY"} + {variantLabels[variant]} +
+
CONTRAST
{percent(meanDrop(effect, "spike_contrast"))}
+
PEAK
{percent(meanDrop(effect, "peak_normalized"))}
+
STATUS
{effect.material_response_passed ? "PASS" : "NOT EST."}
+
+
+ ); + })} +
+
+
SELECTED QUERY / KEY NORM留在 AdamW param groups

保持 optimizer 结构与 reference 一致。

+ × +
COMPUTATION GRAPH没有被 forward 调用

不是 stop-gradient；参数结构性不可达。

+ → +
FINAL AUDIT0 hook · 0 state · byte-exact init

不能写成“训练了但没有移动”。

+
+
+ + + + + + + + + + +
+ + + + diff --git a/src/data/k3-attnres-forward-compact.json b/src/data/k3-attnres-forward-compact.json new file mode 100644 index 0000000..c8dc6fb --- /dev/null +++ b/src/data/k3-attnres-forward-compact.json @@ -0,0 +1,3602 @@ +{ + "aggregate_sha256": "eecf05c623e473ec5eba8afa54d555d2733a0af2fc5488398f1da095206d50c8", + "canonical_sha256_without_self": "c3e672adb6879f99efccf3bd2e0fabaab1a1159225a8ddaf4f5f9ea7a002784b", + "effects": { + "uniform_group_6_forward": { + "attenuation_passed": true, + "cells": [ + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.04626897714722, + "relative_drop": 0.5800152615825866, + "seed": 2026073001, + "variant": 1.2793864795162566 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.219066994145898, + "relative_drop": 0.24116240807746625, + "seed": 2026073001, + "variant": 2.442749046074982 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.3328477735441013, + "relative_drop": 0.6578260733512802, + "seed": 2026073002, + "variant": 1.1404136095960284 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.6189242296270336, + "relative_drop": 0.4994801034992014, + "seed": 2026073002, + "variant": 1.8113435808571552 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 1.8814916302334765, + "relative_drop": 0.4463254057279825, + "seed": 2026073003, + "variant": 1.0417341149957169 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 2.2882673690329693, + "relative_drop": 0.2472255709371281, + "seed": 2026073003, + "variant": 1.7225491622669935 + } + ], + "material_response_passed": true, + "passed_cells": 6, + "quality": { + "mean_delta_bpc": 0.009688047863022131, + "mean_maximum": 0.03, + "mean_passed": true, + "passed": true, + "passed_checks": 4, + "per_seed": { + "2026073001": { + "delta_bpc": 0.006538199944357226, + "passed": true, + "reference_bpc": 1.7123525940760473, + "variant_bpc": 1.7188907940204046 + }, + "2026073002": { + "delta_bpc": 0.012622129214292066, + "passed": true, + "reference_bpc": 1.7093240655856785, + "variant_bpc": 1.7219461947999706 + }, + "2026073003": { + "delta_bpc": 0.009903814430417102, + "passed": true, + "reference_bpc": 1.7030966812870931, + "variant_bpc": 1.7130004957175102 + } + }, + "per_seed_maximum": 0.05, + "required_checks": 4 + }, + "required_cells": 6, + "threshold": 0.2, + "variant": "uniform_group_6_forward" + }, + "uniform_group_7_forward": { + "attenuation_passed": true, + "cells": [ + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.04626897714722, + "relative_drop": 0.5886666348497648, + "seed": 2026073001, + "variant": 1.2530320695227308 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.219066994145898, + "relative_drop": 0.40991690985788126, + "seed": 2026073001, + "variant": 1.899516999280113 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.3328477735441013, + "relative_drop": 0.6029454996221533, + "seed": 2026073002, + "variant": 1.323322207559972 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.6189242296270336, + "relative_drop": 0.42497271322062063, + "seed": 2026073002, + "variant": 2.0809801808225887 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 1.8814916302334765, + "relative_drop": 0.4164606288913148, + "seed": 2026073003, + "variant": 1.0979244426526977 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 2.2882673690329693, + "relative_drop": 0.29135113042743926, + "seed": 2026073003, + "variant": 1.6215780843449914 + } + ], + "material_response_passed": true, + "passed_cells": 6, + "quality": { + "mean_delta_bpc": 0.0022818873094791283, + "mean_maximum": 0.03, + "mean_passed": true, + "passed": true, + "passed_checks": 4, + "per_seed": { + "2026073001": { + "delta_bpc": -0.003005805282623397, + "passed": true, + "reference_bpc": 1.7123525940760473, + "variant_bpc": 1.709346788793424 + }, + "2026073002": { + "delta_bpc": 0.005744005560504251, + "passed": true, + "reference_bpc": 1.7093240655856785, + "variant_bpc": 1.7150680711461828 + }, + "2026073003": { + "delta_bpc": 0.004107461650556532, + "passed": true, + "reference_bpc": 1.7030966812870931, + "variant_bpc": 1.7072041429376497 + } + }, + "per_seed_maximum": 0.05, + "required_checks": 4 + }, + "required_cells": 6, + "threshold": 0.2, + "variant": "uniform_group_7_forward" + }, + "uniform_group_7_mlp_forward": { + "attenuation_passed": true, + "cells": [ + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.04626897714722, + "relative_drop": 0.5659319887472724, + "seed": 2026073001, + "variant": 1.3222879166511747 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.219066994145898, + "relative_drop": 0.37304900548175707, + "seed": 2026073001, + "variant": 2.0181972534006216 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.3328477735441013, + "relative_drop": 0.5738819822015593, + "seed": 2026073002, + "variant": 1.420186486886559 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.6189242296270336, + "relative_drop": 0.3924400935136531, + "seed": 2026073002, + "variant": 2.1987132665333755 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 1.8814916302334765, + "relative_drop": 0.39856400381461277, + "seed": 2026073003, + "variant": 1.1315967929439392 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 2.2882673690329693, + "relative_drop": 0.27376190068536765, + "seed": 2026073003, + "variant": 1.661826944810198 + } + ], + "material_response_passed": true, + "passed_cells": 6, + "quality": { + "mean_delta_bpc": -0.00037419125261473063, + "mean_maximum": 0.03, + "mean_passed": true, + "passed": true, + "passed_checks": 4, + "per_seed": { + "2026073001": { + "delta_bpc": 0.002418420036432778, + "passed": true, + "reference_bpc": 1.7123525940760473, + "variant_bpc": 1.7147710141124801 + }, + "2026073002": { + "delta_bpc": 0.0017558568739839409, + "passed": true, + "reference_bpc": 1.7093240655856785, + "variant_bpc": 1.7110799224596625 + }, + "2026073003": { + "delta_bpc": -0.005296850668260911, + "passed": true, + "reference_bpc": 1.7030966812870931, + "variant_bpc": 1.6977998306188322 + } + }, + "per_seed_maximum": 0.05, + "required_checks": 4 + }, + "required_cells": 6, + "threshold": 0.2, + "variant": "uniform_group_7_mlp_forward" + }, + "uniform_groups_6_7_forward": { + "attenuation_passed": true, + "cells": [ + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.04626897714722, + "relative_drop": 0.7354105664383516, + "seed": 2026073001, + "variant": 0.8060105831398049 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.219066994145898, + "relative_drop": 0.5540266425004673, + "seed": 2026073001, + "variant": 1.4356181153951746 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.3328477735441013, + "relative_drop": 0.7725064680163986, + "seed": 2026073002, + "variant": 0.75820131156723 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.6189242296270336, + "relative_drop": 0.620048308391513, + "seed": 2026073002, + "variant": 1.3750163828497322 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 1.8814916302334765, + "relative_drop": 0.6211080443849011, + "seed": 2026073003, + "variant": 0.7128820432526025 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 2.2882673690329693, + "relative_drop": 0.3234439150908039, + "seed": 2026073003, + "variant": 1.5481412124184124 + } + ], + "material_response_passed": true, + "passed_cells": 6, + "quality": { + "mean_delta_bpc": 0.0065784582165467525, + "mean_maximum": 0.03, + "mean_passed": true, + "passed": true, + "passed_checks": 4, + "per_seed": { + "2026073001": { + "delta_bpc": 0.00569589341392529, + "passed": true, + "reference_bpc": 1.7123525940760473, + "variant_bpc": 1.7180484874899726 + }, + "2026073002": { + "delta_bpc": 0.009598287251159299, + "passed": true, + "reference_bpc": 1.7093240655856785, + "variant_bpc": 1.7189223528368378 + }, + "2026073003": { + "delta_bpc": 0.004441193984555669, + "passed": true, + "reference_bpc": 1.7030966812870931, + "variant_bpc": 1.7075378752716488 + } + }, + "per_seed_maximum": 0.05, + "required_checks": 4 + }, + "required_cells": 6, + "threshold": 0.2, + "variant": "uniform_groups_6_7_forward" + } + }, + "final_spectra": [ + { + "global_mean": 7.614457786075945e-05, + "normalized": [ + 0.7854618236676242, + 0.9130540849409994, + 0.8147131127464502, + 0.7522410092502885, + 0.9690072688462938, + 0.8892440039434748, + 0.8242354060534288, + 0.7396083689563772, + 0.6126026690255093, + 0.5670644008562938, + 0.5359283946772616, + 0.5145269561429746, + 0.7290174418180176, + 0.6731434248219464, + 0.6214223500409742, + 0.5883307730946398, + 0.7535191463387267, + 0.6934458909688136, + 0.6978155985835179, + 0.8168473226649812, + 3.219066994145898, + 2.5077095541323673, + 1.973834673592298, + 1.5759251613516239, + 2.2647260999477155, + 1.6880724220748888, + 1.092454232265147, + 0.8732751258075867, + 0.8206258345435784, + 0.5459870832104624, + 0.48994656248557816, + 0.45714680900426247 + ], + "peak_layer_1based": 21, + "peak_normalized": 3.219066994145898, + "reference_mean": 5.7697108584559625e-05, + "seed": 2026073001, + "spike_contrast": 3.04626897714722, + "spike_mean": 0.00017576091195223854, + "top_five_layers_1based": [ + 21, + 22, + 25, + 23, + 26 + ], + "values": [ + 5.980865898891352e-05, + 6.95241178618744e-05, + 6.203598604770377e-05, + 5.727907409891486e-05, + 7.378464943030849e-05, + 6.77111092954874e-05, + 6.276105705183e-05, + 5.631716703646816e-05, + 4.664637162932195e-05, + 4.3178879423066974e-05, + 4.080804137629457e-05, + 3.917843787348829e-05, + 5.551072536036372e-05, + 5.1256221922812983e-05, + 4.731794251711108e-05, + 4.47981983597856e-05, + 5.737639730796218e-05, + 5.2802144637098536e-05, + 5.3134874178795144e-05, + 6.219849456101656e-05, + 0.0002451144973747432, + 0.00019094848539680243, + 0.00015029680798761547, + 0.00011999815615126863, + 0.0001724466128507629, + 0.00012853756197728217, + 8.318446634802967e-05, + 6.64951658109203e-05, + 6.248620775295421e-05, + 4.157395596848801e-05, + 3.7306774174794555e-05, + 3.4809250792022794e-05 + ], + "variant": "learned_reference" + }, + { + "global_mean": 7.03343731061068e-05, + "normalized": [ + 0.6809932372926891, + 0.7475632825133391, + 0.6323153694166137, + 0.6077851857944574, + 0.7056447138179579, + 0.5942446047107566, + 0.530223262177042, + 0.504089839823597, + 0.6368974480554273, + 0.5978888745536934, + 0.5559539610577222, + 0.5232972563751404, + 0.4434509399790644, + 0.41411238498840724, + 0.4174418363858031, + 0.5192489699581264, + 1.2557404086895352, + 1.2676084925356499, + 1.1574347089891701, + 1.060138766241709, + 3.6189242296270336, + 2.81239769967395, + 2.2018329150221976, + 1.7615672973682381, + 1.8179181698124194, + 1.4871187506667403, + 1.086854135909831, + 0.872703317202576, + 0.8717669049561169, + 0.6523365087620864, + 0.4999226915339381, + 0.46458383610897114 + ], + "peak_layer_1based": 21, + "peak_normalized": 3.6189242296270336, + "reference_mean": 5.15456125598306e-05, + "seed": 2026073002, + "spike_contrast": 3.3328477735441013, + "spike_mean": 0.00017179368005599828, + "top_five_layers_1based": [ + 21, + 22, + 23, + 25, + 24 + ], + "values": [ + 4.789723243447952e-05, + 5.257939483271912e-05, + 4.447350511327386e-05, + 4.274819002603181e-05, + 4.96310785820242e-05, + 4.17958217440173e-05, + 3.729292075149715e-05, + 3.545484287315048e-05, + 4.479578274185769e-05, + 4.205213917884976e-05, + 3.91026733268518e-05, + 3.680578447529115e-05, + 3.1189843866741285e-05, + 2.9126334993634373e-05, + 2.9360509870457463e-05, + 3.65210507879965e-05, + 8.83217144291848e-05, + 8.915644866647199e-05, + 8.140744466800243e-05, + 7.45641955290921e-05, + 0.0002545347670093179, + 0.0001978082291316241, + 0.000154864537762478, + 0.00012389873154461384, + 0.00012786213483195752, + 0.00010459556506248191, + 7.644320430699736e-05, + 6.138104072306305e-05, + 6.131517875473946e-05, + 4.588167939800769e-05, + 3.516174911055714e-05, + 3.2676212867954746e-05 + ], + "variant": "learned_reference" + }, + { + "global_mean": 8.757388354752038e-05, + "normalized": [ + 0.7720675832605176, + 0.803585946279625, + 0.7960780094481321, + 0.752473219070419, + 1.180467409539882, + 1.1778079020041263, + 1.029829529518624, + 0.9385078067458692, + 1.0206984705622164, + 0.9298229900567075, + 0.8640759923745102, + 0.8130195958778703, + 0.7736070401530517, + 0.7280910766467854, + 0.7061969571118035, + 0.6826252956632183, + 1.642168884788434, + 1.5117445082936212, + 1.3334176736203447, + 1.1637251407789435, + 2.2882673690329693, + 1.8954517085610634, + 1.5552436162725376, + 1.306771834154562, + 1.2228641636923565, + 0.9319785952743014, + 0.7801153976292595, + 0.661371584878177, + 0.561946672486053, + 0.4345672285647456, + 0.37483694087850516, + 0.36657385678076787 + ], + "peak_layer_1based": 21, + "peak_normalized": 2.2882673690329693, + "reference_mean": 7.697225831819836e-05, + "seed": 2026073003, + "spike_contrast": 1.8814916302334765, + "spike_mean": 0.0001448226597858593, + "top_five_layers_1based": [ + 21, + 22, + 17, + 23, + 18 + ], + "values": [ + 6.761295662727207e-05, + 7.037314207991585e-05, + 6.971564289415255e-05, + 6.589700205950066e-05, + 0.00010337811545468867, + 0.00010314521205145866, + 9.018617129186168e-05, + 8.218877337640151e-05, + 8.93865289981477e-05, + 8.14282102510333e-05, + 7.567049033241346e-05, + 7.11992834112607e-05, + 6.774777284590527e-05, + 6.376176315825433e-05, + 6.184441008372232e-05, + 5.9780148149002343e-05, + 0.00014381110668182373, + 0.00013238933752290905, + 0.0001167725640698336, + 0.00010191192995989695, + 0.0002003924601012841, + 0.0001659920671954751, + 0.0001361987233394757, + 0.00011443908442743123, + 0.00010709096386563033, + 8.16169849713333e-05, + 6.831773498561233e-05, + 5.791887815576047e-05, + 4.921185245621018e-05, + 3.8056739867897704e-05, + 3.282592660980299e-05, + 3.210229624528438e-05 + ], + "variant": "learned_reference" + }, + { + "global_mean": 9.461854074288567e-05, + "normalized": [ + 1.049728483120271, + 1.106104490273285, + 1.0534380329881132, + 0.9962730544079834, + 1.2318533093339465, + 1.1552933983529543, + 1.0861468966704533, + 0.9912509359934122, + 0.9519187879066954, + 0.9095594041307715, + 0.8736030694602711, + 0.8511453745866416, + 0.9964773718623544, + 0.9452695889437478, + 0.8851965662642772, + 0.8230819838948948, + 1.055843626862817, + 0.9503284645066779, + 0.9178198426953132, + 0.83569345377416, + 1.001707960212492, + 0.9726643532652439, + 0.8840872386027205, + 0.8281518558432126, + 2.442749046074982, + 1.8545856115377921, + 1.2448015915768342, + 0.8974028616357965, + 0.7847156721750955, + 0.5164169028929032, + 0.48169339375062104, + 0.42499737640326557 + ], + "peak_layer_1based": 25, + "peak_normalized": 2.442749046074982, + "reference_mean": 9.066082081214215e-05, + "seed": 2026073001, + "spike_contrast": 1.2793864795162566, + "spike_mean": 0.00011599022836890071, + "top_five_layers_1based": [ + 25, + 26, + 27, + 5, + 6 + ], + "values": [ + 9.932377724908292e-05, + 0.0001046579927788116, + 9.967476944439113e-05, + 9.426590258954093e-05, + 0.00011655616253847256, + 0.00010931217548204586, + 0.00010276963439537212, + 9.379071707371622e-05, + 9.0069166617468e-05, + 8.606118353782222e-05, + 8.265904762083665e-05, + 8.053413330344483e-05, + 9.428523480892181e-05, + 8.944002911448479e-05, + 8.375600737053901e-05, + 7.787881622789428e-05, + 9.990238322643563e-05, + 8.991869253804907e-05, + 8.684277418069541e-05, + 7.90720951044932e-05, + 9.478014544583857e-05, + 9.203208173858002e-05, + 8.36510444059968e-05, + 7.835852011339739e-05, + 0.0002311293501406908, + 0.0001754781842464581, + 0.00011778131010942161, + 8.49109492264688e-05, + 7.42486517992802e-05, + 4.886261376668699e-05, + 4.557712600217201e-05, + 4.02126315748319e-05 + ], + "variant": "uniform_group_6_forward" + }, + { + "global_mean": 9.923219420215901e-05, + "normalized": [ + 1.1210080243157192, + 1.1779108698425953, + 1.1110378438066775, + 1.0873712307498702, + 1.2745569576226279, + 1.1932568398188368, + 1.1102274829736989, + 1.0642653152400943, + 1.187072449259881, + 1.1407628792799271, + 1.075379840801642, + 0.9909432042363304, + 0.9657037491393293, + 0.9056250839636294, + 0.859253263137928, + 0.8561867674125543, + 1.1033690382039603, + 1.1437505532630414, + 1.0398233895296574, + 0.8632063018267061, + 1.0230976357898443, + 0.9682149731864415, + 0.9085527067775245, + 0.868443658317733, + 1.8113435808571552, + 1.4139391675400481, + 0.9502866562886768, + 0.700638694754292, + 0.724043106368941, + 0.5551299824318824, + 0.4277691513860246, + 0.3778296018767301 + ], + "peak_layer_1based": 25, + "peak_normalized": 1.8113435808571552, + "reference_mean": 9.7101816613253e-05, + "seed": 2026073002, + "spike_contrast": 1.1404136095960284, + "spike_mean": 0.00011073623318225145, + "top_five_layers_1based": [ + 25, + 26, + 5, + 6, + 9 + ], + "values": [ + 0.00011124008597107604, + 0.00011688668018905446, + 0.00011025072308257222, + 0.00010790223313961178, + 0.00012647708354052156, + 0.00011840949446195737, + 0.00011017030919902027, + 0.000105609382444527, + 0.00011779580381698906, + 0.0001132004035753198, + 0.00010671230120351538, + 9.833346848608926e-05, + 9.582890197634697e-05, + 8.986716420622543e-05, + 8.526558667654172e-05, + 8.496129157720134e-05, + 0.00010948973067570478, + 0.00011349687702022493, + 0.00010318395652575418, + 8.565785537939519e-05, + 0.00010152422328246757, + 9.607809624867514e-05, + 9.015767864184454e-05, + 8.61775697558187e-05, + 0.00017974359798245132, + 0.0001403082860633731, + 9.42990300245583e-05, + 6.95259150234051e-05, + 7.184838614193723e-05, + 5.5086766224121675e-05, + 4.244847150403075e-05, + 3.74928604287561e-05 + ], + "variant": "uniform_group_6_forward" + }, + { + "global_mean": 9.917417844462761e-05, + "normalized": [ + 1.1028432904984338, + 1.1270987866010154, + 1.1163555912049457, + 1.0645032432277715, + 1.4143648500362929, + 1.4417529038578565, + 1.297502061738055, + 1.1918983746366243, + 1.2326024032371155, + 1.1513560434742482, + 1.0785282950209376, + 1.004374246074776, + 0.8587004078072071, + 0.8434604976303082, + 0.847621932333013, + 0.8791225594717167, + 1.3801494080155523, + 1.1693864816711759, + 0.9848436312832127, + 0.8570887157380096, + 0.93528776842758, + 0.8808606602005659, + 0.8340094144302829, + 0.8022181134032798, + 1.7225491622669935, + 1.320199275814519, + 0.9817677850522812, + 0.7360945725690212, + 0.581644906646078, + 0.4241043178935042, + 0.37239268483752097, + 0.3653176149001062 + ], + "peak_layer_1based": 25, + "peak_normalized": 1.7225491622669935, + "reference_mean": 9.853165789132213e-05, + "seed": 2026073003, + "spike_contrast": 1.0417341149957169, + "spike_mean": 0.0001026437894324772, + "top_five_layers_1based": [ + 25, + 6, + 5, + 17, + 26 + ], + "values": [ + 0.00010937357728835195, + 0.00011177909618709236, + 0.00011071364860981703, + 0.00010557123459875584, + 0.00014026847202330828, + 0.0001429846597602591, + 0.0001286787010030821, + 0.00011820554209407419, + 0.00012224233068991452, + 0.00011418478970881552, + 0.00010696215758798644, + 9.960799070540816e-05, + 8.516090747434646e-05, + 8.364950190298259e-05, + 8.40622087707743e-05, + 8.718625758774579e-05, + 0.00013687518367078155, + 0.00011597294360399246, + 9.767105802893639e-05, + 8.500106923747808e-05, + 9.275639604311436e-05, + 8.73586322995834e-05, + 8.271219849120826e-05, + 7.955932233016938e-05, + 0.00017083239799831063, + 0.00013092967856209725, + 9.736601350596175e-05, + 7.3001574492082e-05, + 5.768415576312691e-05, + 4.2060197301907465e-05, + 3.693173857755028e-05, + 3.6230074329068884e-05 + ], + "variant": "uniform_group_6_forward" + }, + { + "global_mean": 9.944817804807826e-05, + "normalized": [ + 1.168387143183479, + 1.2401657617613517, + 1.182335288746566, + 1.1426773374969352, + 1.3063371444652658, + 1.2500084214967857, + 1.1963762785466578, + 1.0978585928746885, + 1.0186348761426658, + 0.9698923101848725, + 0.9394670561774453, + 0.9082441026301723, + 1.0762223006504932, + 1.0045589874449765, + 0.935031603833723, + 0.8850145330621985, + 1.0867928621305778, + 0.9915017514752766, + 0.9834621282857434, + 0.988038493159388, + 1.899516999280113, + 1.4879874428901083, + 1.1118576600996788, + 0.8280383879871337, + 0.6994795556712323, + 0.6618611779401082, + 0.6268707530450582, + 0.6053853235605826, + 0.992054646588788, + 0.6333634112643542, + 0.5808856023956639, + 0.501692065527916 + ], + "peak_layer_1based": 21, + "peak_normalized": 1.899516999280113, + "reference_mean": 9.566590583913408e-05, + "seed": 2026073001, + "spike_contrast": 1.2530320695227308, + "spike_mean": 0.00011987244797637687, + "top_five_layers_1based": [ + 21, + 22, + 5, + 6, + 2 + ], + "values": [ + 0.00011619397264439613, + 0.00012333222548477352, + 0.00011758109030779451, + 0.00011363717931089923, + 0.0001299128489335999, + 0.0001243110600626096, + 0.0001189774411614053, + 0.00010918003681581467, + 0.00010130138252861798, + 9.645402315072715e-05, + 9.342828707303852e-05, + 9.032322122948244e-05, + 0.00010702834697440267, + 9.990156104322523e-05, + 9.298718941863626e-05, + 8.801308285910636e-05, + 0.00010807957005454227, + 9.860304271569476e-05, + 9.78035168373026e-05, + 9.825862798606977e-05, + 0.00018890350474976003, + 0.00014797764015384018, + 0.00011057221854571253, + 8.234690903918818e-05, + 6.956196739338338e-05, + 6.582088826689869e-05, + 6.234115426195785e-05, + 6.020446744514629e-05, + 9.865802712738514e-05, + 6.298683729255572e-05, + 5.776801481260918e-05, + 4.9892361857928336e-05 + ], + "variant": "uniform_group_7_forward" + }, + { + "global_mean": 9.281247480430466e-05, + "normalized": [ + 1.1237833551331617, + 1.158687029916818, + 1.1074600411220088, + 1.0933043261157995, + 1.2290412572656306, + 1.1553665661391181, + 1.0761233798039562, + 1.0372293629967688, + 1.1441068888341974, + 1.1017626658857236, + 1.033201313373911, + 0.9601560490976951, + 0.8799180406514036, + 0.8396732137735404, + 0.8279983609051412, + 0.8670270589090915, + 1.3941783567635277, + 1.3888596254040555, + 1.1960254710619764, + 1.0567738153628634, + 2.0809801808225887, + 1.593322301655694, + 1.2033761794995008, + 0.8879522679216262, + 0.5327896030780552, + 0.5035235279704214, + 0.4742412252680671, + 0.4569905081383433, + 0.8840222894146883, + 0.6881066951417837, + 0.5265942288298747, + 0.4974248137429677 + ], + "peak_layer_1based": 21, + "peak_normalized": 2.0809801808225887, + "reference_mean": 8.834915543384677e-05, + "seed": 2026073002, + "spike_contrast": 1.323322207559972, + "spike_mean": 0.00011691439940477721, + "top_five_layers_1based": [ + 21, + 22, + 17, + 18, + 5 + ], + "values": [ + 0.00010430111433379352, + 0.00010754061077022925, + 0.00010278610716341063, + 0.00010147228022105992, + 0.00011407036072341725, + 0.0001072324303095229, + 9.987767407437786e-05, + 9.626782411942258e-05, + 0.00010618739179335535, + 0.00010225731966784224, + 9.589397086529061e-05, + 8.911445911508054e-05, + 8.166737097781152e-05, + 7.793214899720624e-05, + 7.684857700951397e-05, + 8.047092705965042e-05, + 0.0001293971436098218, + 0.0001289034989895299, + 0.00011100608389824629, + 9.808179311221465e-05, + 0.00019314092060085386, + 0.0001478801859775558, + 0.00011168832133989781, + 8.24130474939011e-05, + 4.944952161167748e-05, + 4.673326475312933e-05, + 4.401550177135505e-05, + 4.241442002239637e-05, + 8.204829646274447e-05, + 6.386488530552015e-05, + 4.887451359536499e-05, + 4.6167227992555127e-05 + ], + "variant": "uniform_group_7_forward" + }, + { + "global_mean": 9.948732213160838e-05, + "normalized": [ + 1.1315008092677528, + 1.1668656649355191, + 1.1366208839234715, + 1.101693372689152, + 1.3429705189947834, + 1.3541548349677262, + 1.2663030159282187, + 1.1984341076366907, + 1.2506532523938356, + 1.178608289839053, + 1.0962174254559829, + 1.031747955952998, + 0.8844731687384572, + 0.8675687093679173, + 0.8798558207776407, + 0.8771834122588129, + 1.4774236545882187, + 1.3583096070854734, + 1.177294574444737, + 1.0471476725613014, + 1.6215780843449914, + 1.3337352379969853, + 1.0512149026578388, + 0.8088985879483863, + 0.5914661841860449, + 0.5588135222163362, + 0.5288816472508608, + 0.5066587724046175, + 0.6923084127530423, + 0.5454130839988593, + 0.47793325135715387, + 0.45807156107714075 + ], + "peak_layer_1based": 21, + "peak_normalized": 1.6215780843449914, + "reference_mean": 9.798803714349763e-05, + "seed": 2026073003, + "spike_contrast": 1.0979244426526977, + "spike_mean": 0.00010758346106740647, + "top_five_layers_1based": [ + 21, + 17, + 18, + 6, + 5 + ], + "values": [ + 0.00011256998550379649, + 0.00011608834029175341, + 0.00011307936802040786, + 0.00010960452345898375, + 0.0001336085406364873, + 0.00013472123828250915, + 0.0001259810960618779, + 0.00011922900011995807, + 0.00012442414299584925, + 0.00011725658259820193, + 0.00010905973613262177, + 0.0001026458412525244, + 8.79938670550473e-05, + 8.631208766018972e-05, + 8.753449947107583e-05, + 8.726862870389596e-05, + 0.00014698492304887623, + 0.0001351345854345709, + 0.00011712588457157835, + 0.00010417791781947017, + 0.00016132646123878658, + 0.00013268974726088345, + 0.00010458255565026775, + 8.047515439102426e-05, + 5.884338679607026e-05, + 5.559486089623533e-05, + 5.261701880954206e-05, + 5.040612450102344e-05, + 6.887591007398441e-05, + 5.42616871825885e-05, + 4.7548299335176125e-05, + 4.5572312956210226e-05 + ], + "variant": "uniform_group_7_forward" + }, + { + "global_mean": 0.00010924439834525401, + "normalized": [ + 1.1802831793473259, + 1.2700138904861653, + 1.1942950278819688, + 1.1430133139888945, + 1.4356181153951746, + 1.3802623223364756, + 1.3279057761796307, + 1.1930909866337074, + 1.1655275133184586, + 1.115130814366578, + 1.0705665024105082, + 1.0317385365452825, + 1.2195044999869864, + 1.163096652743226, + 1.0753236575271325, + 0.9976493434875472, + 1.0906690227048055, + 0.9850593266028592, + 0.947438465704084, + 0.8382655321432377, + 0.9180309004493582, + 0.8783840559942786, + 0.7682382587032985, + 0.6856989599873513, + 0.9056733931341846, + 0.8569565441858251, + 0.814728517156546, + 0.781674128218417, + 0.8819737346732008, + 0.6398484858150421, + 0.5759106186859398, + 0.4684299232065091 + ], + "peak_layer_1based": 5, + "peak_normalized": 1.4356181153951746, + "reference_mean": 0.0001126591938642647, + "seed": 2026073001, + "spike_contrast": 0.8060105831398049, + "spike_mean": 9.080450254259632e-05, + "top_five_layers_1based": [ + 5, + 6, + 7, + 2, + 13 + ], + "values": [ + 0.00012893932580482215, + 0.00013874190335627645, + 0.00013047004176769406, + 0.0001248678017873317, + 0.0001568332372698933, + 0.00015078592696227133, + 0.00014506626757793128, + 0.00013033850700594485, + 0.00012732735194731504, + 0.00012182179489172995, + 0.00011695339344441891, + 0.00011271165567450225, + 0.00013322403538040817, + 0.00012706179404631257, + 0.00011747308599296957, + 0.00010898760228883475, + 0.00011914948117919266, + 0.00010761221346911043, + 0.00010350234515499324, + 9.157581371255219e-05, + 0.00010028973338194191, + 9.595853771315888e-05, + 8.392572635784745e-05, + 7.49087703297846e-05, + 9.893974493024871e-05, + 9.361770207760856e-05, + 8.900452667148784e-05, + 8.53935198392719e-05, + 9.635069000069052e-05, + 6.989986286498606e-05, + 6.29150090389885e-05, + 5.117334512760863e-05 + ], + "variant": "uniform_groups_6_7_forward" + }, + { + "global_mean": 0.00010773822862120142, + "normalized": [ + 1.2021320872392602, + 1.239737264026566, + 1.1872892678543405, + 1.1702441756008668, + 1.3188412036166208, + 1.2665431321842968, + 1.2036994085742312, + 1.1756571340479791, + 1.3750163828497322, + 1.3316928603748481, + 1.281390543858547, + 1.149992314721499, + 1.1427451418889247, + 1.07163124546429, + 1.0057863257095283, + 0.9800428296614334, + 1.185063358202469, + 1.1602375771501046, + 1.0699312204744063, + 0.9180188972304961, + 0.879376133189205, + 0.826676641642478, + 0.7496293723357395, + 0.6894108849482833, + 0.7947654984227666, + 0.7537914046760064, + 0.714759241512971, + 0.6857459679280232, + 0.8575389194858783, + 0.6740471108590389, + 0.4954098247900116, + 0.4431566294791582 + ], + "peak_layer_1based": 9, + "peak_normalized": 1.3750163828497322, + "reference_mean": 0.00011196851617704076, + "seed": 2026073002, + "spike_contrast": 0.75820131156723, + "spike_mean": 8.489467581966892e-05, + "top_five_layers_1based": [ + 9, + 10, + 5, + 11, + 6 + ], + "values": [ + 0.00012951558164786547, + 0.00013356709678191692, + 0.00012791644257958978, + 0.00012608003453351557, + 0.00014208961511030793, + 0.0001364551135338843, + 0.00012968444207217544, + 0.0001266632170882076, + 0.00014814182941336185, + 0.00014347422984428704, + 0.00013805474736727774, + 0.00012389813491608948, + 0.00012311733735259622, + 0.00011545565212145448, + 0.00010836163710337132, + 0.00010558807844063267, + 0.0001276766270166263, + 0.00012500194134190679, + 0.00011527249444043264, + 9.89057298284024e-05, + 9.474242688156664e-05, + 8.906467701308429e-05, + 8.076374069787562e-05, + 7.427590753650293e-05, + 8.562662696931511e-05, + 8.121215068968013e-05, + 7.700689457124099e-05, + 7.388105586869642e-05, + 9.238972415914759e-05, + 7.262064173119143e-05, + 5.337457696441561e-05, + 4.774491026182659e-05 + ], + "variant": "uniform_groups_6_7_forward" + }, + { + "global_mean": 0.00011232519011628028, + "normalized": [ + 1.2171245344139678, + 1.2518726224846988, + 1.2474388463543393, + 1.1988192730216298, + 1.4927538601863934, + 1.5481412124184124, + 1.4360475424686572, + 1.3509742038640624, + 1.4243829697937715, + 1.3501314702932352, + 1.2526613330118617, + 1.1834896837873747, + 1.1372245823864426, + 1.0931400945849206, + 1.0210171936909138, + 1.0031357605655953, + 1.2385446064509475, + 1.1180676483584988, + 0.9704150912522292, + 0.8389663380052635, + 0.8500759113538735, + 0.7841120908683951, + 0.7240270009917109, + 0.6649331756680142, + 0.7086799870587199, + 0.664155606583909, + 0.6264421457123408, + 0.5953397710031234, + 0.6167462401154395, + 0.5144908854286474, + 0.4516352036771426, + 0.4250131141454677 + ], + "peak_layer_1based": 6, + "peak_normalized": 1.5481412124184124, + "reference_mean": 0.00011760102872223663, + "seed": 2026073003, + "spike_contrast": 0.7128820432526025, + "spike_mean": 8.383566164411604e-05, + "top_five_layers_1based": [ + 6, + 5, + 7, + 9, + 8 + ], + "values": [ + 0.00013671374472323805, + 0.00014061683032196015, + 0.00014011880557518452, + 0.0001346576027572155, + 0.00016767386114224792, + 0.00017389525601174682, + 0.000161304313223809, + 0.0001517484342912212, + 0.0001599940878804773, + 0.00015165377408266068, + 0.00014070542238187045, + 0.0001329357037320733, + 0.00012773896742146462, + 0.0001227871689479798, + 0.00011468595039332286, + 0.0001126774150179699, + 0.00013911975838709623, + 0.0001255871611647308, + 0.00010900205961661413, + 9.423705341760069e-05, + 9.548493835609406e-05, + 8.807553967926651e-05, + 8.132647053571418e-05, + 7.46887453715317e-05, + 7.960261427797377e-05, + 7.460140477633104e-05, + 7.036523311398923e-05, + 6.68716529617086e-05, + 6.92761386744678e-05, + 5.7790286518866196e-05, + 5.073001011624001e-05, + 4.773967884830199e-05 + ], + "variant": "uniform_groups_6_7_forward" + }, + { + "global_mean": 9.644760370974836e-05, + "normalized": [ + 1.2063054387246115, + 1.325204706753435, + 1.2220751579218956, + 1.159101443775728, + 1.3059872521838534, + 1.2144440767529154, + 1.1380447745464686, + 1.0346072062950988, + 1.0430331178846848, + 0.9731423573010324, + 0.9349717881195196, + 0.8850937651629682, + 0.9944058061604059, + 0.9359444294077571, + 0.8662547904927963, + 0.8178904608154263, + 0.9633915774647989, + 0.9200487262694695, + 0.9372542851967641, + 1.0738157473594052, + 2.0181972534006216, + 1.5498465965170765, + 1.1628327561745115, + 0.8642229033868173, + 0.6993665796671015, + 0.6514957751047279, + 0.6160923262561516, + 0.6002126162888178, + 1.0695106422329144, + 0.6705551085124273, + 0.629767581294126, + 0.5168829525756723 + ], + "peak_layer_1based": 21, + "peak_normalized": 2.0181972534006216, + "reference_mean": 9.182359806598267e-05, + "seed": 2026073001, + "spike_contrast": 1.3222879166511747, + "spike_mean": 0.00012141723418608308, + "top_five_layers_1based": [ + 21, + 22, + 2, + 5, + 3 + ], + "values": [ + 0.00011634526890702546, + 0.00012781281839124858, + 0.00011786622053477913, + 0.0001117925567086786, + 0.0001259593409486115, + 0.00011713022104231641, + 0.00010976169141940773, + 9.978538582799956e-05, + 0.00010059804480988532, + 9.385724843014032e-05, + 9.017578850034624e-05, + 8.536517270840704e-05, + 9.590805711923167e-05, + 9.026959742186591e-05, + 8.354819874512032e-05, + 7.888357504270971e-05, + 9.291680908063427e-05, + 8.873649494489655e-05, + 9.039592987392098e-05, + 0.00010356695565860718, + 0.00019465028890408576, + 0.00014947899035178125, + 0.00011215243284823373, + 8.33522281027399e-05, + 6.745223072357476e-05, + 6.283520633587614e-05, + 5.942062853137031e-05, + 5.788906855741516e-05, + 0.00010315173858543858, + 6.46734333713539e-05, + 6.0739574109902605e-05, + 4.9852122174343094e-05 + ], + "variant": "uniform_group_7_mlp_forward" + }, + { + "global_mean": 9.294552717165061e-05, + "normalized": [ + 1.1804207293783469, + 1.2244457182848458, + 1.1777345623601965, + 1.161365335790943, + 1.201585743382261, + 1.1521788700409004, + 1.0825081645663768, + 1.0517854717798363, + 1.070778396366818, + 1.043787326113011, + 0.9805167967494025, + 0.8948937161182166, + 0.9628742358933498, + 0.8986009929628155, + 0.8549655350664673, + 0.8542336770652816, + 1.2505796011268457, + 1.2251506472933684, + 1.0881850936912019, + 0.9949638099034118, + 2.1987132665333755, + 1.6868568375183757, + 1.2775504746361555, + 0.9575103563082339, + 0.542818468880225, + 0.49531325994081177, + 0.45802909107422135, + 0.4404704484743748, + 0.8766450963633319, + 0.7024747909383445, + 0.5268035464411853, + 0.48525993895746805 + ], + "peak_layer_1based": 21, + "peak_normalized": 2.1987132665333755, + "reference_mean": 8.721922414325591e-05, + "seed": 2026073002, + "spike_contrast": 1.420186486886559, + "spike_mean": 0.00012386756352498197, + "top_five_layers_1based": [ + 21, + 22, + 23, + 17, + 18 + ], + "values": [ + 0.00010971482697641477, + 0.00011380675277905539, + 0.00010946515976684168, + 0.00010794371337397024, + 0.00011168202036060393, + 0.0001070898724719882, + 0.0001006142920232378, + 9.775875514606014e-05, + 9.952406253432855e-05, + 9.70153632806614e-05, + 9.11346505745314e-05, + 8.317636820720509e-05, + 8.949485345510766e-05, + 8.352094300789759e-05, + 7.946522237034515e-05, + 7.939719944261014e-05, + 0.00011623578029684722, + 0.00011387227277737111, + 0.00010114193719346076, + 9.247743582818657e-05, + 0.00020436056365724653, + 0.0001567857980262488, + 0.00011874260235344991, + 8.899630483938381e-05, + 5.045254874858074e-05, + 4.603715206030756e-05, + 4.257175532984547e-05, + 4.093975803698413e-05, + 8.148024062393233e-05, + 6.529188976855949e-05, + 4.8964033339871094e-05, + 4.510274084168486e-05 + ], + "variant": "uniform_group_7_mlp_forward" + }, + { + "global_mean": 9.711810719181813e-05, + "normalized": [ + 1.1620973642676113, + 1.1894598283879425, + 1.1638141251636995, + 1.129343004808355, + 1.323780054970507, + 1.3108024963433114, + 1.214125895326903, + 1.1368690328501372, + 1.1781773042479384, + 1.0956284129945206, + 1.0240851465456522, + 0.9655277546942812, + 0.9261251890313171, + 0.910704008752465, + 0.9353340398201178, + 0.9580511729624018, + 1.505920482127743, + 1.3295117813120352, + 1.1710289411277004, + 1.030240238186059, + 1.661826944810198, + 1.3665022660755686, + 1.0818738730339919, + 0.8302713446442275, + 0.603514051148169, + 0.5612462965823031, + 0.5267519889043605, + 0.5075502261808718, + 0.701860701281538, + 0.5409652557672129, + 0.49076357856219416, + 0.46624719908866613 + ], + "peak_layer_1based": 21, + "peak_normalized": 1.661826944810198, + "reference_mean": 9.516139861834408e-05, + "seed": 2026073003, + "spike_contrast": 1.1315967929439392, + "spike_mean": 0.00010768433348857797, + "top_five_layers_1based": [ + 21, + 17, + 22, + 18, + 5 + ], + "values": [ + 0.00011286069639027119, + 0.00011551808711374179, + 0.0001130274249590002, + 0.00010967965499730781, + 0.00012856301327701658, + 0.0001273026573471725, + 0.00011791360884672031, + 0.00011041056859539822, + 0.0001144223497249186, + 0.00010640535765560344, + 9.945721103576943e-05, + 9.377022797707468e-05, + 8.994352538138628e-05, + 8.844584954204038e-05, + 9.083787153940648e-05, + 9.304411651100963e-05, + 0.00014625214680563658, + 0.0001291196676902473, + 0.00011372811422916129, + 0.0001000549818854779, + 0.00016139348736032844, + 0.00013271211355458945, + 0.00010506954276934266, + 8.063438144745305e-05, + 5.8612142311176285e-05, + 5.450717799249105e-05, + 5.115715612191707e-05, + 4.9292317271465436e-05, + 6.816338282078505e-05, + 5.253752169664949e-05, + 4.7662029828643426e-05, + 4.528104545897804e-05 + ], + "variant": "uniform_group_7_mlp_forward" + } + ], + "interaction": { + "cells": [ + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073001, + "step": 0 + }, + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073001, + "step": 0 + }, + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073002, + "step": 0 + }, + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073002, + "step": 0 + }, + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073003, + "step": 0 + }, + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073003, + "step": 0 + }, + { + "interaction_residual": 9.315360079922494e-05, + "log_effects": { + "group6": -0.0020673620169685087, + "group7": -0.009910978729338916, + "groups6_7": -0.011885187145508201 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": -0.002069500483230477, + "group7": -0.009960255136960057, + "groups6_7": -0.011956096627949948 + }, + "seed": 2026073001, + "step": 100 + }, + { + "interaction_residual": -0.00015048504886481726, + "log_effects": { + "group6": 0.008204350063220466, + "group7": 0.00633861468044224, + "groups6_7": 0.014392479694797889 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.008170786235759111, + "group7": 0.006318568040747243, + "groups6_7": 0.01428940306090045 + }, + "seed": 2026073001, + "step": 100 + }, + { + "interaction_residual": -0.00022727724861329457, + "log_effects": { + "group6": -0.003024542543237395, + "group7": -0.010274063449398723, + "groups6_7": -0.013525883241249412 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": -0.003029121086871636, + "group7": -0.010327022853315526, + "groups6_7": -0.013617771823999139 + }, + "seed": 2026073002, + "step": 100 + }, + { + "interaction_residual": 0.00011391084988867078, + "log_effects": { + "group6": 0.008212012433931862, + "group7": 0.006225826243211901, + "groups6_7": 0.014551749527032434 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.008178385969761304, + "group7": 0.00620648594426831, + "groups6_7": 0.014446384520769564 + }, + "seed": 2026073002, + "step": 100 + }, + { + "interaction_residual": -0.00018594689551645893, + "log_effects": { + "group6": -0.0024512267787523503, + "group7": -0.010289094447499883, + "groups6_7": -0.012926268121768692 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": -0.002454233491322386, + "group7": -0.0103422091910102, + "groups6_7": -0.013010173463425477 + }, + "seed": 2026073003, + "step": 100 + }, + { + "interaction_residual": 0.00030197360149875237, + "log_effects": { + "group6": 0.007655924297294366, + "group7": 0.00598750540180943, + "groups6_7": 0.013945403300602548 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.007626692355618399, + "group7": 0.005969616013418889, + "groups6_7": 0.01384861659623099 + }, + "seed": 2026073003, + "step": 100 + }, + { + "interaction_residual": 0.029771830971208096, + "log_effects": { + "group6": 0.07485913168415881, + "group7": 0.015857159559656917, + "groups6_7": 0.12048812221502382 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.07212581480354684, + "group7": 0.01573209672471893, + "groups6_7": 0.11351238320785877 + }, + "seed": 2026073001, + "step": 500 + }, + { + "interaction_residual": -0.03192884846129315, + "log_effects": { + "group6": 0.020905879605847052, + "group7": 0.06104413401889223, + "groups6_7": 0.05002116516344613 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.020688866618218654, + "group7": 0.05921828161836901, + "groups6_7": 0.048790708212473444 + }, + "seed": 2026073001, + "step": 500 + }, + { + "interaction_residual": -0.1953811590475397, + "log_effects": { + "group6": 0.26705691975010704, + "group7": 0.1531184204309005, + "groups6_7": 0.22479418113346783 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.23437050921420513, + "group7": 0.14197189226332155, + "groups6_7": 0.20131941462327516 + }, + "seed": 2026073002, + "step": 500 + }, + { + "interaction_residual": 0.04533443185931518, + "log_effects": { + "group6": -0.05291484187072523, + "group7": -0.003412415847688265, + "groups6_7": -0.01099282585909831 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": -0.05433985568652999, + "group7": -0.0034182447669933603, + "groups6_7": -0.011053468978655284 + }, + "seed": 2026073002, + "step": 500 + }, + { + "interaction_residual": -0.05323030205979561, + "log_effects": { + "group6": 0.07126733637611304, + "group7": 0.002081326536511284, + "groups6_7": 0.020118360852828716 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.06878708823689964, + "group7": 0.002079162078343878, + "groups6_7": 0.01991733697848525 + }, + "seed": 2026073003, + "step": 500 + }, + { + "interaction_residual": 0.01258056327461619, + "log_effects": { + "group6": 0.03983966621344984, + "group7": 0.017361821564114133, + "groups6_7": 0.06978205105218016 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.03905650148868994, + "group7": 0.01721197360452348, + "groups6_7": 0.0674029436952744 + }, + "seed": 2026073003, + "step": 500 + }, + { + "interaction_residual": -0.3006404214639773, + "log_effects": { + "group6": 0.6920418427357188, + "group7": 0.6605562686145591, + "groups6_7": 1.0519576898863006 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.49944702553239, + "group7": 0.4834360937330096, + "groups6_7": 0.6507466503415881 + }, + "seed": 2026073001, + "step": 2000 + }, + { + "interaction_residual": -0.08939179391663415, + "log_effects": { + "group6": 0.35196456473519977, + "group7": 0.41393426496383595, + "groups6_7": 0.6765070357824016 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.2966949566647632, + "group7": 0.3389555963844964, + "groups6_7": 0.49161031844044 + }, + "seed": 2026073001, + "step": 2000 + }, + { + "interaction_residual": -0.4113553796722864, + "log_effects": { + "group6": 0.9164467937806046, + "group7": 0.7244113804086706, + "groups6_7": 1.2295027945169887 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.6000624198917696, + "group7": 0.5153902641956247, + "groups6_7": 0.7075620567115803 + }, + "seed": 2026073002, + "step": 2000 + }, + { + "interaction_residual": -0.5102556710901462, + "log_effects": { + "group6": 0.8512692533880561, + "group7": 0.472554150149919, + "groups6_7": 0.8135677324478289 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.5731272217637525, + "group7": 0.3765920444307726, + "groups6_7": 0.5567262404629003 + }, + "seed": 2026073002, + "step": 2000 + }, + { + "interaction_residual": -0.042537792021102805, + "log_effects": { + "group6": 0.3766095689864746, + "group7": 0.33907117265811954, + "groups6_7": 0.6731429496234913 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.3138160709099704, + "group7": 0.2875682583760594, + "groups6_7": 0.48989717177853653 + }, + "seed": 2026073003, + "step": 2000 + }, + { + "interaction_residual": -0.23802610783761186, + "log_effects": { + "group6": 0.1369711438469228, + "group7": 0.21929899602618774, + "groups6_7": 0.11824403203549867 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.1280046117985281, + "group7": 0.19691843594350228, + "groups6_7": 0.11152079123254297 + }, + "seed": 2026073003, + "step": 2000 + }, + { + "interaction_residual": -0.3567901087347324, + "log_effects": { + "group6": 0.7962036667755509, + "group7": 0.7974826538576361, + "groups6_7": 1.2368962118984546 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.5489619914090167, + "group7": 0.5495384944469774, + "groups6_7": 0.709716199433785 + }, + "seed": 2026073001, + "step": 4000 + }, + { + "interaction_residual": -0.025278590898171305, + "log_effects": { + "group6": 0.30819818474854643, + "group7": 0.5116588900282694, + "groups6_7": 0.7945784838786445 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.26523031656668117, + "group7": 0.40049975151541234, + "groups6_7": 0.5482283761836112 + }, + "seed": 2026073001, + "step": 4000 + }, + { + "interaction_residual": -0.4690849356804383, + "log_effects": { + "group6": 1.0154628434911215, + "group7": 0.8444211614256345, + "groups6_7": 1.3907990692363177 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.6377652669616457, + "group7": 0.570193925444063, + "groups6_7": 0.7511236442842244 + }, + "seed": 2026073002, + "step": 4000 + }, + { + "interaction_residual": -0.30335764563544154, + "log_effects": { + "group6": 0.7295561615678221, + "group7": 0.5114340535088001, + "groups6_7": 0.9376325694411807 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.517877072705919, + "group7": 0.4003649468121799, + "groups6_7": 0.6084462848212776 + }, + "seed": 2026073002, + "step": 4000 + }, + { + "interaction_residual": -0.11945708203112537, + "log_effects": { + "group6": 0.5220048555645129, + "group7": 0.46526410081521685, + "groups6_7": 0.8678118743486044 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.4066701858351324, + "group7": 0.3720307638996611, + "groups6_7": 0.5801307284410583 + }, + "seed": 2026073003, + "step": 4000 + }, + { + "interaction_residual": -0.2049100207501896, + "log_effects": { + "group6": 0.23807868703777696, + "group7": 0.278434203129335, + "groups6_7": 0.31160286941692233 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.21185932780108593, + "group7": 0.24303192775825497, + "groups6_7": 0.2677277217915467 + }, + "seed": 2026073003, + "step": 4000 + }, + { + "interaction_residual": -0.4263122295724949, + "log_effects": { + "group6": 0.8675369054663282, + "group7": 0.8883512858342638, + "groups6_7": 1.329575961728097 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.5800152615825866, + "group7": 0.5886666348497648, + "groups6_7": 0.7354105664383516 + }, + "seed": 2026073001, + "step": 8000 + }, + { + "interaction_residual": 0.004036643176398624, + "log_effects": { + "group6": 0.275967500869369, + "group7": 0.5274919212487371, + "groups6_7": 0.8074960652945047 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.24116240807746625, + "group7": 0.40991690985788126, + "groups6_7": 0.5540266425004673 + }, + "seed": 2026073001, + "step": 8000 + }, + { + "interaction_residual": -0.5154843690703684, + "log_effects": { + "group6": 1.07243611373865, + "group7": 0.9236817271683265, + "groups6_7": 1.480633471836608 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.6578260733512802, + "group7": 0.6029454996221533, + "groups6_7": 0.7725064680163986 + }, + "seed": 2026073002, + "step": 8000 + }, + { + "interaction_residual": -0.2777345501217605, + "log_effects": { + "group6": 0.6921079277686524, + "group7": 0.5533377840422664, + "groups6_7": 0.9677111616891583 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.4994801034992014, + "group7": 0.42497271322062063, + "groups6_7": 0.620048308391513 + }, + "seed": 2026073002, + "step": 8000 + }, + { + "interaction_residual": -0.15931730317353043, + "log_effects": { + "group6": 0.5911781397673622, + "group7": 0.5386433554809235, + "groups6_7": 0.9705041920747552 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.4463254057279825, + "group7": 0.4164606288913148, + "groups6_7": 0.6211080443849011 + }, + "seed": 2026073003, + "step": 8000 + }, + { + "interaction_residual": -0.2376448513270391, + "log_effects": { + "group6": 0.2839896590577151, + "group7": 0.34439512255491495, + "groups6_7": 0.39073993028559095 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.2472255709371281, + "group7": 0.29135113042743926, + "groups6_7": 0.3234439150908039 + }, + "seed": 2026073003, + "step": 8000 + } + ], + "definition": "I67=ln(Xref/X67)-ln(Xref/X6)-ln(Xref/X7)", + "interpretation": "descriptive cross-run log-attenuation residual from three independently trained variants; not a causal interaction", + "summaries": [ + { + "maximum": 0.0, + "mean_interaction_residual": 0.0, + "metric": "spike_contrast", + "minimum": 0.0, + "step": 0 + }, + { + "maximum": 0.0, + "mean_interaction_residual": 0.0, + "metric": "peak_normalized", + "minimum": 0.0, + "step": 0 + }, + { + "maximum": 9.315360079922494e-05, + "mean_interaction_residual": -0.00010669018111017618, + "metric": "spike_contrast", + "minimum": -0.00022727724861329457, + "step": 100 + }, + { + "maximum": 0.00030197360149875237, + "mean_interaction_residual": 8.846646750753529e-05, + "metric": "peak_normalized", + "minimum": -0.00015048504886481726, + "step": 100 + }, + { + "maximum": 0.029771830971208096, + "mean_interaction_residual": -0.07294654337870907, + "metric": "spike_contrast", + "minimum": -0.1953811590475397, + "step": 500 + }, + { + "maximum": 0.04533443185931518, + "mean_interaction_residual": 0.008662048890879407, + "metric": "peak_normalized", + "minimum": -0.03192884846129315, + "step": 500 + }, + { + "maximum": -0.042537792021102805, + "mean_interaction_residual": -0.25151119771912217, + "metric": "spike_contrast", + "minimum": -0.4113553796722864, + "step": 2000 + }, + { + "maximum": -0.08939179391663415, + "mean_interaction_residual": -0.2792245242814641, + "metric": "peak_normalized", + "minimum": -0.5102556710901462, + "step": 2000 + }, + { + "maximum": -0.11945708203112537, + "mean_interaction_residual": -0.315110708815432, + "metric": "spike_contrast", + "minimum": -0.4690849356804383, + "step": 4000 + }, + { + "maximum": -0.025278590898171305, + "mean_interaction_residual": -0.17784875242793416, + "metric": "peak_normalized", + "minimum": -0.30335764563544154, + "step": 4000 + }, + { + "maximum": -0.15931730317353043, + "mean_interaction_residual": -0.3670379672721313, + "metric": "spike_contrast", + "minimum": -0.5154843690703684, + "step": 8000 + }, + { + "maximum": 0.004036643176398624, + "mean_interaction_residual": -0.17044758609080035, + "metric": "peak_normalized", + "minimum": -0.2777345501217605, + "step": 8000 + } + ] + }, + "metadata_warnings": [], + "primary": { + "attenuation_passed": true, + "cells": [ + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.04626897714722, + "relative_drop": 0.7354105664383516, + "seed": 2026073001, + "variant": 0.8060105831398049 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.219066994145898, + "relative_drop": 0.5540266425004673, + "seed": 2026073001, + "variant": 1.4356181153951746 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.3328477735441013, + "relative_drop": 0.7725064680163986, + "seed": 2026073002, + "variant": 0.75820131156723 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.6189242296270336, + "relative_drop": 0.620048308391513, + "seed": 2026073002, + "variant": 1.3750163828497322 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 1.8814916302334765, + "relative_drop": 0.6211080443849011, + "seed": 2026073003, + "variant": 0.7128820432526025 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 2.2882673690329693, + "relative_drop": 0.3234439150908039, + "seed": 2026073003, + "variant": 1.5481412124184124 + } + ], + "material_response_passed": true, + "passed_cells": 6, + "quality": { + "mean_delta_bpc": 0.0065784582165467525, + "mean_maximum": 0.03, + "mean_passed": true, + "passed": true, + "passed_checks": 4, + "per_seed": { + "2026073001": { + "delta_bpc": 0.00569589341392529, + "passed": true, + "reference_bpc": 1.7123525940760473, + "variant_bpc": 1.7180484874899726 + }, + "2026073002": { + "delta_bpc": 0.009598287251159299, + "passed": true, + "reference_bpc": 1.7093240655856785, + "variant_bpc": 1.7189223528368378 + }, + "2026073003": { + "delta_bpc": 0.004441193984555669, + "passed": true, + "reference_bpc": 1.7030966812870931, + "variant_bpc": 1.7075378752716488 + } + }, + "per_seed_maximum": 0.05, + "required_checks": 4 + }, + "required_cells": 6, + "threshold": 0.2, + "variant": "uniform_groups_6_7_forward" + }, + "primary_step": 8000, + "processed_target_bytes": { + "formal_12_cells": 786432000, + "per_cell": 65536000, + "primary_replay": 65536000, + "total": 851968000 + }, + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "replay": { + "excluded": [ + "run_kind", + "timing", + "self hashes", + "manifest path strings", + "GPU/version metadata" + ], + "passed": true, + "scientific_payload_sha256": "b85563ca5cb53e60b39c3801d372376206105b8a089a8633b3e81973a7f0c051" + }, + "reporting_boundary": "C can change through spike-window numerator and the 27-layer reference denominator; layers 26-28 are intervened but belong to the denominator.", + "schema_version": 1, + "secondary_status": { + "uniform_group_6_forward": "secondary_material_response", + "uniform_group_7_forward": "secondary_material_response", + "uniform_group_7_mlp_forward": "secondary_material_response" + }, + "spike_layers_1based": [ + 21, + 22, + 23, + 24, + 25 + ], + "status": "forward_training_attenuation_established_within_reduced_protocol", + "thresholds": { + "final_bpc_delta_mean_maximum": 0.03, + "final_bpc_delta_per_seed_maximum": 0.05, + "loss_scale_ratio_abs_error": 1e-05, + "loss_scale_shape_abs_error": 1e-06, + "material_relative_drop": 0.2, + "positive_denominator_epsilon": 1e-30, + "uniform_weight_max_abs_error": 1e-12 + }, + "trajectories": [ + { + "peak_normalized": 1.7717451736709768, + "reference_mean": 0.00015645541295978345, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.7816308178118705, + "spike_mean": 0.00012229037238284946, + "step": 0, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.145023601546889, + "reference_mean": 3.5063370281325966e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.5596113913680666, + "spike_mean": 1.962186142918654e-05, + "step": 100, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.1343785185339303, + "reference_mean": 7.428176730981149e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.5221542367874475, + "spike_mean": 3.8786539516877386e-05, + "step": 500, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.863607509132394, + "reference_mean": 4.200080675892187e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 2.4966108002124767, + "spike_mean": 0.00010485966777196154, + "step": 2000, + "variant": "learned_reference" + }, + { + "peak_normalized": 3.115049509935762, + "reference_mean": 3.6590978736596835e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 2.8510104941924466, + "spike_mean": 0.00010432126437081024, + "step": 4000, + "variant": "learned_reference" + }, + { + "peak_normalized": 3.219066994145898, + "reference_mean": 5.7697108584559625e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 3.04626897714722, + "spike_mean": 0.00017576091195223854, + "step": 8000, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.16920774571012, + "reference_mean": 0.0001952685916977417, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.6908033638721538, + "spike_mean": 0.0001348922000033781, + "step": 0, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.254829765809383, + "reference_mean": 3.313880080424025e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.5561530270868602, + "spike_mean": 1.8430244381306693e-05, + "step": 100, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.173061151786652, + "reference_mean": 7.33956448702547e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.5734017265344913, + "spike_mean": 4.2085189488716426e-05, + "step": 500, + "variant": "learned_reference" + }, + { + "peak_normalized": 3.189751492495938, + "reference_mean": 3.9160933603718013e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 2.740373880872814, + "spike_mean": 0.00010731559959822334, + "step": 2000, + "variant": "learned_reference" + }, + { + "peak_normalized": 3.420348943656972, + "reference_mean": 3.154654101914881e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 3.1572107254761765, + "spike_mean": 9.959907765733078e-05, + "step": 4000, + "variant": "learned_reference" + }, + { + "peak_normalized": 3.6189242296270336, + "reference_mean": 5.15456125598306e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 3.3328477735441013, + "spike_mean": 0.00017179368005599828, + "step": 8000, + "variant": "learned_reference" + }, + { + "peak_normalized": 1.7301666786506156, + "reference_mean": 0.0001660444721454513, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.7974048446155259, + "spike_mean": 0.0001324046665104106, + "step": 0, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.133789250541258, + "reference_mean": 3.45711278323843e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.5746026267549044, + "spike_mean": 1.98646608623676e-05, + "step": 100, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.195594858531836, + "reference_mean": 7.680312336122841e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.47779529155407807, + "spike_mean": 3.669617071864195e-05, + "step": 500, + "variant": "learned_reference" + }, + { + "peak_normalized": 1.8057756999530474, + "reference_mean": 5.286508732953937e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 1.4606649819879878, + "spike_mean": 7.721818183199502e-05, + "step": 2000, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.1130451955668876, + "reference_mean": 4.555695098436748e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 1.7734534826874866, + "spike_mean": 8.079313338384963e-05, + "step": 4000, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.2882673690329693, + "reference_mean": 7.697225831819836e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 1.8814916302334765, + "spike_mean": 0.0001448226597858593, + "step": 8000, + "variant": "learned_reference" + }, + { + "peak_normalized": 1.7717451736709768, + "reference_mean": 0.00015645541295978345, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.7816308178118705, + "spike_mean": 0.00012229037238284946, + "step": 0, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.127497072227991, + "reference_mean": 3.516218108685549e-05, + "relative_drop": { + "peak_normalized": 0.008170786235759111, + "spike_contrast": -0.002069500483230477 + }, + "seed": 2026073001, + "spike_contrast": 0.5607695074129241, + "spike_mean": 1.971787896763999e-05, + "step": 100, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.0902206460511907, + "reference_mean": 7.938710263665093e-05, + "relative_drop": { + "peak_normalized": 0.020688866618218654, + "spike_contrast": 0.07212581480354684 + }, + "seed": 2026073001, + "spike_contrast": 0.48449343700602876, + "spike_mean": 3.8462530210381376e-05, + "step": 500, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.013989603305468, + "reference_mean": 5.593089116262962e-05, + "relative_drop": { + "peak_normalized": 0.2966949566647632, + "spike_contrast": 0.49944702553239 + }, + "seed": 2026073001, + "spike_contrast": 1.2496859621343153, + "spike_mean": 6.989604953560047e-05, + "step": 2000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.2888439422946147, + "reference_mean": 5.321969008407366e-05, + "relative_drop": { + "peak_normalized": 0.26523031656668117, + "spike_contrast": 0.5489619914090167 + }, + "seed": 2026073001, + "spike_contrast": 1.2859140957725563, + "spike_mean": 6.843594965175726e-05, + "step": 4000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.442749046074982, + "reference_mean": 9.066082081214215e-05, + "relative_drop": { + "peak_normalized": 0.24116240807746625, + "spike_contrast": 0.5800152615825866 + }, + "seed": 2026073001, + "spike_contrast": 1.2793864795162566, + "spike_mean": 0.00011599022836890071, + "step": 8000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.16920774571012, + "reference_mean": 0.0001952685916977417, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.6908033638721538, + "spike_mean": 0.0001348922000033781, + "step": 0, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.2363888976884874, + "reference_mean": 3.3230999987507756e-05, + "relative_drop": { + "peak_normalized": 0.008178385969761304, + "spike_contrast": -0.003029121086871636 + }, + "seed": 2026073002, + "spike_contrast": 0.5578376819487365, + "spike_mean": 1.853750400186982e-05, + "step": 100, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.291144981172743, + "reference_mean": 7.969680357038871e-05, + "relative_drop": { + "peak_normalized": -0.05433985568652999, + "spike_contrast": 0.23437050921420513 + }, + "seed": 2026073002, + "spike_contrast": 0.43901327190229816, + "spike_mean": 3.498795449559111e-05, + "step": 500, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.3616180814849579, + "reference_mean": 6.0796766370509024e-05, + "relative_drop": { + "peak_normalized": 0.5731272217637525, + "spike_contrast": 0.6000624198917696 + }, + "seed": 2026073002, + "spike_contrast": 1.0959784985080736, + "spike_mean": 6.663194872089662e-05, + "step": 2000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.649028645083117, + "reference_mean": 5.410763259095364e-05, + "relative_drop": { + "peak_normalized": 0.517877072705919, + "spike_contrast": 0.6377652669616457 + }, + "seed": 2026073002, + "spike_contrast": 1.1436513842886917, + "spike_mean": 6.188026891322807e-05, + "step": 4000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.8113435808571552, + "reference_mean": 9.7101816613253e-05, + "relative_drop": { + "peak_normalized": 0.4994801034992014, + "spike_contrast": 0.6578260733512802 + }, + "seed": 2026073002, + "spike_contrast": 1.1404136095960284, + "spike_mean": 0.00011073623318225145, + "step": 8000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.7301666786506156, + "reference_mean": 0.0001660444721454513, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.7974048446155259, + "spike_mean": 0.0001324046665104106, + "step": 0, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.1175154963756544, + "reference_mean": 3.463190794853425e-05, + "relative_drop": { + "peak_normalized": 0.007626692355618399, + "spike_contrast": -0.002454233491322386 + }, + "seed": 2026073003, + "spike_contrast": 0.5760128357656881, + "spike_mean": 1.9948423505411485e-05, + "step": 100, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.1098426046710275, + "reference_mean": 8.283220328545819e-05, + "relative_drop": { + "peak_normalized": 0.03905650148868994, + "spike_contrast": 0.06878708823689964 + }, + "seed": 2026073003, + "spike_contrast": 0.4449291446747725, + "spike_mean": 3.6854461359325794e-05, + "step": 500, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.5746280824853423, + "reference_mean": 6.005889426214034e-05, + "relative_drop": { + "peak_normalized": 0.1280046117985281, + "spike_contrast": 0.3138160709099704 + }, + "seed": 2026073003, + "spike_contrast": 1.0022848364247348, + "spike_mean": 6.0196119011379776e-05, + "step": 2000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.6653768608207726, + "reference_mean": 5.561273489244437e-05, + "relative_drop": { + "peak_normalized": 0.21185932780108593, + "spike_contrast": 0.4066701858351324 + }, + "seed": 2026073003, + "spike_contrast": 1.0522428253130036, + "spike_mean": 5.851810128660872e-05, + "step": 4000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.7225491622669935, + "reference_mean": 9.853165789132213e-05, + "relative_drop": { + "peak_normalized": 0.2472255709371281, + "spike_contrast": 0.4463254057279825 + }, + "seed": 2026073003, + "spike_contrast": 1.0417341149957169, + "spike_mean": 0.0001026437894324772, + "step": 8000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.7717451736709768, + "reference_mean": 0.00015645541295978345, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.7816308178118705, + "spike_mean": 0.00012229037238284946, + "step": 0, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.131470123971506, + "reference_mean": 3.5108908681043736e-05, + "relative_drop": { + "peak_normalized": 0.006318568040747243, + "spike_contrast": -0.009960255136960057 + }, + "seed": 2026073001, + "spike_contrast": 0.5651852636036417, + "spike_mean": 1.984303780773189e-05, + "step": 100, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.007984290343191, + "reference_mean": 7.690677624918021e-05, + "relative_drop": { + "peak_normalized": 0.05921828161836901, + "spike_contrast": 0.01573209672471893 + }, + "seed": 2026073001, + "spike_contrast": 0.5139396558290856, + "spike_mean": 3.952544211642817e-05, + "step": 500, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.8929717180633012, + "reference_mean": 6.043750208509327e-05, + "relative_drop": { + "peak_normalized": 0.3389555963844964, + "spike_contrast": 0.4834360937330096 + }, + "seed": 2026073001, + "spike_contrast": 1.2896590273861137, + "spike_mean": 7.794377015670761e-05, + "step": 2000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.8674729552482823, + "reference_mean": 5.6604490695193546e-05, + "relative_drop": { + "peak_normalized": 0.40049975151541234, + "spike_contrast": 0.5495384944469774 + }, + "seed": 2026073001, + "spike_contrast": 1.2842704795613964, + "spike_mean": 7.269547641044482e-05, + "step": 4000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.899516999280113, + "reference_mean": 9.566590583913408e-05, + "relative_drop": { + "peak_normalized": 0.40991690985788126, + "spike_contrast": 0.5886666348497648 + }, + "seed": 2026073001, + "spike_contrast": 1.2530320695227308, + "spike_mean": 0.00011987244797637687, + "step": 8000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.16920774571012, + "reference_mean": 0.0001952685916977417, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.6908033638721538, + "spike_mean": 0.0001348922000033781, + "step": 0, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.2408351965611693, + "reference_mean": 3.318128242595035e-05, + "relative_drop": { + "peak_normalized": 0.00620648594426831, + "spike_contrast": -0.010327022853315526 + }, + "seed": 2026073002, + "spike_contrast": 0.5618964321075268, + "spike_mean": 1.8644444207893684e-05, + "step": 100, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.180489206697103, + "reference_mean": 7.674864826657102e-05, + "relative_drop": { + "peak_normalized": -0.0034182447669933603, + "spike_contrast": 0.14197189226332155 + }, + "seed": 2026073002, + "spike_contrast": 0.4919947983913339, + "spike_mean": 3.7759935730719006e-05, + "step": 500, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.9885164567107845, + "reference_mean": 5.209253834403137e-05, + "relative_drop": { + "peak_normalized": 0.3765920444307726, + "spike_contrast": 0.5153902641956247 + }, + "seed": 2026073002, + "spike_contrast": 1.328011862414985, + "spike_mean": 6.917950886418112e-05, + "step": 2000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.0509611207506526, + "reference_mean": 5.1951786628234445e-05, + "relative_drop": { + "peak_normalized": 0.4003649468121799, + "spike_contrast": 0.570193925444063 + }, + "seed": 2026073002, + "spike_contrast": 1.3569883484628174, + "spike_mean": 7.049796913634054e-05, + "step": 4000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.0809801808225887, + "reference_mean": 8.834915543384677e-05, + "relative_drop": { + "peak_normalized": 0.42497271322062063, + "spike_contrast": 0.6029454996221533 + }, + "seed": 2026073002, + "spike_contrast": 1.323322207559972, + "spike_mean": 0.00011691439940477721, + "step": 8000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.7301666786506156, + "reference_mean": 0.0001660444721454513, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.7974048446155259, + "spike_mean": 0.0001324046665104106, + "step": 0, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.121051348061966, + "reference_mean": 3.459093777227854e-05, + "relative_drop": { + "peak_normalized": 0.005969616013418889, + "spike_contrast": -0.0103422091910102 + }, + "seed": 2026073003, + "spike_contrast": 0.5805452873225075, + "spike_mean": 2.0081605907762422e-05, + "step": 100, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.1578043377805587, + "reference_mean": 7.838830411115109e-05, + "relative_drop": { + "peak_normalized": 0.01721197360452348, + "spike_contrast": 0.002079162078343878 + }, + "seed": 2026073003, + "spike_contrast": 0.4768018777026676, + "spike_mean": 3.7375690590124576e-05, + "step": 500, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.4501851734535103, + "reference_mean": 6.012620542605873e-05, + "relative_drop": { + "peak_normalized": 0.19691843594350228, + "spike_contrast": 0.2875682583760594 + }, + "seed": 2026073003, + "spike_contrast": 1.040624097046804, + "spike_mean": 6.256877823034302e-05, + "step": 2000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.599507748247948, + "reference_mean": 5.5299108242723846e-05, + "relative_drop": { + "peak_normalized": 0.24303192775825497, + "spike_contrast": 0.3720307638996611 + }, + "seed": 2026073003, + "spike_contrast": 1.1136742287827466, + "spike_mean": 6.15851917245891e-05, + "step": 4000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.6215780843449914, + "reference_mean": 9.798803714349763e-05, + "relative_drop": { + "peak_normalized": 0.29135113042743926, + "spike_contrast": 0.4164606288913148 + }, + "seed": 2026073003, + "spike_contrast": 1.0979244426526977, + "spike_mean": 0.00010758346106740647, + "step": 8000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.7717451736709768, + "reference_mean": 0.00015645541295978345, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.7816308178118705, + "spike_mean": 0.00012229037238284946, + "step": 0, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.114372494729241, + "reference_mean": 3.5219062098163975e-05, + "relative_drop": { + "peak_normalized": 0.01428940306090045, + "spike_contrast": -0.011956096627949948 + }, + "seed": 2026073001, + "spike_contrast": 0.5663021592373647, + "spike_mean": 1.994463091250509e-05, + "step": 100, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.03024067902117, + "reference_mean": 7.452956594567729e-05, + "relative_drop": { + "peak_normalized": 0.048790708212473444, + "spike_contrast": 0.11351238320785877 + }, + "seed": 2026073001, + "spike_contrast": 0.4628832649676238, + "spike_mean": 3.449848882155493e-05, + "step": 500, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.4558285096793826, + "reference_mean": 6.297526043555181e-05, + "relative_drop": { + "peak_normalized": 0.49161031844044, + "spike_contrast": 0.6507466503415881 + }, + "seed": 2026073001, + "spike_contrast": 0.8719496847675758, + "spike_mean": 5.491125848493539e-05, + "step": 2000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.4072909753721254, + "reference_mean": 6.464099647861008e-05, + "relative_drop": { + "peak_normalized": 0.5482283761836112, + "spike_contrast": 0.709716199433785 + }, + "seed": 2026073001, + "spike_contrast": 0.8276021617083458, + "spike_mean": 5.349702842067927e-05, + "step": 4000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.4356181153951746, + "reference_mean": 0.0001126591938642647, + "relative_drop": { + "peak_normalized": 0.5540266425004673, + "spike_contrast": 0.7354105664383516 + }, + "seed": 2026073001, + "spike_contrast": 0.8060105831398049, + "spike_mean": 9.080450254259632e-05, + "step": 8000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.16920774571012, + "reference_mean": 0.0001952685916977417, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.6908033638721538, + "spike_mean": 0.0001348922000033781, + "step": 0, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.222255627983624, + "reference_mean": 3.3272086107495445e-05, + "relative_drop": { + "peak_normalized": 0.014446384520769564, + "spike_contrast": -0.013617771823999139 + }, + "seed": 2026073002, + "spike_contrast": 0.5637265921089555, + "spike_mean": 1.875635971373413e-05, + "step": 100, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.1970810158166465, + "reference_mean": 7.526881664590392e-05, + "relative_drop": { + "peak_normalized": -0.011053468978655284, + "spike_contrast": 0.20131941462327516 + }, + "seed": 2026073002, + "spike_contrast": 0.4579648266045922, + "spike_mean": 3.4470470563974234e-05, + "step": 500, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.4139331360677492, + "reference_mean": 6.686618683118097e-05, + "relative_drop": { + "peak_normalized": 0.5567262404629003, + "spike_contrast": 0.7075620567115803 + }, + "seed": 2026073002, + "spike_contrast": 0.8013893015637505, + "spike_mean": 5.358584676287137e-05, + "step": 2000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.339250336096506, + "reference_mean": 6.513453505855675e-05, + "relative_drop": { + "peak_normalized": 0.6084462848212776, + "spike_contrast": 0.7511236442842244 + }, + "seed": 2026073002, + "spike_contrast": 0.785755099583271, + "spike_mean": 5.117979308124632e-05, + "step": 4000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.3750163828497322, + "reference_mean": 0.00011196851617704076, + "relative_drop": { + "peak_normalized": 0.620048308391513, + "spike_contrast": 0.7725064680163986 + }, + "seed": 2026073002, + "spike_contrast": 0.75820131156723, + "spike_mean": 8.489467581966892e-05, + "step": 8000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.7301666786506156, + "reference_mean": 0.0001660444721454513, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.7974048446155259, + "spike_mean": 0.0001324046665104106, + "step": 0, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.104239221313353, + "reference_mean": 3.465675830806571e-05, + "relative_drop": { + "peak_normalized": 0.01384861659623099, + "spike_contrast": -0.013010173463425477 + }, + "seed": 2026073003, + "spike_contrast": 0.5820783066015256, + "spike_mean": 2.017294718825724e-05, + "step": 100, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.047605301904581, + "reference_mean": 8.027602099766093e-05, + "relative_drop": { + "peak_normalized": 0.0674029436952744, + "spike_contrast": 0.01991733697848525 + }, + "seed": 2026073003, + "spike_contrast": 0.4682788817254619, + "spike_mean": 3.7591565342154357e-05, + "step": 500, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.6043941651057845, + "reference_mean": 6.416649369014151e-05, + "relative_drop": { + "peak_normalized": 0.11152079123254297, + "spike_contrast": 0.48989717177853653 + }, + "seed": 2026073003, + "spike_contrast": 0.7450893383961256, + "spike_mean": 4.7809770330786704e-05, + "step": 2000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.5473244193151916, + "reference_mean": 6.484813436852871e-05, + "relative_drop": { + "peak_normalized": 0.2677277217915467, + "spike_contrast": 0.5801307284410583 + }, + "seed": 2026073003, + "spike_contrast": 0.7446186219196633, + "spike_mean": 4.828712844755501e-05, + "step": 4000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.5481412124184124, + "reference_mean": 0.00011760102872223663, + "relative_drop": { + "peak_normalized": 0.3234439150908039, + "spike_contrast": 0.6211080443849011 + }, + "seed": 2026073003, + "spike_contrast": 0.7128820432526025, + "spike_mean": 8.383566164411604e-05, + "step": 8000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.7717451736709768, + "reference_mean": 0.00015645541295978345, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.7816308178118705, + "spike_mean": 0.00012229037238284946, + "step": 0, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.1313742213368374, + "reference_mean": 3.5109654803549075e-05, + "relative_drop": { + "peak_normalized": 0.0063632774018002325, + "spike_contrast": -0.009908666739421946 + }, + "seed": 2026073001, + "spike_contrast": 0.565156394148717, + "spike_mean": 1.9842445908579975e-05, + "step": 100, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.074096058122276, + "reference_mean": 7.514289326435473e-05, + "relative_drop": { + "peak_normalized": 0.02824356593181118, + "spike_contrast": 0.002103655349041084 + }, + "seed": 2026073001, + "spike_contrast": 0.5210558042342052, + "spike_mean": 3.9153640682343396e-05, + "step": 500, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.008182813956302, + "reference_mean": 5.4975213128340397e-05, + "relative_drop": { + "peak_normalized": 0.2987227448063459, + "spike_contrast": 0.44974147877714904 + }, + "seed": 2026073001, + "spike_contrast": 1.373781366993916, + "spike_mean": 7.552392344223335e-05, + "step": 2000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.006420683947552, + "reference_mean": 5.4418700596283156e-05, + "relative_drop": { + "peak_normalized": 0.35589444805038495, + "spike_contrast": 0.5194674071926733 + }, + "seed": 2026073001, + "spike_contrast": 1.3700034648951942, + "spike_mean": 7.45538083720021e-05, + "step": 4000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.0181972534006216, + "reference_mean": 9.182359806598267e-05, + "relative_drop": { + "peak_normalized": 0.37304900548175707, + "spike_contrast": 0.5659319887472724 + }, + "seed": 2026073001, + "spike_contrast": 1.3222879166511747, + "spike_mean": 0.00012141723418608308, + "step": 8000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.16920774571012, + "reference_mean": 0.0001952685916977417, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.6908033638721538, + "spike_mean": 0.0001348922000033781, + "step": 0, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.24083237123207, + "reference_mean": 3.3190639981141967e-05, + "relative_drop": { + "peak_normalized": 0.006207738956421139, + "spike_contrast": -0.010320676434344202 + }, + "seed": 2026073002, + "spike_contrast": 0.5618929025274048, + "spike_mean": 1.8649585035745987e-05, + "step": 100, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.2055180055903696, + "reference_mean": 7.56468436722773e-05, + "relative_drop": { + "peak_normalized": -0.014936005724934306, + "spike_contrast": 0.15494331212416607 + }, + "seed": 2026073002, + "spike_contrast": 0.4845569638475219, + "spike_mean": 3.6655204894486815e-05, + "step": 500, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.1937881935672334, + "reference_mean": 5.264968812870966e-05, + "relative_drop": { + "peak_normalized": 0.3122385243087939, + "spike_contrast": 0.46581551256749454 + }, + "seed": 2026073002, + "spike_contrast": 1.46386521692747, + "spike_mean": 7.70720471336972e-05, + "step": 2000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.2271117803789005, + "reference_mean": 5.202572019361132e-05, + "relative_drop": { + "peak_normalized": 0.348864160626338, + "spike_contrast": 0.5229153387055521 + }, + "seed": 2026073002, + "spike_contrast": 1.5062568095989999, + "spike_mean": 7.836409531591926e-05, + "step": 4000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.1987132665333755, + "reference_mean": 8.721922414325591e-05, + "relative_drop": { + "peak_normalized": 0.3924400935136531, + "spike_contrast": 0.5738819822015593 + }, + "seed": 2026073002, + "spike_contrast": 1.420186486886559, + "spike_mean": 0.00012386756352498197, + "step": 8000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 1.7301666786506156, + "reference_mean": 0.0001660444721454513, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.7974048446155259, + "spike_mean": 0.0001324046665104106, + "step": 0, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.1207412846841187, + "reference_mean": 3.459105876875886e-05, + "relative_drop": { + "peak_normalized": 0.006114927167165012, + "spike_contrast": -0.010453253577363798 + }, + "seed": 2026073003, + "spike_contrast": 0.5806090937185927, + "spike_mean": 2.0083883282495662e-05, + "step": 100, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.053846006906672, + "reference_mean": 7.881468479387911e-05, + "relative_drop": { + "peak_normalized": 0.064560568209724, + "spike_contrast": 0.01034983598448287 + }, + "seed": 2026073003, + "spike_contrast": 0.4728501886523352, + "spike_mean": 3.726753857336007e-05, + "step": 500, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 1.4998578508974612, + "reference_mean": 5.674074310701582e-05, + "relative_drop": { + "peak_normalized": 0.16941076849330763, + "spike_contrast": 0.26928770181550915 + }, + "seed": 2026073003, + "spike_contrast": 1.0673258658660505, + "spike_mean": 6.056086276657879e-05, + "step": 2000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 1.639948309138832, + "reference_mean": 5.3547755736939575e-05, + "relative_drop": { + "peak_normalized": 0.2238934062653275, + "spike_contrast": 0.3481531423133718 + }, + "seed": 2026073003, + "spike_contrast": 1.1560200799432452, + "spike_mean": 6.190228086779825e-05, + "step": 4000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 1.661826944810198, + "reference_mean": 9.516139861834408e-05, + "relative_drop": { + "peak_normalized": 0.27376190068536765, + "spike_contrast": 0.39856400381461277 + }, + "seed": 2026073003, + "spike_contrast": 1.1315967929439392, + "spike_mean": 0.00010768433348857797, + "step": 8000, + "variant": "uniform_group_7_mlp_forward" + } + ] +} diff --git a/src/data/k3-attnres-forward.json b/src/data/k3-attnres-forward.json new file mode 100644 index 0000000..250a98b --- /dev/null +++ b/src/data/k3-attnres-forward.json @@ -0,0 +1,4162 @@ +{ + "canonical_sha256_without_self": "eecf05c623e473ec5eba8afa54d555d2733a0af2fc5488398f1da095206d50c8", + "effects": { + "uniform_group_6_forward": { + "attenuation_passed": true, + "cells": [ + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.04626897714722, + "relative_drop": 0.5800152615825866, + "seed": 2026073001, + "variant": 1.2793864795162566 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.219066994145898, + "relative_drop": 0.24116240807746625, + "seed": 2026073001, + "variant": 2.442749046074982 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.3328477735441013, + "relative_drop": 0.6578260733512802, + "seed": 2026073002, + "variant": 1.1404136095960284 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.6189242296270336, + "relative_drop": 0.4994801034992014, + "seed": 2026073002, + "variant": 1.8113435808571552 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 1.8814916302334765, + "relative_drop": 0.4463254057279825, + "seed": 2026073003, + "variant": 1.0417341149957169 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 2.2882673690329693, + "relative_drop": 0.2472255709371281, + "seed": 2026073003, + "variant": 1.7225491622669935 + } + ], + "material_response_passed": true, + "passed_cells": 6, + "quality": { + "mean_delta_bpc": 0.009688047863022131, + "mean_maximum": 0.03, + "mean_passed": true, + "passed": true, + "passed_checks": 4, + "per_seed": { + "2026073001": { + "delta_bpc": 0.006538199944357226, + "passed": true, + "reference_bpc": 1.7123525940760473, + "variant_bpc": 1.7188907940204046 + }, + "2026073002": { + "delta_bpc": 0.012622129214292066, + "passed": true, + "reference_bpc": 1.7093240655856785, + "variant_bpc": 1.7219461947999706 + }, + "2026073003": { + "delta_bpc": 0.009903814430417102, + "passed": true, + "reference_bpc": 1.7030966812870931, + "variant_bpc": 1.7130004957175102 + } + }, + "per_seed_maximum": 0.05, + "required_checks": 4 + }, + "required_cells": 6, + "threshold": 0.2, + "variant": "uniform_group_6_forward" + }, + "uniform_group_7_forward": { + "attenuation_passed": true, + "cells": [ + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.04626897714722, + "relative_drop": 0.5886666348497648, + "seed": 2026073001, + "variant": 1.2530320695227308 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.219066994145898, + "relative_drop": 0.40991690985788126, + "seed": 2026073001, + "variant": 1.899516999280113 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.3328477735441013, + "relative_drop": 0.6029454996221533, + "seed": 2026073002, + "variant": 1.323322207559972 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.6189242296270336, + "relative_drop": 0.42497271322062063, + "seed": 2026073002, + "variant": 2.0809801808225887 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 1.8814916302334765, + "relative_drop": 0.4164606288913148, + "seed": 2026073003, + "variant": 1.0979244426526977 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 2.2882673690329693, + "relative_drop": 0.29135113042743926, + "seed": 2026073003, + "variant": 1.6215780843449914 + } + ], + "material_response_passed": true, + "passed_cells": 6, + "quality": { + "mean_delta_bpc": 0.0022818873094791283, + "mean_maximum": 0.03, + "mean_passed": true, + "passed": true, + "passed_checks": 4, + "per_seed": { + "2026073001": { + "delta_bpc": -0.003005805282623397, + "passed": true, + "reference_bpc": 1.7123525940760473, + "variant_bpc": 1.709346788793424 + }, + "2026073002": { + "delta_bpc": 0.005744005560504251, + "passed": true, + "reference_bpc": 1.7093240655856785, + "variant_bpc": 1.7150680711461828 + }, + "2026073003": { + "delta_bpc": 0.004107461650556532, + "passed": true, + "reference_bpc": 1.7030966812870931, + "variant_bpc": 1.7072041429376497 + } + }, + "per_seed_maximum": 0.05, + "required_checks": 4 + }, + "required_cells": 6, + "threshold": 0.2, + "variant": "uniform_group_7_forward" + }, + "uniform_group_7_mlp_forward": { + "attenuation_passed": true, + "cells": [ + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.04626897714722, + "relative_drop": 0.5659319887472724, + "seed": 2026073001, + "variant": 1.3222879166511747 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.219066994145898, + "relative_drop": 0.37304900548175707, + "seed": 2026073001, + "variant": 2.0181972534006216 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.3328477735441013, + "relative_drop": 0.5738819822015593, + "seed": 2026073002, + "variant": 1.420186486886559 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.6189242296270336, + "relative_drop": 0.3924400935136531, + "seed": 2026073002, + "variant": 2.1987132665333755 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 1.8814916302334765, + "relative_drop": 0.39856400381461277, + "seed": 2026073003, + "variant": 1.1315967929439392 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 2.2882673690329693, + "relative_drop": 0.27376190068536765, + "seed": 2026073003, + "variant": 1.661826944810198 + } + ], + "material_response_passed": true, + "passed_cells": 6, + "quality": { + "mean_delta_bpc": -0.00037419125261473063, + "mean_maximum": 0.03, + "mean_passed": true, + "passed": true, + "passed_checks": 4, + "per_seed": { + "2026073001": { + "delta_bpc": 0.002418420036432778, + "passed": true, + "reference_bpc": 1.7123525940760473, + "variant_bpc": 1.7147710141124801 + }, + "2026073002": { + "delta_bpc": 0.0017558568739839409, + "passed": true, + "reference_bpc": 1.7093240655856785, + "variant_bpc": 1.7110799224596625 + }, + "2026073003": { + "delta_bpc": -0.005296850668260911, + "passed": true, + "reference_bpc": 1.7030966812870931, + "variant_bpc": 1.6977998306188322 + } + }, + "per_seed_maximum": 0.05, + "required_checks": 4 + }, + "required_cells": 6, + "threshold": 0.2, + "variant": "uniform_group_7_mlp_forward" + }, + "uniform_groups_6_7_forward": { + "attenuation_passed": true, + "cells": [ + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.04626897714722, + "relative_drop": 0.7354105664383516, + "seed": 2026073001, + "variant": 0.8060105831398049 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.219066994145898, + "relative_drop": 0.5540266425004673, + "seed": 2026073001, + "variant": 1.4356181153951746 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.3328477735441013, + "relative_drop": 0.7725064680163986, + "seed": 2026073002, + "variant": 0.75820131156723 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.6189242296270336, + "relative_drop": 0.620048308391513, + "seed": 2026073002, + "variant": 1.3750163828497322 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 1.8814916302334765, + "relative_drop": 0.6211080443849011, + "seed": 2026073003, + "variant": 0.7128820432526025 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 2.2882673690329693, + "relative_drop": 0.3234439150908039, + "seed": 2026073003, + "variant": 1.5481412124184124 + } + ], + "material_response_passed": true, + "passed_cells": 6, + "quality": { + "mean_delta_bpc": 0.0065784582165467525, + "mean_maximum": 0.03, + "mean_passed": true, + "passed": true, + "passed_checks": 4, + "per_seed": { + "2026073001": { + "delta_bpc": 0.00569589341392529, + "passed": true, + "reference_bpc": 1.7123525940760473, + "variant_bpc": 1.7180484874899726 + }, + "2026073002": { + "delta_bpc": 0.009598287251159299, + "passed": true, + "reference_bpc": 1.7093240655856785, + "variant_bpc": 1.7189223528368378 + }, + "2026073003": { + "delta_bpc": 0.004441193984555669, + "passed": true, + "reference_bpc": 1.7030966812870931, + "variant_bpc": 1.7075378752716488 + } + }, + "per_seed_maximum": 0.05, + "required_checks": 4 + }, + "required_cells": 6, + "threshold": 0.2, + "variant": "uniform_groups_6_7_forward" + } + }, + "final_spectra": [ + { + "global_mean": 7.614457786075945e-05, + "normalized": [ + 0.7854618236676242, + 0.9130540849409994, + 0.8147131127464502, + 0.7522410092502885, + 0.9690072688462938, + 0.8892440039434748, + 0.8242354060534288, + 0.7396083689563772, + 0.6126026690255093, + 0.5670644008562938, + 0.5359283946772616, + 0.5145269561429746, + 0.7290174418180176, + 0.6731434248219464, + 0.6214223500409742, + 0.5883307730946398, + 0.7535191463387267, + 0.6934458909688136, + 0.6978155985835179, + 0.8168473226649812, + 3.219066994145898, + 2.5077095541323673, + 1.973834673592298, + 1.5759251613516239, + 2.2647260999477155, + 1.6880724220748888, + 1.092454232265147, + 0.8732751258075867, + 0.8206258345435784, + 0.5459870832104624, + 0.48994656248557816, + 0.45714680900426247 + ], + "peak_layer_1based": 21, + "peak_normalized": 3.219066994145898, + "reference_mean": 5.7697108584559625e-05, + "seed": 2026073001, + "spike_contrast": 3.04626897714722, + "spike_mean": 0.00017576091195223854, + "top_five_layers_1based": [ + 21, + 22, + 25, + 23, + 26 + ], + "values": [ + 5.980865898891352e-05, + 6.95241178618744e-05, + 6.203598604770377e-05, + 5.727907409891486e-05, + 7.378464943030849e-05, + 6.77111092954874e-05, + 6.276105705183e-05, + 5.631716703646816e-05, + 4.664637162932195e-05, + 4.3178879423066974e-05, + 4.080804137629457e-05, + 3.917843787348829e-05, + 5.551072536036372e-05, + 5.1256221922812983e-05, + 4.731794251711108e-05, + 4.47981983597856e-05, + 5.737639730796218e-05, + 5.2802144637098536e-05, + 5.3134874178795144e-05, + 6.219849456101656e-05, + 0.0002451144973747432, + 0.00019094848539680243, + 0.00015029680798761547, + 0.00011999815615126863, + 0.0001724466128507629, + 0.00012853756197728217, + 8.318446634802967e-05, + 6.64951658109203e-05, + 6.248620775295421e-05, + 4.157395596848801e-05, + 3.7306774174794555e-05, + 3.4809250792022794e-05 + ], + "variant": "learned_reference" + }, + { + "global_mean": 7.03343731061068e-05, + "normalized": [ + 0.6809932372926891, + 0.7475632825133391, + 0.6323153694166137, + 0.6077851857944574, + 0.7056447138179579, + 0.5942446047107566, + 0.530223262177042, + 0.504089839823597, + 0.6368974480554273, + 0.5978888745536934, + 0.5559539610577222, + 0.5232972563751404, + 0.4434509399790644, + 0.41411238498840724, + 0.4174418363858031, + 0.5192489699581264, + 1.2557404086895352, + 1.2676084925356499, + 1.1574347089891701, + 1.060138766241709, + 3.6189242296270336, + 2.81239769967395, + 2.2018329150221976, + 1.7615672973682381, + 1.8179181698124194, + 1.4871187506667403, + 1.086854135909831, + 0.872703317202576, + 0.8717669049561169, + 0.6523365087620864, + 0.4999226915339381, + 0.46458383610897114 + ], + "peak_layer_1based": 21, + "peak_normalized": 3.6189242296270336, + "reference_mean": 5.15456125598306e-05, + "seed": 2026073002, + "spike_contrast": 3.3328477735441013, + "spike_mean": 0.00017179368005599828, + "top_five_layers_1based": [ + 21, + 22, + 23, + 25, + 24 + ], + "values": [ + 4.789723243447952e-05, + 5.257939483271912e-05, + 4.447350511327386e-05, + 4.274819002603181e-05, + 4.96310785820242e-05, + 4.17958217440173e-05, + 3.729292075149715e-05, + 3.545484287315048e-05, + 4.479578274185769e-05, + 4.205213917884976e-05, + 3.91026733268518e-05, + 3.680578447529115e-05, + 3.1189843866741285e-05, + 2.9126334993634373e-05, + 2.9360509870457463e-05, + 3.65210507879965e-05, + 8.83217144291848e-05, + 8.915644866647199e-05, + 8.140744466800243e-05, + 7.45641955290921e-05, + 0.0002545347670093179, + 0.0001978082291316241, + 0.000154864537762478, + 0.00012389873154461384, + 0.00012786213483195752, + 0.00010459556506248191, + 7.644320430699736e-05, + 6.138104072306305e-05, + 6.131517875473946e-05, + 4.588167939800769e-05, + 3.516174911055714e-05, + 3.2676212867954746e-05 + ], + "variant": "learned_reference" + }, + { + "global_mean": 8.757388354752038e-05, + "normalized": [ + 0.7720675832605176, + 0.803585946279625, + 0.7960780094481321, + 0.752473219070419, + 1.180467409539882, + 1.1778079020041263, + 1.029829529518624, + 0.9385078067458692, + 1.0206984705622164, + 0.9298229900567075, + 0.8640759923745102, + 0.8130195958778703, + 0.7736070401530517, + 0.7280910766467854, + 0.7061969571118035, + 0.6826252956632183, + 1.642168884788434, + 1.5117445082936212, + 1.3334176736203447, + 1.1637251407789435, + 2.2882673690329693, + 1.8954517085610634, + 1.5552436162725376, + 1.306771834154562, + 1.2228641636923565, + 0.9319785952743014, + 0.7801153976292595, + 0.661371584878177, + 0.561946672486053, + 0.4345672285647456, + 0.37483694087850516, + 0.36657385678076787 + ], + "peak_layer_1based": 21, + "peak_normalized": 2.2882673690329693, + "reference_mean": 7.697225831819836e-05, + "seed": 2026073003, + "spike_contrast": 1.8814916302334765, + "spike_mean": 0.0001448226597858593, + "top_five_layers_1based": [ + 21, + 22, + 17, + 23, + 18 + ], + "values": [ + 6.761295662727207e-05, + 7.037314207991585e-05, + 6.971564289415255e-05, + 6.589700205950066e-05, + 0.00010337811545468867, + 0.00010314521205145866, + 9.018617129186168e-05, + 8.218877337640151e-05, + 8.93865289981477e-05, + 8.14282102510333e-05, + 7.567049033241346e-05, + 7.11992834112607e-05, + 6.774777284590527e-05, + 6.376176315825433e-05, + 6.184441008372232e-05, + 5.9780148149002343e-05, + 0.00014381110668182373, + 0.00013238933752290905, + 0.0001167725640698336, + 0.00010191192995989695, + 0.0002003924601012841, + 0.0001659920671954751, + 0.0001361987233394757, + 0.00011443908442743123, + 0.00010709096386563033, + 8.16169849713333e-05, + 6.831773498561233e-05, + 5.791887815576047e-05, + 4.921185245621018e-05, + 3.8056739867897704e-05, + 3.282592660980299e-05, + 3.210229624528438e-05 + ], + "variant": "learned_reference" + }, + { + "global_mean": 9.461854074288567e-05, + "normalized": [ + 1.049728483120271, + 1.106104490273285, + 1.0534380329881132, + 0.9962730544079834, + 1.2318533093339465, + 1.1552933983529543, + 1.0861468966704533, + 0.9912509359934122, + 0.9519187879066954, + 0.9095594041307715, + 0.8736030694602711, + 0.8511453745866416, + 0.9964773718623544, + 0.9452695889437478, + 0.8851965662642772, + 0.8230819838948948, + 1.055843626862817, + 0.9503284645066779, + 0.9178198426953132, + 0.83569345377416, + 1.001707960212492, + 0.9726643532652439, + 0.8840872386027205, + 0.8281518558432126, + 2.442749046074982, + 1.8545856115377921, + 1.2448015915768342, + 0.8974028616357965, + 0.7847156721750955, + 0.5164169028929032, + 0.48169339375062104, + 0.42499737640326557 + ], + "peak_layer_1based": 25, + "peak_normalized": 2.442749046074982, + "reference_mean": 9.066082081214215e-05, + "seed": 2026073001, + "spike_contrast": 1.2793864795162566, + "spike_mean": 0.00011599022836890071, + "top_five_layers_1based": [ + 25, + 26, + 27, + 5, + 6 + ], + "values": [ + 9.932377724908292e-05, + 0.0001046579927788116, + 9.967476944439113e-05, + 9.426590258954093e-05, + 0.00011655616253847256, + 0.00010931217548204586, + 0.00010276963439537212, + 9.379071707371622e-05, + 9.0069166617468e-05, + 8.606118353782222e-05, + 8.265904762083665e-05, + 8.053413330344483e-05, + 9.428523480892181e-05, + 8.944002911448479e-05, + 8.375600737053901e-05, + 7.787881622789428e-05, + 9.990238322643563e-05, + 8.991869253804907e-05, + 8.684277418069541e-05, + 7.90720951044932e-05, + 9.478014544583857e-05, + 9.203208173858002e-05, + 8.36510444059968e-05, + 7.835852011339739e-05, + 0.0002311293501406908, + 0.0001754781842464581, + 0.00011778131010942161, + 8.49109492264688e-05, + 7.42486517992802e-05, + 4.886261376668699e-05, + 4.557712600217201e-05, + 4.02126315748319e-05 + ], + "variant": "uniform_group_6_forward" + }, + { + "global_mean": 9.923219420215901e-05, + "normalized": [ + 1.1210080243157192, + 1.1779108698425953, + 1.1110378438066775, + 1.0873712307498702, + 1.2745569576226279, + 1.1932568398188368, + 1.1102274829736989, + 1.0642653152400943, + 1.187072449259881, + 1.1407628792799271, + 1.075379840801642, + 0.9909432042363304, + 0.9657037491393293, + 0.9056250839636294, + 0.859253263137928, + 0.8561867674125543, + 1.1033690382039603, + 1.1437505532630414, + 1.0398233895296574, + 0.8632063018267061, + 1.0230976357898443, + 0.9682149731864415, + 0.9085527067775245, + 0.868443658317733, + 1.8113435808571552, + 1.4139391675400481, + 0.9502866562886768, + 0.700638694754292, + 0.724043106368941, + 0.5551299824318824, + 0.4277691513860246, + 0.3778296018767301 + ], + "peak_layer_1based": 25, + "peak_normalized": 1.8113435808571552, + "reference_mean": 9.7101816613253e-05, + "seed": 2026073002, + "spike_contrast": 1.1404136095960284, + "spike_mean": 0.00011073623318225145, + "top_five_layers_1based": [ + 25, + 26, + 5, + 6, + 9 + ], + "values": [ + 0.00011124008597107604, + 0.00011688668018905446, + 0.00011025072308257222, + 0.00010790223313961178, + 0.00012647708354052156, + 0.00011840949446195737, + 0.00011017030919902027, + 0.000105609382444527, + 0.00011779580381698906, + 0.0001132004035753198, + 0.00010671230120351538, + 9.833346848608926e-05, + 9.582890197634697e-05, + 8.986716420622543e-05, + 8.526558667654172e-05, + 8.496129157720134e-05, + 0.00010948973067570478, + 0.00011349687702022493, + 0.00010318395652575418, + 8.565785537939519e-05, + 0.00010152422328246757, + 9.607809624867514e-05, + 9.015767864184454e-05, + 8.61775697558187e-05, + 0.00017974359798245132, + 0.0001403082860633731, + 9.42990300245583e-05, + 6.95259150234051e-05, + 7.184838614193723e-05, + 5.5086766224121675e-05, + 4.244847150403075e-05, + 3.74928604287561e-05 + ], + "variant": "uniform_group_6_forward" + }, + { + "global_mean": 9.917417844462761e-05, + "normalized": [ + 1.1028432904984338, + 1.1270987866010154, + 1.1163555912049457, + 1.0645032432277715, + 1.4143648500362929, + 1.4417529038578565, + 1.297502061738055, + 1.1918983746366243, + 1.2326024032371155, + 1.1513560434742482, + 1.0785282950209376, + 1.004374246074776, + 0.8587004078072071, + 0.8434604976303082, + 0.847621932333013, + 0.8791225594717167, + 1.3801494080155523, + 1.1693864816711759, + 0.9848436312832127, + 0.8570887157380096, + 0.93528776842758, + 0.8808606602005659, + 0.8340094144302829, + 0.8022181134032798, + 1.7225491622669935, + 1.320199275814519, + 0.9817677850522812, + 0.7360945725690212, + 0.581644906646078, + 0.4241043178935042, + 0.37239268483752097, + 0.3653176149001062 + ], + "peak_layer_1based": 25, + "peak_normalized": 1.7225491622669935, + "reference_mean": 9.853165789132213e-05, + "seed": 2026073003, + "spike_contrast": 1.0417341149957169, + "spike_mean": 0.0001026437894324772, + "top_five_layers_1based": [ + 25, + 6, + 5, + 17, + 26 + ], + "values": [ + 0.00010937357728835195, + 0.00011177909618709236, + 0.00011071364860981703, + 0.00010557123459875584, + 0.00014026847202330828, + 0.0001429846597602591, + 0.0001286787010030821, + 0.00011820554209407419, + 0.00012224233068991452, + 0.00011418478970881552, + 0.00010696215758798644, + 9.960799070540816e-05, + 8.516090747434646e-05, + 8.364950190298259e-05, + 8.40622087707743e-05, + 8.718625758774579e-05, + 0.00013687518367078155, + 0.00011597294360399246, + 9.767105802893639e-05, + 8.500106923747808e-05, + 9.275639604311436e-05, + 8.73586322995834e-05, + 8.271219849120826e-05, + 7.955932233016938e-05, + 0.00017083239799831063, + 0.00013092967856209725, + 9.736601350596175e-05, + 7.3001574492082e-05, + 5.768415576312691e-05, + 4.2060197301907465e-05, + 3.693173857755028e-05, + 3.6230074329068884e-05 + ], + "variant": "uniform_group_6_forward" + }, + { + "global_mean": 9.944817804807826e-05, + "normalized": [ + 1.168387143183479, + 1.2401657617613517, + 1.182335288746566, + 1.1426773374969352, + 1.3063371444652658, + 1.2500084214967857, + 1.1963762785466578, + 1.0978585928746885, + 1.0186348761426658, + 0.9698923101848725, + 0.9394670561774453, + 0.9082441026301723, + 1.0762223006504932, + 1.0045589874449765, + 0.935031603833723, + 0.8850145330621985, + 1.0867928621305778, + 0.9915017514752766, + 0.9834621282857434, + 0.988038493159388, + 1.899516999280113, + 1.4879874428901083, + 1.1118576600996788, + 0.8280383879871337, + 0.6994795556712323, + 0.6618611779401082, + 0.6268707530450582, + 0.6053853235605826, + 0.992054646588788, + 0.6333634112643542, + 0.5808856023956639, + 0.501692065527916 + ], + "peak_layer_1based": 21, + "peak_normalized": 1.899516999280113, + "reference_mean": 9.566590583913408e-05, + "seed": 2026073001, + "spike_contrast": 1.2530320695227308, + "spike_mean": 0.00011987244797637687, + "top_five_layers_1based": [ + 21, + 22, + 5, + 6, + 2 + ], + "values": [ + 0.00011619397264439613, + 0.00012333222548477352, + 0.00011758109030779451, + 0.00011363717931089923, + 0.0001299128489335999, + 0.0001243110600626096, + 0.0001189774411614053, + 0.00010918003681581467, + 0.00010130138252861798, + 9.645402315072715e-05, + 9.342828707303852e-05, + 9.032322122948244e-05, + 0.00010702834697440267, + 9.990156104322523e-05, + 9.298718941863626e-05, + 8.801308285910636e-05, + 0.00010807957005454227, + 9.860304271569476e-05, + 9.78035168373026e-05, + 9.825862798606977e-05, + 0.00018890350474976003, + 0.00014797764015384018, + 0.00011057221854571253, + 8.234690903918818e-05, + 6.956196739338338e-05, + 6.582088826689869e-05, + 6.234115426195785e-05, + 6.020446744514629e-05, + 9.865802712738514e-05, + 6.298683729255572e-05, + 5.776801481260918e-05, + 4.9892361857928336e-05 + ], + "variant": "uniform_group_7_forward" + }, + { + "global_mean": 9.281247480430466e-05, + "normalized": [ + 1.1237833551331617, + 1.158687029916818, + 1.1074600411220088, + 1.0933043261157995, + 1.2290412572656306, + 1.1553665661391181, + 1.0761233798039562, + 1.0372293629967688, + 1.1441068888341974, + 1.1017626658857236, + 1.033201313373911, + 0.9601560490976951, + 0.8799180406514036, + 0.8396732137735404, + 0.8279983609051412, + 0.8670270589090915, + 1.3941783567635277, + 1.3888596254040555, + 1.1960254710619764, + 1.0567738153628634, + 2.0809801808225887, + 1.593322301655694, + 1.2033761794995008, + 0.8879522679216262, + 0.5327896030780552, + 0.5035235279704214, + 0.4742412252680671, + 0.4569905081383433, + 0.8840222894146883, + 0.6881066951417837, + 0.5265942288298747, + 0.4974248137429677 + ], + "peak_layer_1based": 21, + "peak_normalized": 2.0809801808225887, + "reference_mean": 8.834915543384677e-05, + "seed": 2026073002, + "spike_contrast": 1.323322207559972, + "spike_mean": 0.00011691439940477721, + "top_five_layers_1based": [ + 21, + 22, + 17, + 18, + 5 + ], + "values": [ + 0.00010430111433379352, + 0.00010754061077022925, + 0.00010278610716341063, + 0.00010147228022105992, + 0.00011407036072341725, + 0.0001072324303095229, + 9.987767407437786e-05, + 9.626782411942258e-05, + 0.00010618739179335535, + 0.00010225731966784224, + 9.589397086529061e-05, + 8.911445911508054e-05, + 8.166737097781152e-05, + 7.793214899720624e-05, + 7.684857700951397e-05, + 8.047092705965042e-05, + 0.0001293971436098218, + 0.0001289034989895299, + 0.00011100608389824629, + 9.808179311221465e-05, + 0.00019314092060085386, + 0.0001478801859775558, + 0.00011168832133989781, + 8.24130474939011e-05, + 4.944952161167748e-05, + 4.673326475312933e-05, + 4.401550177135505e-05, + 4.241442002239637e-05, + 8.204829646274447e-05, + 6.386488530552015e-05, + 4.887451359536499e-05, + 4.6167227992555127e-05 + ], + "variant": "uniform_group_7_forward" + }, + { + "global_mean": 9.948732213160838e-05, + "normalized": [ + 1.1315008092677528, + 1.1668656649355191, + 1.1366208839234715, + 1.101693372689152, + 1.3429705189947834, + 1.3541548349677262, + 1.2663030159282187, + 1.1984341076366907, + 1.2506532523938356, + 1.178608289839053, + 1.0962174254559829, + 1.031747955952998, + 0.8844731687384572, + 0.8675687093679173, + 0.8798558207776407, + 0.8771834122588129, + 1.4774236545882187, + 1.3583096070854734, + 1.177294574444737, + 1.0471476725613014, + 1.6215780843449914, + 1.3337352379969853, + 1.0512149026578388, + 0.8088985879483863, + 0.5914661841860449, + 0.5588135222163362, + 0.5288816472508608, + 0.5066587724046175, + 0.6923084127530423, + 0.5454130839988593, + 0.47793325135715387, + 0.45807156107714075 + ], + "peak_layer_1based": 21, + "peak_normalized": 1.6215780843449914, + "reference_mean": 9.798803714349763e-05, + "seed": 2026073003, + "spike_contrast": 1.0979244426526977, + "spike_mean": 0.00010758346106740647, + "top_five_layers_1based": [ + 21, + 17, + 18, + 6, + 5 + ], + "values": [ + 0.00011256998550379649, + 0.00011608834029175341, + 0.00011307936802040786, + 0.00010960452345898375, + 0.0001336085406364873, + 0.00013472123828250915, + 0.0001259810960618779, + 0.00011922900011995807, + 0.00012442414299584925, + 0.00011725658259820193, + 0.00010905973613262177, + 0.0001026458412525244, + 8.79938670550473e-05, + 8.631208766018972e-05, + 8.753449947107583e-05, + 8.726862870389596e-05, + 0.00014698492304887623, + 0.0001351345854345709, + 0.00011712588457157835, + 0.00010417791781947017, + 0.00016132646123878658, + 0.00013268974726088345, + 0.00010458255565026775, + 8.047515439102426e-05, + 5.884338679607026e-05, + 5.559486089623533e-05, + 5.261701880954206e-05, + 5.040612450102344e-05, + 6.887591007398441e-05, + 5.42616871825885e-05, + 4.7548299335176125e-05, + 4.5572312956210226e-05 + ], + "variant": "uniform_group_7_forward" + }, + { + "global_mean": 0.00010924439834525401, + "normalized": [ + 1.1802831793473259, + 1.2700138904861653, + 1.1942950278819688, + 1.1430133139888945, + 1.4356181153951746, + 1.3802623223364756, + 1.3279057761796307, + 1.1930909866337074, + 1.1655275133184586, + 1.115130814366578, + 1.0705665024105082, + 1.0317385365452825, + 1.2195044999869864, + 1.163096652743226, + 1.0753236575271325, + 0.9976493434875472, + 1.0906690227048055, + 0.9850593266028592, + 0.947438465704084, + 0.8382655321432377, + 0.9180309004493582, + 0.8783840559942786, + 0.7682382587032985, + 0.6856989599873513, + 0.9056733931341846, + 0.8569565441858251, + 0.814728517156546, + 0.781674128218417, + 0.8819737346732008, + 0.6398484858150421, + 0.5759106186859398, + 0.4684299232065091 + ], + "peak_layer_1based": 5, + "peak_normalized": 1.4356181153951746, + "reference_mean": 0.0001126591938642647, + "seed": 2026073001, + "spike_contrast": 0.8060105831398049, + "spike_mean": 9.080450254259632e-05, + "top_five_layers_1based": [ + 5, + 6, + 7, + 2, + 13 + ], + "values": [ + 0.00012893932580482215, + 0.00013874190335627645, + 0.00013047004176769406, + 0.0001248678017873317, + 0.0001568332372698933, + 0.00015078592696227133, + 0.00014506626757793128, + 0.00013033850700594485, + 0.00012732735194731504, + 0.00012182179489172995, + 0.00011695339344441891, + 0.00011271165567450225, + 0.00013322403538040817, + 0.00012706179404631257, + 0.00011747308599296957, + 0.00010898760228883475, + 0.00011914948117919266, + 0.00010761221346911043, + 0.00010350234515499324, + 9.157581371255219e-05, + 0.00010028973338194191, + 9.595853771315888e-05, + 8.392572635784745e-05, + 7.49087703297846e-05, + 9.893974493024871e-05, + 9.361770207760856e-05, + 8.900452667148784e-05, + 8.53935198392719e-05, + 9.635069000069052e-05, + 6.989986286498606e-05, + 6.29150090389885e-05, + 5.117334512760863e-05 + ], + "variant": "uniform_groups_6_7_forward" + }, + { + "global_mean": 0.00010773822862120142, + "normalized": [ + 1.2021320872392602, + 1.239737264026566, + 1.1872892678543405, + 1.1702441756008668, + 1.3188412036166208, + 1.2665431321842968, + 1.2036994085742312, + 1.1756571340479791, + 1.3750163828497322, + 1.3316928603748481, + 1.281390543858547, + 1.149992314721499, + 1.1427451418889247, + 1.07163124546429, + 1.0057863257095283, + 0.9800428296614334, + 1.185063358202469, + 1.1602375771501046, + 1.0699312204744063, + 0.9180188972304961, + 0.879376133189205, + 0.826676641642478, + 0.7496293723357395, + 0.6894108849482833, + 0.7947654984227666, + 0.7537914046760064, + 0.714759241512971, + 0.6857459679280232, + 0.8575389194858783, + 0.6740471108590389, + 0.4954098247900116, + 0.4431566294791582 + ], + "peak_layer_1based": 9, + "peak_normalized": 1.3750163828497322, + "reference_mean": 0.00011196851617704076, + "seed": 2026073002, + "spike_contrast": 0.75820131156723, + "spike_mean": 8.489467581966892e-05, + "top_five_layers_1based": [ + 9, + 10, + 5, + 11, + 6 + ], + "values": [ + 0.00012951558164786547, + 0.00013356709678191692, + 0.00012791644257958978, + 0.00012608003453351557, + 0.00014208961511030793, + 0.0001364551135338843, + 0.00012968444207217544, + 0.0001266632170882076, + 0.00014814182941336185, + 0.00014347422984428704, + 0.00013805474736727774, + 0.00012389813491608948, + 0.00012311733735259622, + 0.00011545565212145448, + 0.00010836163710337132, + 0.00010558807844063267, + 0.0001276766270166263, + 0.00012500194134190679, + 0.00011527249444043264, + 9.89057298284024e-05, + 9.474242688156664e-05, + 8.906467701308429e-05, + 8.076374069787562e-05, + 7.427590753650293e-05, + 8.562662696931511e-05, + 8.121215068968013e-05, + 7.700689457124099e-05, + 7.388105586869642e-05, + 9.238972415914759e-05, + 7.262064173119143e-05, + 5.337457696441561e-05, + 4.774491026182659e-05 + ], + "variant": "uniform_groups_6_7_forward" + }, + { + "global_mean": 0.00011232519011628028, + "normalized": [ + 1.2171245344139678, + 1.2518726224846988, + 1.2474388463543393, + 1.1988192730216298, + 1.4927538601863934, + 1.5481412124184124, + 1.4360475424686572, + 1.3509742038640624, + 1.4243829697937715, + 1.3501314702932352, + 1.2526613330118617, + 1.1834896837873747, + 1.1372245823864426, + 1.0931400945849206, + 1.0210171936909138, + 1.0031357605655953, + 1.2385446064509475, + 1.1180676483584988, + 0.9704150912522292, + 0.8389663380052635, + 0.8500759113538735, + 0.7841120908683951, + 0.7240270009917109, + 0.6649331756680142, + 0.7086799870587199, + 0.664155606583909, + 0.6264421457123408, + 0.5953397710031234, + 0.6167462401154395, + 0.5144908854286474, + 0.4516352036771426, + 0.4250131141454677 + ], + "peak_layer_1based": 6, + "peak_normalized": 1.5481412124184124, + "reference_mean": 0.00011760102872223663, + "seed": 2026073003, + "spike_contrast": 0.7128820432526025, + "spike_mean": 8.383566164411604e-05, + "top_five_layers_1based": [ + 6, + 5, + 7, + 9, + 8 + ], + "values": [ + 0.00013671374472323805, + 0.00014061683032196015, + 0.00014011880557518452, + 0.0001346576027572155, + 0.00016767386114224792, + 0.00017389525601174682, + 0.000161304313223809, + 0.0001517484342912212, + 0.0001599940878804773, + 0.00015165377408266068, + 0.00014070542238187045, + 0.0001329357037320733, + 0.00012773896742146462, + 0.0001227871689479798, + 0.00011468595039332286, + 0.0001126774150179699, + 0.00013911975838709623, + 0.0001255871611647308, + 0.00010900205961661413, + 9.423705341760069e-05, + 9.548493835609406e-05, + 8.807553967926651e-05, + 8.132647053571418e-05, + 7.46887453715317e-05, + 7.960261427797377e-05, + 7.460140477633104e-05, + 7.036523311398923e-05, + 6.68716529617086e-05, + 6.92761386744678e-05, + 5.7790286518866196e-05, + 5.073001011624001e-05, + 4.773967884830199e-05 + ], + "variant": "uniform_groups_6_7_forward" + }, + { + "global_mean": 9.644760370974836e-05, + "normalized": [ + 1.2063054387246115, + 1.325204706753435, + 1.2220751579218956, + 1.159101443775728, + 1.3059872521838534, + 1.2144440767529154, + 1.1380447745464686, + 1.0346072062950988, + 1.0430331178846848, + 0.9731423573010324, + 0.9349717881195196, + 0.8850937651629682, + 0.9944058061604059, + 0.9359444294077571, + 0.8662547904927963, + 0.8178904608154263, + 0.9633915774647989, + 0.9200487262694695, + 0.9372542851967641, + 1.0738157473594052, + 2.0181972534006216, + 1.5498465965170765, + 1.1628327561745115, + 0.8642229033868173, + 0.6993665796671015, + 0.6514957751047279, + 0.6160923262561516, + 0.6002126162888178, + 1.0695106422329144, + 0.6705551085124273, + 0.629767581294126, + 0.5168829525756723 + ], + "peak_layer_1based": 21, + "peak_normalized": 2.0181972534006216, + "reference_mean": 9.182359806598267e-05, + "seed": 2026073001, + "spike_contrast": 1.3222879166511747, + "spike_mean": 0.00012141723418608308, + "top_five_layers_1based": [ + 21, + 22, + 2, + 5, + 3 + ], + "values": [ + 0.00011634526890702546, + 0.00012781281839124858, + 0.00011786622053477913, + 0.0001117925567086786, + 0.0001259593409486115, + 0.00011713022104231641, + 0.00010976169141940773, + 9.978538582799956e-05, + 0.00010059804480988532, + 9.385724843014032e-05, + 9.017578850034624e-05, + 8.536517270840704e-05, + 9.590805711923167e-05, + 9.026959742186591e-05, + 8.354819874512032e-05, + 7.888357504270971e-05, + 9.291680908063427e-05, + 8.873649494489655e-05, + 9.039592987392098e-05, + 0.00010356695565860718, + 0.00019465028890408576, + 0.00014947899035178125, + 0.00011215243284823373, + 8.33522281027399e-05, + 6.745223072357476e-05, + 6.283520633587614e-05, + 5.942062853137031e-05, + 5.788906855741516e-05, + 0.00010315173858543858, + 6.46734333713539e-05, + 6.0739574109902605e-05, + 4.9852122174343094e-05 + ], + "variant": "uniform_group_7_mlp_forward" + }, + { + "global_mean": 9.294552717165061e-05, + "normalized": [ + 1.1804207293783469, + 1.2244457182848458, + 1.1777345623601965, + 1.161365335790943, + 1.201585743382261, + 1.1521788700409004, + 1.0825081645663768, + 1.0517854717798363, + 1.070778396366818, + 1.043787326113011, + 0.9805167967494025, + 0.8948937161182166, + 0.9628742358933498, + 0.8986009929628155, + 0.8549655350664673, + 0.8542336770652816, + 1.2505796011268457, + 1.2251506472933684, + 1.0881850936912019, + 0.9949638099034118, + 2.1987132665333755, + 1.6868568375183757, + 1.2775504746361555, + 0.9575103563082339, + 0.542818468880225, + 0.49531325994081177, + 0.45802909107422135, + 0.4404704484743748, + 0.8766450963633319, + 0.7024747909383445, + 0.5268035464411853, + 0.48525993895746805 + ], + "peak_layer_1based": 21, + "peak_normalized": 2.1987132665333755, + "reference_mean": 8.721922414325591e-05, + "seed": 2026073002, + "spike_contrast": 1.420186486886559, + "spike_mean": 0.00012386756352498197, + "top_five_layers_1based": [ + 21, + 22, + 23, + 17, + 18 + ], + "values": [ + 0.00010971482697641477, + 0.00011380675277905539, + 0.00010946515976684168, + 0.00010794371337397024, + 0.00011168202036060393, + 0.0001070898724719882, + 0.0001006142920232378, + 9.775875514606014e-05, + 9.952406253432855e-05, + 9.70153632806614e-05, + 9.11346505745314e-05, + 8.317636820720509e-05, + 8.949485345510766e-05, + 8.352094300789759e-05, + 7.946522237034515e-05, + 7.939719944261014e-05, + 0.00011623578029684722, + 0.00011387227277737111, + 0.00010114193719346076, + 9.247743582818657e-05, + 0.00020436056365724653, + 0.0001567857980262488, + 0.00011874260235344991, + 8.899630483938381e-05, + 5.045254874858074e-05, + 4.603715206030756e-05, + 4.257175532984547e-05, + 4.093975803698413e-05, + 8.148024062393233e-05, + 6.529188976855949e-05, + 4.8964033339871094e-05, + 4.510274084168486e-05 + ], + "variant": "uniform_group_7_mlp_forward" + }, + { + "global_mean": 9.711810719181813e-05, + "normalized": [ + 1.1620973642676113, + 1.1894598283879425, + 1.1638141251636995, + 1.129343004808355, + 1.323780054970507, + 1.3108024963433114, + 1.214125895326903, + 1.1368690328501372, + 1.1781773042479384, + 1.0956284129945206, + 1.0240851465456522, + 0.9655277546942812, + 0.9261251890313171, + 0.910704008752465, + 0.9353340398201178, + 0.9580511729624018, + 1.505920482127743, + 1.3295117813120352, + 1.1710289411277004, + 1.030240238186059, + 1.661826944810198, + 1.3665022660755686, + 1.0818738730339919, + 0.8302713446442275, + 0.603514051148169, + 0.5612462965823031, + 0.5267519889043605, + 0.5075502261808718, + 0.701860701281538, + 0.5409652557672129, + 0.49076357856219416, + 0.46624719908866613 + ], + "peak_layer_1based": 21, + "peak_normalized": 1.661826944810198, + "reference_mean": 9.516139861834408e-05, + "seed": 2026073003, + "spike_contrast": 1.1315967929439392, + "spike_mean": 0.00010768433348857797, + "top_five_layers_1based": [ + 21, + 17, + 22, + 18, + 5 + ], + "values": [ + 0.00011286069639027119, + 0.00011551808711374179, + 0.0001130274249590002, + 0.00010967965499730781, + 0.00012856301327701658, + 0.0001273026573471725, + 0.00011791360884672031, + 0.00011041056859539822, + 0.0001144223497249186, + 0.00010640535765560344, + 9.945721103576943e-05, + 9.377022797707468e-05, + 8.994352538138628e-05, + 8.844584954204038e-05, + 9.083787153940648e-05, + 9.304411651100963e-05, + 0.00014625214680563658, + 0.0001291196676902473, + 0.00011372811422916129, + 0.0001000549818854779, + 0.00016139348736032844, + 0.00013271211355458945, + 0.00010506954276934266, + 8.063438144745305e-05, + 5.8612142311176285e-05, + 5.450717799249105e-05, + 5.115715612191707e-05, + 4.9292317271465436e-05, + 6.816338282078505e-05, + 5.253752169664949e-05, + 4.7662029828643426e-05, + 4.528104545897804e-05 + ], + "variant": "uniform_group_7_mlp_forward" + } + ], + "historical_pairing": { + "uniform_group_6_forward:2026073001": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + }, + "uniform_group_6_forward:2026073002": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + }, + "uniform_group_6_forward:2026073003": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + }, + "uniform_group_7_forward:2026073001": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + }, + "uniform_group_7_forward:2026073002": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + }, + "uniform_group_7_forward:2026073003": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + }, + "uniform_group_7_mlp_forward:2026073001": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + }, + "uniform_group_7_mlp_forward:2026073002": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + }, + "uniform_group_7_mlp_forward:2026073003": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + }, + "uniform_groups_6_7_forward:2026073001": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + }, + "uniform_groups_6_7_forward:2026073002": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + }, + "uniform_groups_6_7_forward:2026073003": { + "checks": { + "architecture": true, + "batch_size": true, + "depth": true, + "initial_mixer_parameters": true, + "initial_public_parameters": true, + "manifest.diagnostic_tensor_sha256": true, + "manifest.formal_schedule_sha256": true, + "manifest.input_gate_tensor_hashes": true, + "manifest.validation_tensor_sha256": true, + "model_topology": true, + "optimizer_hyperparameters": true, + "scientific_environment": true, + "seed": true, + "steps": true + }, + "metadata_equal": true, + "passed": true, + "reference_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + }, + "run_environment": { + "compute_capability": [ + 12, + 0 + ], + "cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "torch": "2.11.0+cu128" + } + } + }, + "historical_reference_target_bytes": 196608000, + "input_files": { + "formal": [ + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073001.json", + "seed": 2026073001, + "sha256": "483d3d62b6a798234ff6db808ce62dd3327179e731161f87cdeea9e61f84935b", + "variant": "uniform_group_6_forward" + }, + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073002.json", + "seed": 2026073002, + "sha256": "105faa234dcc2ad6f0ff122eb424cfbb33d4f3d1d01b8bd5d4538c7834095b9d", + "variant": "uniform_group_6_forward" + }, + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_group_6_forward-seed-2026073003.json", + "seed": 2026073003, + "sha256": "4138071822e7a8c0baf9d19b7a8d85ae84b7e82768b458216ab97678c181cd77", + "variant": "uniform_group_6_forward" + }, + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073001.json", + "seed": 2026073001, + "sha256": "783bb123b8b2f6b5dbf829dcbd9ca56c75fcbf419a1ced85d96760133516c1be", + "variant": "uniform_group_7_forward" + }, + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073002.json", + "seed": 2026073002, + "sha256": "1e8a6913df2e0e934cef8b74ef4b88a6edb5b96c4e897b1b8ca33392ac8869b2", + "variant": "uniform_group_7_forward" + }, + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_forward-seed-2026073003.json", + "seed": 2026073003, + "sha256": "bdb78be112bd58d5bf14956ad32d7ff6a49d33586b48a9edf70845777530781a", + "variant": "uniform_group_7_forward" + }, + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073001.json", + "seed": 2026073001, + "sha256": "ed122082b524813e3d970f5d03bd7c81b399543c4cc781974c0f685fb49b278c", + "variant": "uniform_group_7_mlp_forward" + }, + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073002.json", + "seed": 2026073002, + "sha256": "e2774b01ade04cb970d099d3bf4c33ac78af0cc95d580f0a5f91db8d370b4074", + "variant": "uniform_group_7_mlp_forward" + }, + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_group_7_mlp_forward-seed-2026073003.json", + "seed": 2026073003, + "sha256": "b5b0788716f56657ee92d48bd118aab5968492ad6531a181916eeeec0886e8b2", + "variant": "uniform_group_7_mlp_forward" + }, + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073001.json", + "seed": 2026073001, + "sha256": "0962ebd1a00a11e61ac795282bfa99412166c8752f2a3731f137030d7f134dc1", + "variant": "uniform_groups_6_7_forward" + }, + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073002.json", + "seed": 2026073002, + "sha256": "d86eb5d93013e47be47c37552fa5ef7e2aa2176883fda6c6ca1827c993dae6a4", + "variant": "uniform_groups_6_7_forward" + }, + { + "path": "experiments/k3/attnres_forward/results/raw/formal-uniform_groups_6_7_forward-seed-2026073003.json", + "seed": 2026073003, + "sha256": "ab3fd4d2afdc775b0ede6500f62376e904c72e26f3bec7732386218e51480fea", + "variant": "uniform_groups_6_7_forward" + } + ], + "manifest": { + "path": "experiments/k3/attnres_forward/manifest.json", + "sha256": "49546ed5baf36bcb30885062b7c671fafe4ccff2e606624cd7dbe23717f9a712" + }, + "references": [ + { + "path": "experiments/k3/attnres_gradient/results/raw/formal-depth-32-block-seed-2026073001.json", + "seed": 2026073001, + "sha256": "29e1d638b481619c7b32de402122523db8b17cc1fc67a8881fa1ba132a1d5c38" + }, + { + "path": "experiments/k3/attnres_gradient/results/raw/formal-depth-32-block-seed-2026073002.json", + "seed": 2026073002, + "sha256": "21199deb2199395061e51220fd8c7afd04a1135a6381e406da9b5795e3ad5032" + }, + { + "path": "experiments/k3/attnres_gradient/results/raw/formal-depth-32-block-seed-2026073003.json", + "seed": 2026073003, + "sha256": "c0d7f1bcfa9fa7f8f3134ca4571bdf23a951182d03b7de9611a6b4b89667d4ac" + } + ], + "replay": { + "path": "experiments/k3/attnres_forward/results/raw/replay-uniform_groups_6_7_forward-seed-2026073001.json", + "sha256": "b85ac8062b2b0b8b3f2305d4b22a7c0212466fbfb28a9d63099cadd0a6917c8e" + } + }, + "interaction": { + "cells": [ + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073001, + "step": 0 + }, + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073001, + "step": 0 + }, + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073002, + "step": 0 + }, + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073002, + "step": 0 + }, + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073003, + "step": 0 + }, + { + "interaction_residual": 0.0, + "log_effects": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.0, + "group7": 0.0, + "groups6_7": 0.0 + }, + "seed": 2026073003, + "step": 0 + }, + { + "interaction_residual": 9.315360079922494e-05, + "log_effects": { + "group6": -0.0020673620169685087, + "group7": -0.009910978729338916, + "groups6_7": -0.011885187145508201 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": -0.002069500483230477, + "group7": -0.009960255136960057, + "groups6_7": -0.011956096627949948 + }, + "seed": 2026073001, + "step": 100 + }, + { + "interaction_residual": -0.00015048504886481726, + "log_effects": { + "group6": 0.008204350063220466, + "group7": 0.00633861468044224, + "groups6_7": 0.014392479694797889 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.008170786235759111, + "group7": 0.006318568040747243, + "groups6_7": 0.01428940306090045 + }, + "seed": 2026073001, + "step": 100 + }, + { + "interaction_residual": -0.00022727724861329457, + "log_effects": { + "group6": -0.003024542543237395, + "group7": -0.010274063449398723, + "groups6_7": -0.013525883241249412 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": -0.003029121086871636, + "group7": -0.010327022853315526, + "groups6_7": -0.013617771823999139 + }, + "seed": 2026073002, + "step": 100 + }, + { + "interaction_residual": 0.00011391084988867078, + "log_effects": { + "group6": 0.008212012433931862, + "group7": 0.006225826243211901, + "groups6_7": 0.014551749527032434 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.008178385969761304, + "group7": 0.00620648594426831, + "groups6_7": 0.014446384520769564 + }, + "seed": 2026073002, + "step": 100 + }, + { + "interaction_residual": -0.00018594689551645893, + "log_effects": { + "group6": -0.0024512267787523503, + "group7": -0.010289094447499883, + "groups6_7": -0.012926268121768692 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": -0.002454233491322386, + "group7": -0.0103422091910102, + "groups6_7": -0.013010173463425477 + }, + "seed": 2026073003, + "step": 100 + }, + { + "interaction_residual": 0.00030197360149875237, + "log_effects": { + "group6": 0.007655924297294366, + "group7": 0.00598750540180943, + "groups6_7": 0.013945403300602548 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.007626692355618399, + "group7": 0.005969616013418889, + "groups6_7": 0.01384861659623099 + }, + "seed": 2026073003, + "step": 100 + }, + { + "interaction_residual": 0.029771830971208096, + "log_effects": { + "group6": 0.07485913168415881, + "group7": 0.015857159559656917, + "groups6_7": 0.12048812221502382 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.07212581480354684, + "group7": 0.01573209672471893, + "groups6_7": 0.11351238320785877 + }, + "seed": 2026073001, + "step": 500 + }, + { + "interaction_residual": -0.03192884846129315, + "log_effects": { + "group6": 0.020905879605847052, + "group7": 0.06104413401889223, + "groups6_7": 0.05002116516344613 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.020688866618218654, + "group7": 0.05921828161836901, + "groups6_7": 0.048790708212473444 + }, + "seed": 2026073001, + "step": 500 + }, + { + "interaction_residual": -0.1953811590475397, + "log_effects": { + "group6": 0.26705691975010704, + "group7": 0.1531184204309005, + "groups6_7": 0.22479418113346783 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.23437050921420513, + "group7": 0.14197189226332155, + "groups6_7": 0.20131941462327516 + }, + "seed": 2026073002, + "step": 500 + }, + { + "interaction_residual": 0.04533443185931518, + "log_effects": { + "group6": -0.05291484187072523, + "group7": -0.003412415847688265, + "groups6_7": -0.01099282585909831 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": -0.05433985568652999, + "group7": -0.0034182447669933603, + "groups6_7": -0.011053468978655284 + }, + "seed": 2026073002, + "step": 500 + }, + { + "interaction_residual": -0.05323030205979561, + "log_effects": { + "group6": 0.07126733637611304, + "group7": 0.002081326536511284, + "groups6_7": 0.020118360852828716 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.06878708823689964, + "group7": 0.002079162078343878, + "groups6_7": 0.01991733697848525 + }, + "seed": 2026073003, + "step": 500 + }, + { + "interaction_residual": 0.01258056327461619, + "log_effects": { + "group6": 0.03983966621344984, + "group7": 0.017361821564114133, + "groups6_7": 0.06978205105218016 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.03905650148868994, + "group7": 0.01721197360452348, + "groups6_7": 0.0674029436952744 + }, + "seed": 2026073003, + "step": 500 + }, + { + "interaction_residual": -0.3006404214639773, + "log_effects": { + "group6": 0.6920418427357188, + "group7": 0.6605562686145591, + "groups6_7": 1.0519576898863006 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.49944702553239, + "group7": 0.4834360937330096, + "groups6_7": 0.6507466503415881 + }, + "seed": 2026073001, + "step": 2000 + }, + { + "interaction_residual": -0.08939179391663415, + "log_effects": { + "group6": 0.35196456473519977, + "group7": 0.41393426496383595, + "groups6_7": 0.6765070357824016 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.2966949566647632, + "group7": 0.3389555963844964, + "groups6_7": 0.49161031844044 + }, + "seed": 2026073001, + "step": 2000 + }, + { + "interaction_residual": -0.4113553796722864, + "log_effects": { + "group6": 0.9164467937806046, + "group7": 0.7244113804086706, + "groups6_7": 1.2295027945169887 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.6000624198917696, + "group7": 0.5153902641956247, + "groups6_7": 0.7075620567115803 + }, + "seed": 2026073002, + "step": 2000 + }, + { + "interaction_residual": -0.5102556710901462, + "log_effects": { + "group6": 0.8512692533880561, + "group7": 0.472554150149919, + "groups6_7": 0.8135677324478289 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.5731272217637525, + "group7": 0.3765920444307726, + "groups6_7": 0.5567262404629003 + }, + "seed": 2026073002, + "step": 2000 + }, + { + "interaction_residual": -0.042537792021102805, + "log_effects": { + "group6": 0.3766095689864746, + "group7": 0.33907117265811954, + "groups6_7": 0.6731429496234913 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.3138160709099704, + "group7": 0.2875682583760594, + "groups6_7": 0.48989717177853653 + }, + "seed": 2026073003, + "step": 2000 + }, + { + "interaction_residual": -0.23802610783761186, + "log_effects": { + "group6": 0.1369711438469228, + "group7": 0.21929899602618774, + "groups6_7": 0.11824403203549867 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.1280046117985281, + "group7": 0.19691843594350228, + "groups6_7": 0.11152079123254297 + }, + "seed": 2026073003, + "step": 2000 + }, + { + "interaction_residual": -0.3567901087347324, + "log_effects": { + "group6": 0.7962036667755509, + "group7": 0.7974826538576361, + "groups6_7": 1.2368962118984546 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.5489619914090167, + "group7": 0.5495384944469774, + "groups6_7": 0.709716199433785 + }, + "seed": 2026073001, + "step": 4000 + }, + { + "interaction_residual": -0.025278590898171305, + "log_effects": { + "group6": 0.30819818474854643, + "group7": 0.5116588900282694, + "groups6_7": 0.7945784838786445 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.26523031656668117, + "group7": 0.40049975151541234, + "groups6_7": 0.5482283761836112 + }, + "seed": 2026073001, + "step": 4000 + }, + { + "interaction_residual": -0.4690849356804383, + "log_effects": { + "group6": 1.0154628434911215, + "group7": 0.8444211614256345, + "groups6_7": 1.3907990692363177 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.6377652669616457, + "group7": 0.570193925444063, + "groups6_7": 0.7511236442842244 + }, + "seed": 2026073002, + "step": 4000 + }, + { + "interaction_residual": -0.30335764563544154, + "log_effects": { + "group6": 0.7295561615678221, + "group7": 0.5114340535088001, + "groups6_7": 0.9376325694411807 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.517877072705919, + "group7": 0.4003649468121799, + "groups6_7": 0.6084462848212776 + }, + "seed": 2026073002, + "step": 4000 + }, + { + "interaction_residual": -0.11945708203112537, + "log_effects": { + "group6": 0.5220048555645129, + "group7": 0.46526410081521685, + "groups6_7": 0.8678118743486044 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.4066701858351324, + "group7": 0.3720307638996611, + "groups6_7": 0.5801307284410583 + }, + "seed": 2026073003, + "step": 4000 + }, + { + "interaction_residual": -0.2049100207501896, + "log_effects": { + "group6": 0.23807868703777696, + "group7": 0.278434203129335, + "groups6_7": 0.31160286941692233 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.21185932780108593, + "group7": 0.24303192775825497, + "groups6_7": 0.2677277217915467 + }, + "seed": 2026073003, + "step": 4000 + }, + { + "interaction_residual": -0.4263122295724949, + "log_effects": { + "group6": 0.8675369054663282, + "group7": 0.8883512858342638, + "groups6_7": 1.329575961728097 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.5800152615825866, + "group7": 0.5886666348497648, + "groups6_7": 0.7354105664383516 + }, + "seed": 2026073001, + "step": 8000 + }, + { + "interaction_residual": 0.004036643176398624, + "log_effects": { + "group6": 0.275967500869369, + "group7": 0.5274919212487371, + "groups6_7": 0.8074960652945047 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.24116240807746625, + "group7": 0.40991690985788126, + "groups6_7": 0.5540266425004673 + }, + "seed": 2026073001, + "step": 8000 + }, + { + "interaction_residual": -0.5154843690703684, + "log_effects": { + "group6": 1.07243611373865, + "group7": 0.9236817271683265, + "groups6_7": 1.480633471836608 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.6578260733512802, + "group7": 0.6029454996221533, + "groups6_7": 0.7725064680163986 + }, + "seed": 2026073002, + "step": 8000 + }, + { + "interaction_residual": -0.2777345501217605, + "log_effects": { + "group6": 0.6921079277686524, + "group7": 0.5533377840422664, + "groups6_7": 0.9677111616891583 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.4994801034992014, + "group7": 0.42497271322062063, + "groups6_7": 0.620048308391513 + }, + "seed": 2026073002, + "step": 8000 + }, + { + "interaction_residual": -0.15931730317353043, + "log_effects": { + "group6": 0.5911781397673622, + "group7": 0.5386433554809235, + "groups6_7": 0.9705041920747552 + }, + "metric": "spike_contrast", + "relative_drops": { + "group6": 0.4463254057279825, + "group7": 0.4164606288913148, + "groups6_7": 0.6211080443849011 + }, + "seed": 2026073003, + "step": 8000 + }, + { + "interaction_residual": -0.2376448513270391, + "log_effects": { + "group6": 0.2839896590577151, + "group7": 0.34439512255491495, + "groups6_7": 0.39073993028559095 + }, + "metric": "peak_normalized", + "relative_drops": { + "group6": 0.2472255709371281, + "group7": 0.29135113042743926, + "groups6_7": 0.3234439150908039 + }, + "seed": 2026073003, + "step": 8000 + } + ], + "definition": "I67=ln(Xref/X67)-ln(Xref/X6)-ln(Xref/X7)", + "interpretation": "descriptive cross-run log-attenuation residual from three independently trained variants; not a causal interaction", + "summaries": [ + { + "maximum": 0.0, + "mean_interaction_residual": 0.0, + "metric": "spike_contrast", + "minimum": 0.0, + "step": 0 + }, + { + "maximum": 0.0, + "mean_interaction_residual": 0.0, + "metric": "peak_normalized", + "minimum": 0.0, + "step": 0 + }, + { + "maximum": 9.315360079922494e-05, + "mean_interaction_residual": -0.00010669018111017618, + "metric": "spike_contrast", + "minimum": -0.00022727724861329457, + "step": 100 + }, + { + "maximum": 0.00030197360149875237, + "mean_interaction_residual": 8.846646750753529e-05, + "metric": "peak_normalized", + "minimum": -0.00015048504886481726, + "step": 100 + }, + { + "maximum": 0.029771830971208096, + "mean_interaction_residual": -0.07294654337870907, + "metric": "spike_contrast", + "minimum": -0.1953811590475397, + "step": 500 + }, + { + "maximum": 0.04533443185931518, + "mean_interaction_residual": 0.008662048890879407, + "metric": "peak_normalized", + "minimum": -0.03192884846129315, + "step": 500 + }, + { + "maximum": -0.042537792021102805, + "mean_interaction_residual": -0.25151119771912217, + "metric": "spike_contrast", + "minimum": -0.4113553796722864, + "step": 2000 + }, + { + "maximum": -0.08939179391663415, + "mean_interaction_residual": -0.2792245242814641, + "metric": "peak_normalized", + "minimum": -0.5102556710901462, + "step": 2000 + }, + { + "maximum": -0.11945708203112537, + "mean_interaction_residual": -0.315110708815432, + "metric": "spike_contrast", + "minimum": -0.4690849356804383, + "step": 4000 + }, + { + "maximum": -0.025278590898171305, + "mean_interaction_residual": -0.17784875242793416, + "metric": "peak_normalized", + "minimum": -0.30335764563544154, + "step": 4000 + }, + { + "maximum": -0.15931730317353043, + "mean_interaction_residual": -0.3670379672721313, + "metric": "spike_contrast", + "minimum": -0.5154843690703684, + "step": 8000 + }, + { + "maximum": 0.004036643176398624, + "mean_interaction_residual": -0.17044758609080035, + "metric": "peak_normalized", + "minimum": -0.2777345501217605, + "step": 8000 + } + ] + }, + "metadata_warnings": [], + "primary": { + "attenuation_passed": true, + "cells": [ + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.04626897714722, + "relative_drop": 0.7354105664383516, + "seed": 2026073001, + "variant": 0.8060105831398049 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.219066994145898, + "relative_drop": 0.5540266425004673, + "seed": 2026073001, + "variant": 1.4356181153951746 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 3.3328477735441013, + "relative_drop": 0.7725064680163986, + "seed": 2026073002, + "variant": 0.75820131156723 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 3.6189242296270336, + "relative_drop": 0.620048308391513, + "seed": 2026073002, + "variant": 1.3750163828497322 + }, + { + "metric": "spike_contrast", + "passed": true, + "reference": 1.8814916302334765, + "relative_drop": 0.6211080443849011, + "seed": 2026073003, + "variant": 0.7128820432526025 + }, + { + "metric": "peak_normalized", + "passed": true, + "reference": 2.2882673690329693, + "relative_drop": 0.3234439150908039, + "seed": 2026073003, + "variant": 1.5481412124184124 + } + ], + "material_response_passed": true, + "passed_cells": 6, + "quality": { + "mean_delta_bpc": 0.0065784582165467525, + "mean_maximum": 0.03, + "mean_passed": true, + "passed": true, + "passed_checks": 4, + "per_seed": { + "2026073001": { + "delta_bpc": 0.00569589341392529, + "passed": true, + "reference_bpc": 1.7123525940760473, + "variant_bpc": 1.7180484874899726 + }, + "2026073002": { + "delta_bpc": 0.009598287251159299, + "passed": true, + "reference_bpc": 1.7093240655856785, + "variant_bpc": 1.7189223528368378 + }, + "2026073003": { + "delta_bpc": 0.004441193984555669, + "passed": true, + "reference_bpc": 1.7030966812870931, + "variant_bpc": 1.7075378752716488 + } + }, + "per_seed_maximum": 0.05, + "required_checks": 4 + }, + "required_cells": 6, + "threshold": 0.2, + "variant": "uniform_groups_6_7_forward" + }, + "primary_step": 8000, + "processed_target_bytes": { + "formal_12_cells": 786432000, + "per_cell": 65536000, + "primary_replay": 65536000, + "total": 851968000 + }, + "protocol_id": "llm-atlas-k3-attnres-forward-training-v1", + "replay": { + "excluded": [ + "run_kind", + "timing", + "self hashes", + "manifest path strings", + "GPU/version metadata" + ], + "passed": true, + "scientific_payload_sha256": "b85563ca5cb53e60b39c3801d372376206105b8a089a8633b3e81973a7f0c051" + }, + "reporting_boundary": "C can change through spike-window numerator and the 27-layer reference denominator; layers 26-28 are intervened but belong to the denominator.", + "schema_version": 1, + "scope": "depth-32 reduced Block AttnRes train-time architecture ablation; not a real Kimi-K3 checkpoint result", + "secondary_status": { + "uniform_group_6_forward": "secondary_material_response", + "uniform_group_7_forward": "secondary_material_response", + "uniform_group_7_mlp_forward": "secondary_material_response" + }, + "spike_layers_1based": [ + 21, + 22, + 23, + 24, + 25 + ], + "status": "forward_training_attenuation_established_within_reduced_protocol", + "thresholds": { + "final_bpc_delta_mean_maximum": 0.03, + "final_bpc_delta_per_seed_maximum": 0.05, + "loss_scale_ratio_abs_error": 1e-05, + "loss_scale_shape_abs_error": 1e-06, + "material_relative_drop": 0.2, + "positive_denominator_epsilon": 1e-30, + "uniform_weight_max_abs_error": 1e-12 + }, + "trajectories": [ + { + "peak_normalized": 1.7717451736709768, + "reference_mean": 0.00015645541295978345, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.7816308178118705, + "spike_mean": 0.00012229037238284946, + "step": 0, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.145023601546889, + "reference_mean": 3.5063370281325966e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.5596113913680666, + "spike_mean": 1.962186142918654e-05, + "step": 100, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.1343785185339303, + "reference_mean": 7.428176730981149e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.5221542367874475, + "spike_mean": 3.8786539516877386e-05, + "step": 500, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.863607509132394, + "reference_mean": 4.200080675892187e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 2.4966108002124767, + "spike_mean": 0.00010485966777196154, + "step": 2000, + "variant": "learned_reference" + }, + { + "peak_normalized": 3.115049509935762, + "reference_mean": 3.6590978736596835e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 2.8510104941924466, + "spike_mean": 0.00010432126437081024, + "step": 4000, + "variant": "learned_reference" + }, + { + "peak_normalized": 3.219066994145898, + "reference_mean": 5.7697108584559625e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 3.04626897714722, + "spike_mean": 0.00017576091195223854, + "step": 8000, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.16920774571012, + "reference_mean": 0.0001952685916977417, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.6908033638721538, + "spike_mean": 0.0001348922000033781, + "step": 0, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.254829765809383, + "reference_mean": 3.313880080424025e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.5561530270868602, + "spike_mean": 1.8430244381306693e-05, + "step": 100, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.173061151786652, + "reference_mean": 7.33956448702547e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.5734017265344913, + "spike_mean": 4.2085189488716426e-05, + "step": 500, + "variant": "learned_reference" + }, + { + "peak_normalized": 3.189751492495938, + "reference_mean": 3.9160933603718013e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 2.740373880872814, + "spike_mean": 0.00010731559959822334, + "step": 2000, + "variant": "learned_reference" + }, + { + "peak_normalized": 3.420348943656972, + "reference_mean": 3.154654101914881e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 3.1572107254761765, + "spike_mean": 9.959907765733078e-05, + "step": 4000, + "variant": "learned_reference" + }, + { + "peak_normalized": 3.6189242296270336, + "reference_mean": 5.15456125598306e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 3.3328477735441013, + "spike_mean": 0.00017179368005599828, + "step": 8000, + "variant": "learned_reference" + }, + { + "peak_normalized": 1.7301666786506156, + "reference_mean": 0.0001660444721454513, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.7974048446155259, + "spike_mean": 0.0001324046665104106, + "step": 0, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.133789250541258, + "reference_mean": 3.45711278323843e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.5746026267549044, + "spike_mean": 1.98646608623676e-05, + "step": 100, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.195594858531836, + "reference_mean": 7.680312336122841e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.47779529155407807, + "spike_mean": 3.669617071864195e-05, + "step": 500, + "variant": "learned_reference" + }, + { + "peak_normalized": 1.8057756999530474, + "reference_mean": 5.286508732953937e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 1.4606649819879878, + "spike_mean": 7.721818183199502e-05, + "step": 2000, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.1130451955668876, + "reference_mean": 4.555695098436748e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 1.7734534826874866, + "spike_mean": 8.079313338384963e-05, + "step": 4000, + "variant": "learned_reference" + }, + { + "peak_normalized": 2.2882673690329693, + "reference_mean": 7.697225831819836e-05, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 1.8814916302334765, + "spike_mean": 0.0001448226597858593, + "step": 8000, + "variant": "learned_reference" + }, + { + "peak_normalized": 1.7717451736709768, + "reference_mean": 0.00015645541295978345, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.7816308178118705, + "spike_mean": 0.00012229037238284946, + "step": 0, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.127497072227991, + "reference_mean": 3.516218108685549e-05, + "relative_drop": { + "peak_normalized": 0.008170786235759111, + "spike_contrast": -0.002069500483230477 + }, + "seed": 2026073001, + "spike_contrast": 0.5607695074129241, + "spike_mean": 1.971787896763999e-05, + "step": 100, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.0902206460511907, + "reference_mean": 7.938710263665093e-05, + "relative_drop": { + "peak_normalized": 0.020688866618218654, + "spike_contrast": 0.07212581480354684 + }, + "seed": 2026073001, + "spike_contrast": 0.48449343700602876, + "spike_mean": 3.8462530210381376e-05, + "step": 500, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.013989603305468, + "reference_mean": 5.593089116262962e-05, + "relative_drop": { + "peak_normalized": 0.2966949566647632, + "spike_contrast": 0.49944702553239 + }, + "seed": 2026073001, + "spike_contrast": 1.2496859621343153, + "spike_mean": 6.989604953560047e-05, + "step": 2000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.2888439422946147, + "reference_mean": 5.321969008407366e-05, + "relative_drop": { + "peak_normalized": 0.26523031656668117, + "spike_contrast": 0.5489619914090167 + }, + "seed": 2026073001, + "spike_contrast": 1.2859140957725563, + "spike_mean": 6.843594965175726e-05, + "step": 4000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.442749046074982, + "reference_mean": 9.066082081214215e-05, + "relative_drop": { + "peak_normalized": 0.24116240807746625, + "spike_contrast": 0.5800152615825866 + }, + "seed": 2026073001, + "spike_contrast": 1.2793864795162566, + "spike_mean": 0.00011599022836890071, + "step": 8000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.16920774571012, + "reference_mean": 0.0001952685916977417, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.6908033638721538, + "spike_mean": 0.0001348922000033781, + "step": 0, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.2363888976884874, + "reference_mean": 3.3230999987507756e-05, + "relative_drop": { + "peak_normalized": 0.008178385969761304, + "spike_contrast": -0.003029121086871636 + }, + "seed": 2026073002, + "spike_contrast": 0.5578376819487365, + "spike_mean": 1.853750400186982e-05, + "step": 100, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.291144981172743, + "reference_mean": 7.969680357038871e-05, + "relative_drop": { + "peak_normalized": -0.05433985568652999, + "spike_contrast": 0.23437050921420513 + }, + "seed": 2026073002, + "spike_contrast": 0.43901327190229816, + "spike_mean": 3.498795449559111e-05, + "step": 500, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.3616180814849579, + "reference_mean": 6.0796766370509024e-05, + "relative_drop": { + "peak_normalized": 0.5731272217637525, + "spike_contrast": 0.6000624198917696 + }, + "seed": 2026073002, + "spike_contrast": 1.0959784985080736, + "spike_mean": 6.663194872089662e-05, + "step": 2000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.649028645083117, + "reference_mean": 5.410763259095364e-05, + "relative_drop": { + "peak_normalized": 0.517877072705919, + "spike_contrast": 0.6377652669616457 + }, + "seed": 2026073002, + "spike_contrast": 1.1436513842886917, + "spike_mean": 6.188026891322807e-05, + "step": 4000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.8113435808571552, + "reference_mean": 9.7101816613253e-05, + "relative_drop": { + "peak_normalized": 0.4994801034992014, + "spike_contrast": 0.6578260733512802 + }, + "seed": 2026073002, + "spike_contrast": 1.1404136095960284, + "spike_mean": 0.00011073623318225145, + "step": 8000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.7301666786506156, + "reference_mean": 0.0001660444721454513, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.7974048446155259, + "spike_mean": 0.0001324046665104106, + "step": 0, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.1175154963756544, + "reference_mean": 3.463190794853425e-05, + "relative_drop": { + "peak_normalized": 0.007626692355618399, + "spike_contrast": -0.002454233491322386 + }, + "seed": 2026073003, + "spike_contrast": 0.5760128357656881, + "spike_mean": 1.9948423505411485e-05, + "step": 100, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 2.1098426046710275, + "reference_mean": 8.283220328545819e-05, + "relative_drop": { + "peak_normalized": 0.03905650148868994, + "spike_contrast": 0.06878708823689964 + }, + "seed": 2026073003, + "spike_contrast": 0.4449291446747725, + "spike_mean": 3.6854461359325794e-05, + "step": 500, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.5746280824853423, + "reference_mean": 6.005889426214034e-05, + "relative_drop": { + "peak_normalized": 0.1280046117985281, + "spike_contrast": 0.3138160709099704 + }, + "seed": 2026073003, + "spike_contrast": 1.0022848364247348, + "spike_mean": 6.0196119011379776e-05, + "step": 2000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.6653768608207726, + "reference_mean": 5.561273489244437e-05, + "relative_drop": { + "peak_normalized": 0.21185932780108593, + "spike_contrast": 0.4066701858351324 + }, + "seed": 2026073003, + "spike_contrast": 1.0522428253130036, + "spike_mean": 5.851810128660872e-05, + "step": 4000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.7225491622669935, + "reference_mean": 9.853165789132213e-05, + "relative_drop": { + "peak_normalized": 0.2472255709371281, + "spike_contrast": 0.4463254057279825 + }, + "seed": 2026073003, + "spike_contrast": 1.0417341149957169, + "spike_mean": 0.0001026437894324772, + "step": 8000, + "variant": "uniform_group_6_forward" + }, + { + "peak_normalized": 1.7717451736709768, + "reference_mean": 0.00015645541295978345, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.7816308178118705, + "spike_mean": 0.00012229037238284946, + "step": 0, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.131470123971506, + "reference_mean": 3.5108908681043736e-05, + "relative_drop": { + "peak_normalized": 0.006318568040747243, + "spike_contrast": -0.009960255136960057 + }, + "seed": 2026073001, + "spike_contrast": 0.5651852636036417, + "spike_mean": 1.984303780773189e-05, + "step": 100, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.007984290343191, + "reference_mean": 7.690677624918021e-05, + "relative_drop": { + "peak_normalized": 0.05921828161836901, + "spike_contrast": 0.01573209672471893 + }, + "seed": 2026073001, + "spike_contrast": 0.5139396558290856, + "spike_mean": 3.952544211642817e-05, + "step": 500, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.8929717180633012, + "reference_mean": 6.043750208509327e-05, + "relative_drop": { + "peak_normalized": 0.3389555963844964, + "spike_contrast": 0.4834360937330096 + }, + "seed": 2026073001, + "spike_contrast": 1.2896590273861137, + "spike_mean": 7.794377015670761e-05, + "step": 2000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.8674729552482823, + "reference_mean": 5.6604490695193546e-05, + "relative_drop": { + "peak_normalized": 0.40049975151541234, + "spike_contrast": 0.5495384944469774 + }, + "seed": 2026073001, + "spike_contrast": 1.2842704795613964, + "spike_mean": 7.269547641044482e-05, + "step": 4000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.899516999280113, + "reference_mean": 9.566590583913408e-05, + "relative_drop": { + "peak_normalized": 0.40991690985788126, + "spike_contrast": 0.5886666348497648 + }, + "seed": 2026073001, + "spike_contrast": 1.2530320695227308, + "spike_mean": 0.00011987244797637687, + "step": 8000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.16920774571012, + "reference_mean": 0.0001952685916977417, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.6908033638721538, + "spike_mean": 0.0001348922000033781, + "step": 0, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.2408351965611693, + "reference_mean": 3.318128242595035e-05, + "relative_drop": { + "peak_normalized": 0.00620648594426831, + "spike_contrast": -0.010327022853315526 + }, + "seed": 2026073002, + "spike_contrast": 0.5618964321075268, + "spike_mean": 1.8644444207893684e-05, + "step": 100, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.180489206697103, + "reference_mean": 7.674864826657102e-05, + "relative_drop": { + "peak_normalized": -0.0034182447669933603, + "spike_contrast": 0.14197189226332155 + }, + "seed": 2026073002, + "spike_contrast": 0.4919947983913339, + "spike_mean": 3.7759935730719006e-05, + "step": 500, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.9885164567107845, + "reference_mean": 5.209253834403137e-05, + "relative_drop": { + "peak_normalized": 0.3765920444307726, + "spike_contrast": 0.5153902641956247 + }, + "seed": 2026073002, + "spike_contrast": 1.328011862414985, + "spike_mean": 6.917950886418112e-05, + "step": 2000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.0509611207506526, + "reference_mean": 5.1951786628234445e-05, + "relative_drop": { + "peak_normalized": 0.4003649468121799, + "spike_contrast": 0.570193925444063 + }, + "seed": 2026073002, + "spike_contrast": 1.3569883484628174, + "spike_mean": 7.049796913634054e-05, + "step": 4000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.0809801808225887, + "reference_mean": 8.834915543384677e-05, + "relative_drop": { + "peak_normalized": 0.42497271322062063, + "spike_contrast": 0.6029454996221533 + }, + "seed": 2026073002, + "spike_contrast": 1.323322207559972, + "spike_mean": 0.00011691439940477721, + "step": 8000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.7301666786506156, + "reference_mean": 0.0001660444721454513, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.7974048446155259, + "spike_mean": 0.0001324046665104106, + "step": 0, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.121051348061966, + "reference_mean": 3.459093777227854e-05, + "relative_drop": { + "peak_normalized": 0.005969616013418889, + "spike_contrast": -0.0103422091910102 + }, + "seed": 2026073003, + "spike_contrast": 0.5805452873225075, + "spike_mean": 2.0081605907762422e-05, + "step": 100, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 2.1578043377805587, + "reference_mean": 7.838830411115109e-05, + "relative_drop": { + "peak_normalized": 0.01721197360452348, + "spike_contrast": 0.002079162078343878 + }, + "seed": 2026073003, + "spike_contrast": 0.4768018777026676, + "spike_mean": 3.7375690590124576e-05, + "step": 500, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.4501851734535103, + "reference_mean": 6.012620542605873e-05, + "relative_drop": { + "peak_normalized": 0.19691843594350228, + "spike_contrast": 0.2875682583760594 + }, + "seed": 2026073003, + "spike_contrast": 1.040624097046804, + "spike_mean": 6.256877823034302e-05, + "step": 2000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.599507748247948, + "reference_mean": 5.5299108242723846e-05, + "relative_drop": { + "peak_normalized": 0.24303192775825497, + "spike_contrast": 0.3720307638996611 + }, + "seed": 2026073003, + "spike_contrast": 1.1136742287827466, + "spike_mean": 6.15851917245891e-05, + "step": 4000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.6215780843449914, + "reference_mean": 9.798803714349763e-05, + "relative_drop": { + "peak_normalized": 0.29135113042743926, + "spike_contrast": 0.4164606288913148 + }, + "seed": 2026073003, + "spike_contrast": 1.0979244426526977, + "spike_mean": 0.00010758346106740647, + "step": 8000, + "variant": "uniform_group_7_forward" + }, + { + "peak_normalized": 1.7717451736709768, + "reference_mean": 0.00015645541295978345, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.7816308178118705, + "spike_mean": 0.00012229037238284946, + "step": 0, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.114372494729241, + "reference_mean": 3.5219062098163975e-05, + "relative_drop": { + "peak_normalized": 0.01428940306090045, + "spike_contrast": -0.011956096627949948 + }, + "seed": 2026073001, + "spike_contrast": 0.5663021592373647, + "spike_mean": 1.994463091250509e-05, + "step": 100, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.03024067902117, + "reference_mean": 7.452956594567729e-05, + "relative_drop": { + "peak_normalized": 0.048790708212473444, + "spike_contrast": 0.11351238320785877 + }, + "seed": 2026073001, + "spike_contrast": 0.4628832649676238, + "spike_mean": 3.449848882155493e-05, + "step": 500, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.4558285096793826, + "reference_mean": 6.297526043555181e-05, + "relative_drop": { + "peak_normalized": 0.49161031844044, + "spike_contrast": 0.6507466503415881 + }, + "seed": 2026073001, + "spike_contrast": 0.8719496847675758, + "spike_mean": 5.491125848493539e-05, + "step": 2000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.4072909753721254, + "reference_mean": 6.464099647861008e-05, + "relative_drop": { + "peak_normalized": 0.5482283761836112, + "spike_contrast": 0.709716199433785 + }, + "seed": 2026073001, + "spike_contrast": 0.8276021617083458, + "spike_mean": 5.349702842067927e-05, + "step": 4000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.4356181153951746, + "reference_mean": 0.0001126591938642647, + "relative_drop": { + "peak_normalized": 0.5540266425004673, + "spike_contrast": 0.7354105664383516 + }, + "seed": 2026073001, + "spike_contrast": 0.8060105831398049, + "spike_mean": 9.080450254259632e-05, + "step": 8000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.16920774571012, + "reference_mean": 0.0001952685916977417, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.6908033638721538, + "spike_mean": 0.0001348922000033781, + "step": 0, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.222255627983624, + "reference_mean": 3.3272086107495445e-05, + "relative_drop": { + "peak_normalized": 0.014446384520769564, + "spike_contrast": -0.013617771823999139 + }, + "seed": 2026073002, + "spike_contrast": 0.5637265921089555, + "spike_mean": 1.875635971373413e-05, + "step": 100, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.1970810158166465, + "reference_mean": 7.526881664590392e-05, + "relative_drop": { + "peak_normalized": -0.011053468978655284, + "spike_contrast": 0.20131941462327516 + }, + "seed": 2026073002, + "spike_contrast": 0.4579648266045922, + "spike_mean": 3.4470470563974234e-05, + "step": 500, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.4139331360677492, + "reference_mean": 6.686618683118097e-05, + "relative_drop": { + "peak_normalized": 0.5567262404629003, + "spike_contrast": 0.7075620567115803 + }, + "seed": 2026073002, + "spike_contrast": 0.8013893015637505, + "spike_mean": 5.358584676287137e-05, + "step": 2000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.339250336096506, + "reference_mean": 6.513453505855675e-05, + "relative_drop": { + "peak_normalized": 0.6084462848212776, + "spike_contrast": 0.7511236442842244 + }, + "seed": 2026073002, + "spike_contrast": 0.785755099583271, + "spike_mean": 5.117979308124632e-05, + "step": 4000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.3750163828497322, + "reference_mean": 0.00011196851617704076, + "relative_drop": { + "peak_normalized": 0.620048308391513, + "spike_contrast": 0.7725064680163986 + }, + "seed": 2026073002, + "spike_contrast": 0.75820131156723, + "spike_mean": 8.489467581966892e-05, + "step": 8000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.7301666786506156, + "reference_mean": 0.0001660444721454513, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.7974048446155259, + "spike_mean": 0.0001324046665104106, + "step": 0, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.104239221313353, + "reference_mean": 3.465675830806571e-05, + "relative_drop": { + "peak_normalized": 0.01384861659623099, + "spike_contrast": -0.013010173463425477 + }, + "seed": 2026073003, + "spike_contrast": 0.5820783066015256, + "spike_mean": 2.017294718825724e-05, + "step": 100, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 2.047605301904581, + "reference_mean": 8.027602099766093e-05, + "relative_drop": { + "peak_normalized": 0.0674029436952744, + "spike_contrast": 0.01991733697848525 + }, + "seed": 2026073003, + "spike_contrast": 0.4682788817254619, + "spike_mean": 3.7591565342154357e-05, + "step": 500, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.6043941651057845, + "reference_mean": 6.416649369014151e-05, + "relative_drop": { + "peak_normalized": 0.11152079123254297, + "spike_contrast": 0.48989717177853653 + }, + "seed": 2026073003, + "spike_contrast": 0.7450893383961256, + "spike_mean": 4.7809770330786704e-05, + "step": 2000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.5473244193151916, + "reference_mean": 6.484813436852871e-05, + "relative_drop": { + "peak_normalized": 0.2677277217915467, + "spike_contrast": 0.5801307284410583 + }, + "seed": 2026073003, + "spike_contrast": 0.7446186219196633, + "spike_mean": 4.828712844755501e-05, + "step": 4000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.5481412124184124, + "reference_mean": 0.00011760102872223663, + "relative_drop": { + "peak_normalized": 0.3234439150908039, + "spike_contrast": 0.6211080443849011 + }, + "seed": 2026073003, + "spike_contrast": 0.7128820432526025, + "spike_mean": 8.383566164411604e-05, + "step": 8000, + "variant": "uniform_groups_6_7_forward" + }, + { + "peak_normalized": 1.7717451736709768, + "reference_mean": 0.00015645541295978345, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073001, + "spike_contrast": 0.7816308178118705, + "spike_mean": 0.00012229037238284946, + "step": 0, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.1313742213368374, + "reference_mean": 3.5109654803549075e-05, + "relative_drop": { + "peak_normalized": 0.0063632774018002325, + "spike_contrast": -0.009908666739421946 + }, + "seed": 2026073001, + "spike_contrast": 0.565156394148717, + "spike_mean": 1.9842445908579975e-05, + "step": 100, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.074096058122276, + "reference_mean": 7.514289326435473e-05, + "relative_drop": { + "peak_normalized": 0.02824356593181118, + "spike_contrast": 0.002103655349041084 + }, + "seed": 2026073001, + "spike_contrast": 0.5210558042342052, + "spike_mean": 3.9153640682343396e-05, + "step": 500, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.008182813956302, + "reference_mean": 5.4975213128340397e-05, + "relative_drop": { + "peak_normalized": 0.2987227448063459, + "spike_contrast": 0.44974147877714904 + }, + "seed": 2026073001, + "spike_contrast": 1.373781366993916, + "spike_mean": 7.552392344223335e-05, + "step": 2000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.006420683947552, + "reference_mean": 5.4418700596283156e-05, + "relative_drop": { + "peak_normalized": 0.35589444805038495, + "spike_contrast": 0.5194674071926733 + }, + "seed": 2026073001, + "spike_contrast": 1.3700034648951942, + "spike_mean": 7.45538083720021e-05, + "step": 4000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.0181972534006216, + "reference_mean": 9.182359806598267e-05, + "relative_drop": { + "peak_normalized": 0.37304900548175707, + "spike_contrast": 0.5659319887472724 + }, + "seed": 2026073001, + "spike_contrast": 1.3222879166511747, + "spike_mean": 0.00012141723418608308, + "step": 8000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.16920774571012, + "reference_mean": 0.0001952685916977417, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073002, + "spike_contrast": 0.6908033638721538, + "spike_mean": 0.0001348922000033781, + "step": 0, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.24083237123207, + "reference_mean": 3.3190639981141967e-05, + "relative_drop": { + "peak_normalized": 0.006207738956421139, + "spike_contrast": -0.010320676434344202 + }, + "seed": 2026073002, + "spike_contrast": 0.5618929025274048, + "spike_mean": 1.8649585035745987e-05, + "step": 100, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.2055180055903696, + "reference_mean": 7.56468436722773e-05, + "relative_drop": { + "peak_normalized": -0.014936005724934306, + "spike_contrast": 0.15494331212416607 + }, + "seed": 2026073002, + "spike_contrast": 0.4845569638475219, + "spike_mean": 3.6655204894486815e-05, + "step": 500, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.1937881935672334, + "reference_mean": 5.264968812870966e-05, + "relative_drop": { + "peak_normalized": 0.3122385243087939, + "spike_contrast": 0.46581551256749454 + }, + "seed": 2026073002, + "spike_contrast": 1.46386521692747, + "spike_mean": 7.70720471336972e-05, + "step": 2000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.2271117803789005, + "reference_mean": 5.202572019361132e-05, + "relative_drop": { + "peak_normalized": 0.348864160626338, + "spike_contrast": 0.5229153387055521 + }, + "seed": 2026073002, + "spike_contrast": 1.5062568095989999, + "spike_mean": 7.836409531591926e-05, + "step": 4000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.1987132665333755, + "reference_mean": 8.721922414325591e-05, + "relative_drop": { + "peak_normalized": 0.3924400935136531, + "spike_contrast": 0.5738819822015593 + }, + "seed": 2026073002, + "spike_contrast": 1.420186486886559, + "spike_mean": 0.00012386756352498197, + "step": 8000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 1.7301666786506156, + "reference_mean": 0.0001660444721454513, + "relative_drop": { + "peak_normalized": 0.0, + "spike_contrast": 0.0 + }, + "seed": 2026073003, + "spike_contrast": 0.7974048446155259, + "spike_mean": 0.0001324046665104106, + "step": 0, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.1207412846841187, + "reference_mean": 3.459105876875886e-05, + "relative_drop": { + "peak_normalized": 0.006114927167165012, + "spike_contrast": -0.010453253577363798 + }, + "seed": 2026073003, + "spike_contrast": 0.5806090937185927, + "spike_mean": 2.0083883282495662e-05, + "step": 100, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 2.053846006906672, + "reference_mean": 7.881468479387911e-05, + "relative_drop": { + "peak_normalized": 0.064560568209724, + "spike_contrast": 0.01034983598448287 + }, + "seed": 2026073003, + "spike_contrast": 0.4728501886523352, + "spike_mean": 3.726753857336007e-05, + "step": 500, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 1.4998578508974612, + "reference_mean": 5.674074310701582e-05, + "relative_drop": { + "peak_normalized": 0.16941076849330763, + "spike_contrast": 0.26928770181550915 + }, + "seed": 2026073003, + "spike_contrast": 1.0673258658660505, + "spike_mean": 6.056086276657879e-05, + "step": 2000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 1.639948309138832, + "reference_mean": 5.3547755736939575e-05, + "relative_drop": { + "peak_normalized": 0.2238934062653275, + "spike_contrast": 0.3481531423133718 + }, + "seed": 2026073003, + "spike_contrast": 1.1560200799432452, + "spike_mean": 6.190228086779825e-05, + "step": 4000, + "variant": "uniform_group_7_mlp_forward" + }, + { + "peak_normalized": 1.661826944810198, + "reference_mean": 9.516139861834408e-05, + "relative_drop": { + "peak_normalized": 0.27376190068536765, + "spike_contrast": 0.39856400381461277 + }, + "seed": 2026073003, + "spike_contrast": 1.1315967929439392, + "spike_mean": 0.00010768433348857797, + "step": 8000, + "variant": "uniform_group_7_mlp_forward" + } + ] +} diff --git a/src/pages/index.astro b/src/pages/index.astro index 88c9ac7..3986db7 100644 --- a/src/pages/index.astro +++ b/src/pages/index.astro @@ -128,20 +128,20 @@ const paths = [
-

NEW / K3 ROUND 07 LOCAL MIXER PATH · BIDIRECTIONAL GATE

-

16 个局部 mixer 单侧证据很强，但双向定位仍然没有闭合

+

NEW / K3 ROUND 08 TRAIN-TIME FORWARD · PREREGISTERED GATE

+

局部 uniform routing 进入完整训练后，六个尖峰指标格全部衰减

- 以 14 个冻结 mask 同时检查 sufficiency 与 restoration：groups 6+7 在 learned - 背景的 6 / 6 格全部超过 50% global log gap；但从 uniform 背景恢复时，peak 三个 - seed 全部未过线。因此只能报告 one-sided evidence，不能宣布尖峰已定位到这 16 个 mixer。 + 四个前向架构变体各跑三个 8,000-step seed，再完整 replay 主格：groups 6+7 的 + contrast / peak 六格降幅为 32.3%–77.3%，BPC 质量门 4 / 4 通过。结论只限缩小 + depth-32 Block 协议，不是真实 K3 checkpoint 或 Figure 5(c) 复现。

-
MATRIX
14 masks × 3 seeds
-
SUFFICIENCY
6 / 6 pass
-
RESTORATION
3 / 6 fail
+
MATRIX
12 formal + 1 replay
+
ATTENUATION
6 / 6 pass
+
QUALITY
4 / 4 pass
- +
diff --git a/src/pages/k3/index.astro b/src/pages/k3/index.astro index 180ef87..9578311 100644 --- a/src/pages/k3/index.astro +++ b/src/pages/k3/index.astro @@ -2,6 +2,7 @@ import BaseLayout from "@/layouts/BaseLayout.astro"; import ArchitectureExplorer from "@/components/ArchitectureExplorer.astro"; import K3ArtifactLab from "@/components/K3ArtifactLab.astro"; +import K3AttnResForwardLab from "@/components/K3AttnResForwardLab.astro"; import K3AttnResGradientLab from "@/components/K3AttnResGradientLab.astro"; import K3AttnResLocalPathLab from "@/components/K3AttnResLocalPathLab.astro"; import K3AttnResSpikeLab from "@/components/K3AttnResSpikeLab.astro"; @@ -44,7 +45,8 @@ const toc = [ ["31", "attnres-gradient", "梯度定义与深度扩展"], ["32", "attnres-spike", "尖峰轨迹与反向路径"], ["33", "attnres-local-path", "局部 mixer 双向干预"], - ["34", "audit", "21 张图表审计"], + ["34", "attnres-forward", "训练期前向干预"], + ["35", "audit", "21 张图表审计"], ["↳", "papers", "100 节点阅读链"], ]; @@ -113,13 +115,13 @@ const paperGroups = [
-

ANCHOR REPORT / ROUND 07 KIMI K3 · REPORT → ARTIFACTS → INDEPENDENT PROBE

+

ANCHOR REPORT / ROUND 08 KIMI K3 · REPORT → ARTIFACTS → INDEPENDENT PROBE

不把报告压成摘要
把每个因果环节
重新展开

K3 同时扩展序列、深度、宽度、视觉与 Agent 轨迹。真正值得读的不是 2.8T 这个最大数字， @@ -129,11 +131,11 @@ const paperGroups = [

QUESTIONS
32 张问题账
REPORT
16 Figures · 5 Tables
-
LABS
8 + 4 + 5 + 5 + 5 个交互视图
+
LABS
8 + 4 + 5 + 5 + 5 + 5 个交互视图
READING
100 个一手 / 官方节点
MODEL
2.78T total / 104.2B active
ARTIFACTS
96 shards · 497,220 tensors
-
STATUS
K3 七轮 · 局部路径审计
+
STATUS
K3 八轮 · 训练期消融审计
@@ -977,8 +979,33 @@ const paperGroups = [
+
+

34 TRAIN-TIME FORWARD INTERVENTION

+

诊断期的局部敏感性进入完整训练后，尖峰指标还会衰减吗？

+

+ 第八轮不再只改 diagnostic backward，而是让 group 6 / 7 的 parameter-free + uniform mixer 进入每一次 train、eval 与 diagnostic forward。四个变体各跑三个 + 8,000-step seed，并把同 seed 的 Round 05 learned run 锁为历史配对 reference； + 主变体 groups 6+7 的 contrast 与 peak 六格降幅全部超过 20%，同时 BPC 质量门 + 4 / 4 通过，指定 seed 的完整重训 scientific payload exact。 +

+
+
F / FROZEN12 formal + 1 replay

851,968,000 个新 target bytes；每格 65,536,000。

+
X / ATTENUATION6 / 6 PASS

contrast drop 62.1%–77.3%；peak drop 32.3%–62.0%。

+
X / QUALITY4 / 4 PASS

三 seed ΔBPC 最大 +0.00960；均值 +0.00658。

+
B / BOUNDARYreduced protocol only

训练期架构消融；不是 K3 checkpoint 或 Figure 5(c) 复现。

+
+ +
+ 阅读完整结果审计 + 核对预注册协议 + 查看两阶段实现审阅 + 复跑训练、分析与 replay +
+
+
-

34 FIGURE & TABLE AUDIT

+

35 FIGURE & TABLE AUDIT

Figure 1–16、Table 1–5：每张图究竟支持什么，不能支持什么

{k3FigureAtlas.map(([id, report, title, contract]) => ( diff --git a/src/pages/progress/index.astro b/src/pages/progress/index.astro index 7dacc77..fc740a6 100644 --- a/src/pages/progress/index.astro +++ b/src/pages/progress/index.astro @@ -97,7 +97,7 @@ const workstreams = [
✓

K3 报告已结构化拆解

47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。

✓

17 专题知识图

从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。

✓

编辑式网站系统

响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。

-
✓

一百零九个原创交互视图

K3 三轴图、八联报告实验、四联开放工件实验与四轮二十联 AttnRes 独立实验，DeepSeek 四联公式实验、十三联 Base 工件实验、Chat 行为、completion/full-depth、multi-seed、cross-source 与 task-bootstrap CRN 五轮实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。

+
✓

一百一十四个原创交互视图

K3 三轴图、八联报告实验、四联开放工件实验与五轮二十五联 AttnRes 独立实验，DeepSeek 四联公式实验、十三联 Base 工件实验、Chat 行为、completion/full-depth、multi-seed、cross-source 与 task-bootstrap CRN 五轮实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。

✓

十七篇首版长文

K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。

✓

语言模型前史深度专题

八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。

✓

Transformer 深度专题

十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。

@@ -109,6 +109,7 @@ const workstreams = [
✓

Kimi K3 五轮梯度定义与深度扩展

先确认 Figure 5 没有公开唯一 gradient telemetry 合同，再冻结 16/32 blocks × Baseline/Block × 3 seeds 的 12 个 8,000-step 格。Block 的首尾失衡 6/6 改善但全层 CV 6/6 恶化，两个深度都判为 mixed；指定 32 层格完整重训的模型、优化器与全部冻结字段 exact。

✓

Kimi K3 六轮尖峰轨迹与反向路径

严格复用 Round 05 depth-32 Block 的三个正式格：尖峰在 step 500 后形成，六个位置 3/3 seed 可见，四种 reduction 12/12 格稳健。切断 key/softmax 源梯度没有降低尖峰；uniform value-backward 让 contrast 平均下降 70.2%，只判为全局 backward-rule sensitivity。完整 replay 的 16 组冻结字段 exact。

✓

Kimi K3 七轮局部路径双向审计

冻结 14 个 same-forward mask，把 groups 6+7 的 16 个 depth mixers 同时放进 sufficiency 与 restoration 两个方向。充分性 6/6 过 50%，恢复性却只有 contrast 3/3 通过、peak 0/3 通过，因此正式状态为 one-sided evidence / localization not established。三个正式格、完整 replay、selector 与 forward identity 全部 exact。

+
✓

Kimi K3 八轮训练期前向干预

四个 forward architecture variants × 三 seed × 8,000 steps，加一格完整 replay；groups 6+7 的 contrast / peak 六格降幅全部超过 20%，BPC 质量门 4/4 通过。13 个 raw、自哈希、historical pairing 与 scientific replay 全部过闸；Grok 结果后复算 blocking error 为 0。结论只限固定缩小协议，不是真实 K3 checkpoint 或 Figure 5(c) 复现。

✓

FlashKDA RTX 5090 执行闸门

隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。

✓

Scaling Laws 深度专题

九张账、29 个一手节点、DeepSeek/Kimi 双谱系与曲面—部署—复用—涌现四联实验。

✓

数据工程深度专题

十二张账、31 个一手节点、DeepSeek/Kimi 双谱系与流水线—去重—混合—改写四联实验。

@@ -137,7 +138,6 @@ const workstreams = [
优先级专题本轮交付完成闸门
-
P0K3 七轮后续

前向训练变体 → 非加性局部交互地图 → 等待 A_log 社区候选的官方裁决后进入真实 checkpoint forward

局部机制 + 工件边界
P0DeepSeek 八轮后续

干预式 mediation → SM90 FlashMLA / FP8 / pipeline traces → R1-like RL 小模型复现

运行证据 + 独立复现
P0Transformer 二轮

多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照

逐图笔记 + 实测边界
P0表示、位置与残差二轮

真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融

可复现实验 + 逐图笔记