wuyang/llm-atlas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

research: audit reduced AttnRes study

Browse Source
This commit is contained in:
wuyang
2026-07-30 07:15:05 +08:00
parent 5f49906be6
commit d1d9d22bf3
6 changed files with 25838 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
experiments/k3/attnres/README.md
+21
View File
@@ -41,3 +41,24 @@ python experiments/k3/attnres/train.py \
Raw parquet and checkpoints stay in the local cache. Frozen manifests, metric Raw parquet and checkpoints stay in the local cache. Frozen manifests, metric
JSON, analyses, code, checksums, and a compact website payload enter the public JSON, analyses, code, checksums, and a compact website payload enter the public
repository. repository.
## Validate and aggregate the complete study
After the nine formal cells, the preregistered replay, and the paired smoke runs
exist in the cache:
```bash
python experiments/k3/attnres/analyze.py \
--formal-dir /home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1/formal \
--smoke-dir /home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1/smoke \
--replay /home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1/replay/block-2026073001.json \
--manifest experiments/k3/attnres/manifest.json \
--output src/data/k3-attnres-reduced.json \
--compact-output src/data/k3-attnres-reduced-compact.json \
--reproduction-output experiments/k3/attnres/reproduction.json
```
The aggregator fails closed on protocol identity, grid completeness, byte
budget, schedule hashes, shared initialization, non-finite metrics, diagnostic
shape, smoke mismatch, or formal replay mismatch. Timing is recorded but is not
required to replay bit-for-bit.
experiments/k3/attnres/analyze.py
+536
View File
@@ -0,0 +1,536 @@
#!/usr/bin/env python3
"""Validate, aggregate, and compact the reduced Attention Residuals study."""
from __future__ import annotations
import argparse
import hashlib
import json
import math
import os
import statistics
from pathlib import Path
from typing import Any, Iterable
PROTOCOL_ID = "llm-atlas-k3-attnres-reduced-v1"
ARCHITECTURES = ("baseline", "full", "block")
SEEDS = (2026073001, 2026073002, 2026073003)
REPLAY_FIELDS = (
"manifest",
"model",
"optimizer",
"hashes",
"evaluations",
"training_history",
"diagnostic",
"environment",
)
SMOKE_FIELDS = REPLAY_FIELDS
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument("--formal-dir", type=Path, required=True)
parser.add_argument("--smoke-dir", type=Path, required=True)
parser.add_argument("--replay", type=Path, required=True)
parser.add_argument("--manifest", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--compact-output", type=Path, required=True)
parser.add_argument("--reproduction-output", type=Path, required=True)
return parser.parse_args()
def read_json(path: Path) -> dict[str, Any]:
return json.loads(path.read_text())
def file_sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for block in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(block)
return digest.hexdigest()
def canonical_sha256(value: Any) -> str:
payload = json.dumps(
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
).encode()
return hashlib.sha256(payload).hexdigest()
def write_json(path: Path, value: dict[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_suffix(path.suffix + ".tmp")
temporary.write_text(
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
)
os.replace(temporary, path)
def mean(values: Iterable[float]) -> float:
return statistics.fmean(values)
def elementwise_summary(rows: list[list[float]]) -> dict[str, list[float]]:
length = len(rows[0])
if any(len(row) != length for row in rows):
raise ValueError("array lengths do not match")
return {
"mean": [mean(row[index] for row in rows) for index in range(length)],
"min": [min(row[index] for row in rows) for index in range(length)],
"max": [max(row[index] for row in rows) for index in range(length)],
}
def coefficient_of_variation(values: list[float]) -> float:
average = mean(values)
variance = mean((value - average) ** 2 for value in values)
return math.sqrt(variance) / average
def verdict(deltas: list[float]) -> dict[str, Any]:
average = mean(deltas)
if all(delta < 0 for delta in deltas) and average <= -0.010:
label = "directional support in this reduced protocol"
elif all(delta > 0 for delta in deltas) and average >= 0.010:
label = "directional concern in this reduced protocol"
else:
label = "inconclusive at this budget"
return {
"paired_deltas_bpc": deltas,
"mean_delta_bpc": average,
"min_delta_bpc": min(deltas),
"max_delta_bpc": max(deltas),
"same_direction": len({delta < 0 for delta in deltas}) == 1,
"threshold_bpc": 0.010,
"verdict": label,
}
def average_depth_weights(runs: list[dict[str, Any]]) -> dict[str, Any]:
rows_by_run = [run["diagnostic"]["depth_weights"] for run in runs]
layer_count = len(rows_by_run[0])
if any(len(rows) != layer_count for rows in rows_by_run):
raise ValueError("depth-weight layer counts differ")
rows = []
max_sources = 0
for layer in range(layer_count):
source_count = rows_by_run[0][layer]["sources"]
if any(rows[layer]["sources"] != source_count for rows in rows_by_run):
raise ValueError("source count differs across seeds")
weights = [
mean(rows_by_run[seed_index][layer]["mean_weights"][source]
for seed_index in range(len(runs)))
for source in range(source_count)
]
entropies = [
rows_by_run[seed_index][layer]["entropy_mean"]
for seed_index in range(len(runs))
]
rows.append(
{
"sublayer": layer + 1,
"sources": source_count,
"mean_weights": weights,
"entropy_mean": mean(entropies),
"entropy_min": min(entropies),
"entropy_max": max(entropies),
}
)
max_sources = max(max_sources, source_count)
output_rows = [run["diagnostic"]["output_weights"] for run in runs]
output_source_count = output_rows[0]["sources"]
output_weights = [
mean(row["mean_weights"][source] for row in output_rows)
for source in range(output_source_count)
]
return {
"rows": rows,
"max_sources": max_sources,
"output": {
"sources": output_source_count,
"mean_weights": output_weights,
"entropy_mean": mean(row["entropy_mean"] for row in output_rows),
"entropy_min": min(row["entropy_mean"] for row in output_rows),
"entropy_max": max(row["entropy_mean"] for row in output_rows),
},
}
def main() -> None:
args = parse_args()
manifest = read_json(args.manifest)
if manifest["protocol_id"] != PROTOCOL_ID:
raise ValueError("manifest protocol mismatch")
runs: dict[tuple[str, int], dict[str, Any]] = {}
formal_file_hashes: dict[str, str] = {}
for seed in SEEDS:
for architecture in ARCHITECTURES:
path = args.formal_dir / f"{architecture}-{seed}.json"
run = read_json(path)
if run["protocol_id"] != PROTOCOL_ID:
raise ValueError(f"protocol mismatch: {path}")
if run["run_kind"] != "formal":
raise ValueError(f"not a formal run: {path}")
if run["architecture"] != architecture or run["seed"] != seed:
raise ValueError(f"cell identity mismatch: {path}")
if run["steps"] != 2000 or run["batch_size"] != 32:
raise ValueError(f"formal budget mismatch: {path}")
if run["target_bytes_seen"] != 16_384_000:
raise ValueError(f"target byte count mismatch: {path}")
if run["manifest"]["file_sha256"] != file_sha256(args.manifest):
raise ValueError(f"manifest file hash mismatch: {path}")
if run["manifest"]["formal_schedule_sha256"] != (
manifest["windows"]["formal_schedule_sha256"]
):
raise ValueError(f"schedule mismatch: {path}")
if run["evaluations"][-1]["step"] != 2000:
raise ValueError(f"missing final evaluation: {path}")
if any(
not math.isfinite(value)
for evaluation in run["evaluations"]
for value in (
evaluation["cross_entropy_nats"],
evaluation["bits_per_byte"],
)
):
raise ValueError(f"non-finite evaluation: {path}")
if len(run["diagnostic"]["layer_input_rms"]) != 32:
raise ValueError(f"diagnostic depth mismatch: {path}")
if len(run["diagnostic"]["core_parameter_grad_rms_by_block"]) != 16:
raise ValueError(f"gradient depth mismatch: {path}")
runs[(architecture, seed)] = run
formal_file_hashes[path.name] = file_sha256(path)
common_initial_exact = {}
for seed in SEEDS:
hashes = {
architecture: runs[(architecture, seed)]["hashes"][
"initial_common_parameters"
]
for architecture in ARCHITECTURES
}
common_initial_exact[str(seed)] = {
"hashes": hashes,
"exact": len(set(hashes.values())) == 1,
}
if not common_initial_exact[str(seed)]["exact"]:
raise ValueError(f"common initialization mismatch for seed {seed}")
by_seed = []
for seed in SEEDS:
values = {
architecture: runs[(architecture, seed)]["evaluations"][-1][
"bits_per_byte"
]
for architecture in ARCHITECTURES
}
by_seed.append(
{
"seed": seed,
"final_bpc": values,
"full_minus_baseline": values["full"] - values["baseline"],
"block_minus_baseline": values["block"] - values["baseline"],
"block_minus_full": values["block"] - values["full"],
}
)
final = {
"by_seed": by_seed,
"means": {
architecture: mean(
runs[(architecture, seed)]["evaluations"][-1]["bits_per_byte"]
for seed in SEEDS
)
for architecture in ARCHITECTURES
},
"full_contrast": verdict(
[row["full_minus_baseline"] for row in by_seed]
),
"block_contrast": verdict(
[row["block_minus_baseline"] for row in by_seed]
),
"block_minus_full": {
"paired_deltas_bpc": [row["block_minus_full"] for row in by_seed],
"mean_delta_bpc": mean(row["block_minus_full"] for row in by_seed),
},
}
evaluation_steps = [
evaluation["step"] for evaluation in runs[("baseline", SEEDS[0])]["evaluations"]
]
curves = {}
for architecture in ARCHITECTURES:
curve = []
for index, step in enumerate(evaluation_steps):
values = [
runs[(architecture, seed)]["evaluations"][index]["bits_per_byte"]
for seed in SEEDS
]
if any(
runs[(architecture, seed)]["evaluations"][index]["step"] != step
for seed in SEEDS
):
raise ValueError("evaluation step mismatch")
curve.append(
{
"step": step,
"mean_bpc": mean(values),
"min_bpc": min(values),
"max_bpc": max(values),
"by_seed": values,
}
)
curves[architecture] = curve
timing = {}
for architecture in ARCHITECTURES:
cells = [runs[(architecture, seed)]["timing"] for seed in SEEDS]
timing[architecture] = {
"mean_step_ms": mean(cell["mean_ms"] for cell in cells),
"median_step_ms": mean(cell["median_ms"] for cell in cells),
"p95_step_ms": mean(cell["p95_ms"] for cell in cells),
"mean_peak_allocated_bytes": mean(
cell["peak_allocated_bytes"] for cell in cells
),
"mean_peak_reserved_bytes": mean(
cell["peak_reserved_bytes"] for cell in cells
),
"by_seed": cells,
}
timing["relative_to_baseline"] = {
architecture: {
"step_time_ratio": timing[architecture]["mean_step_ms"]
/ timing["baseline"]["mean_step_ms"],
"allocated_memory_ratio": timing[architecture][
"mean_peak_allocated_bytes"
]
/ timing["baseline"]["mean_peak_allocated_bytes"],
}
for architecture in ("full", "block")
}
parameters = {
architecture: runs[(architecture, SEEDS[0])]["model"]["parameters"]
for architecture in ARCHITECTURES
}
parameters["mixer_overhead_fraction_of_baseline"] = (
parameters["full"]["mixer"] / parameters["baseline"]["total"]
)
traces = {}
gradients = {}
for architecture in ARCHITECTURES:
architecture_runs = [runs[(architecture, seed)] for seed in SEEDS]
traces[architecture] = {
key: elementwise_summary(
[run["diagnostic"][key] for run in architecture_runs]
)
for key in (
"layer_input_rms",
"branch_output_rms",
"stream_state_rms",
)
}
gradient_rows = [
run["diagnostic"]["core_parameter_grad_rms_by_block"]
for run in architecture_runs
]
gradients[architecture] = {
"by_block": elementwise_summary(gradient_rows),
"cv_by_seed": [
coefficient_of_variation(row) for row in gradient_rows
],
"mean_cv": mean(coefficient_of_variation(row) for row in gradient_rows),
"first_last_ratio_by_seed": [
row[0] / row[-1] for row in gradient_rows
],
"mean_first_last_ratio": mean(row[0] / row[-1] for row in gradient_rows),
}
mixers = {
architecture: average_depth_weights(
[runs[(architecture, seed)] for seed in SEEDS]
)
for architecture in ("full", "block")
}
full_branch = traces["full"]["branch_output_rms"]["mean"]
largest_index = max(range(len(full_branch)), key=full_branch.__getitem__)
# Full output source 0 is the embedding; branch l is source l+1.
largest_source_weight = mixers["full"]["output"]["mean_weights"][
largest_index + 1
]
uniform_output_weight = 1 / mixers["full"]["output"]["sources"]
posthoc = {
"label": "post-hoc descriptive callout; not a preregistered endpoint",
"largest_full_branch_sublayer": largest_index + 1,
"largest_full_branch_rms": full_branch[largest_index],
"corresponding_final_output_weight": largest_source_weight,
"uniform_final_output_weight": uniform_output_weight,
"weight_over_uniform": largest_source_weight / uniform_output_weight,
}
replay = read_json(args.replay)
formal_replay_source = runs[("block", 2026073001)]
replay_exact = {
field: formal_replay_source[field] == replay[field]
for field in REPLAY_FIELDS
}
if not all(replay_exact.values()):
raise ValueError(f"formal replay mismatch: {replay_exact}")
smoke = {}
for architecture in ARCHITECTURES:
first_path = args.smoke_dir / f"{architecture}-2026073001-a.json"
second_path = args.smoke_dir / f"{architecture}-2026073001-b.json"
first = read_json(first_path)
second = read_json(second_path)
exact = {field: first[field] == second[field] for field in SMOKE_FIELDS}
if not all(exact.values()):
raise ValueError(f"smoke mismatch for {architecture}: {exact}")
smoke[architecture] = {
"fields": exact,
"all_exact": True,
"first_sha256": file_sha256(first_path),
"second_sha256": file_sha256(second_path),
}
reproduction = {
"schema_version": 1,
"protocol_id": PROTOCOL_ID,
"manifest_sha256": file_sha256(args.manifest),
"formal_files": formal_file_hashes,
"common_initial_parameters": common_initial_exact,
"smoke": smoke,
"formal_replay": {
"architecture": "block",
"seed": 2026073001,
"fields": replay_exact,
"all_numeric_and_hash_fields_exact": all(replay_exact.values()),
"timing_exact_required": False,
"timing_exact_observed": formal_replay_source["timing"] == replay["timing"],
"formal_file_sha256": formal_file_hashes[
"block-2026073001.json"
],
"replay_file_sha256": file_sha256(args.replay),
},
}
reproduction["canonical_sha256_without_self"] = canonical_sha256(reproduction)
analysis = {
"final_validation": final,
"evaluation_curves": curves,
"timing": timing,
"parameters": parameters,
"traces": traces,
"gradients": gradients,
"mixers": mixers,
"posthoc": posthoc,
"interpretation": {
"primary": (
"Both Full and Block AttnRes satisfy the preregistered "
"directional-support rule in this reduced protocol."
),
"bounded_depth_pattern": (
"Block partial-state RMS resets every four residual sublayers; "
"the complete 32-point vectors are reported."
),
"gradient_boundary": (
"The preregistered core-parameter gradient RMS is not more "
"uniform for AttnRes here; this metric and scale do not reproduce "
"the paper's large-model gradient-magnitude result."
),
"scope": (
"Reduced byte-level WikiText-2 mechanism probe; not a K3 "
"checkpoint run, paper-scale reproduction, benchmark, or "
"same-FLOP comparison."
),
},
}
raw = {
"schema_version": 1,
"protocol_id": PROTOCOL_ID,
"manifest": manifest,
"provenance": {
"manifest_file_sha256": file_sha256(args.manifest),
"formal_file_sha256": formal_file_hashes,
"reproduction_sha256": reproduction[
"canonical_sha256_without_self"
],
},
"formal_runs": [
runs[(architecture, seed)]
for seed in SEEDS
for architecture in ARCHITECTURES
],
"analysis": analysis,
"reproduction": reproduction,
}
raw["canonical_sha256_without_self"] = canonical_sha256(raw)
compact = {
"schema_version": 1,
"protocol_id": PROTOCOL_ID,
"dataset": {
"repository": manifest["dataset"]["repository"],
"revision": manifest["dataset"]["revision"],
"train_bytes": manifest["dataset"]["splits"]["train"][
"concatenated_bytes"
],
"schedule_sha256": manifest["windows"]["formal_schedule_sha256"],
"validation_sha256": manifest["windows"][
"validation_tensor_sha256"
],
},
"grid": {
"architectures": list(ARCHITECTURES),
"seeds": list(SEEDS),
"runs": 9,
"steps_per_run": 2000,
"target_bytes_per_run": 16_384_000,
"target_bytes_total": 9 * 16_384_000,
},
"final_validation": final,
"evaluation_curves": curves,
"timing": timing,
"parameters": parameters,
"traces": traces,
"gradients": gradients,
"mixers": mixers,
"posthoc": posthoc,
"interpretation": analysis["interpretation"],
"reproduction": reproduction,
"source_sha256": raw["canonical_sha256_without_self"],
}
compact["canonical_sha256_without_self"] = canonical_sha256(compact)
write_json(args.reproduction_output, reproduction)
write_json(args.output, raw)
write_json(args.compact_output, compact)
print(
json.dumps(
{
"output": str(args.output),
"compact_output": str(args.compact_output),
"reproduction_output": str(args.reproduction_output),
"raw_sha256": file_sha256(args.output),
"compact_sha256": file_sha256(args.compact_output),
"reproduction_sha256": file_sha256(args.reproduction_output),
"full": final["full_contrast"],
"block": final["block_contrast"],
"formal_replay": reproduction["formal_replay"],
},
ensure_ascii=False,
indent=2,
)
)
if __name__ == "__main__":
main()
experiments/k3/attnres/reproduction.json
+109
View File
@@ -0,0 +1,109 @@
{
"canonical_sha256_without_self": "89a127e625ccefbd9b749e6ffca4ba72868e217e0c7a24773024bd2fafde427a",
"common_initial_parameters": {
"2026073001": {
"exact": true,
"hashes": {
"baseline": "af2724a1c34bcfd61d8a8bef402246430898c815e56b6e5c5257949a4eb0e7b1",
"block": "af2724a1c34bcfd61d8a8bef402246430898c815e56b6e5c5257949a4eb0e7b1",
"full": "af2724a1c34bcfd61d8a8bef402246430898c815e56b6e5c5257949a4eb0e7b1"
}
},
"2026073002": {
"exact": true,
"hashes": {
"baseline": "9fd4f04212ab5cea822f469902d8e80ecc368da329f3c20abacfe6b7a50ed523",
"block": "9fd4f04212ab5cea822f469902d8e80ecc368da329f3c20abacfe6b7a50ed523",
"full": "9fd4f04212ab5cea822f469902d8e80ecc368da329f3c20abacfe6b7a50ed523"
}
},
"2026073003": {
"exact": true,
"hashes": {
"baseline": "7a565cd353efdb3b95e9b8b1844581082c029991ea18d438b4b18566970c8c61",
"block": "7a565cd353efdb3b95e9b8b1844581082c029991ea18d438b4b18566970c8c61",
"full": "7a565cd353efdb3b95e9b8b1844581082c029991ea18d438b4b18566970c8c61"
}
}
},
"formal_files": {
"baseline-2026073001.json": "f5cddecace6a70ee3824f272811d2a6336ab6dc24ad1100b18292f4413692e9a",
"baseline-2026073002.json": "4c3b9e1ea2ed6fd3078f1212984327364832a1146bfcbdb3a6f2e61b589bfaa8",
"baseline-2026073003.json": "e5ce5a8944dad0c4811f8bf19b4cfec8ce4a9283858fd59cd739dbc37626707b",
"block-2026073001.json": "5df870369d9a86ccb4ba4191fbd1d6f3642893dd47a60f8f6d1143006bdfbdaf",
"block-2026073002.json": "be1f109a630e27c8469438e33b53806cdaecac2f712af4f885da83f00ef6843b",
"block-2026073003.json": "949ec51ca3a219a11f260171da01f737296241536f5e139cf07d444c27efba92",
"full-2026073001.json": "648cb25ea98da1868779733da155275e9a16aad5efbbeeced7812c19d75817d5",
"full-2026073002.json": "1fa715bbd81a3a04a5aa0ba0fc27feb8883c44a01e9f28e281a309a394447979",
"full-2026073003.json": "4e34fa35bbfec460cabc94bce7a08f44fc188d908705ae2e256500f00813b4e3"
},
"formal_replay": {
"all_numeric_and_hash_fields_exact": true,
"architecture": "block",
"fields": {
"diagnostic": true,
"environment": true,
"evaluations": true,
"hashes": true,
"manifest": true,
"model": true,
"optimizer": true,
"training_history": true
},
"formal_file_sha256": "5df870369d9a86ccb4ba4191fbd1d6f3642893dd47a60f8f6d1143006bdfbdaf",
"replay_file_sha256": "e74d3323e5fe31378bb8aad7a8efa2fb91995cd7224c158cf03006466cdea2a7",
"seed": 2026073001,
"timing_exact_observed": false,
"timing_exact_required": false
},
"manifest_sha256": "9778ade5b1c9dd7676d2cdc52b4e4e7ff5ae513cb56c667974e2422702f9dc2b",
"protocol_id": "llm-atlas-k3-attnres-reduced-v1",
"schema_version": 1,
"smoke": {
"baseline": {
"all_exact": true,
"fields": {
"diagnostic": true,
"environment": true,
"evaluations": true,
"hashes": true,
"manifest": true,
"model": true,
"optimizer": true,
"training_history": true
},
"first_sha256": "fd8b14f70a6a14978e65f9899b694164ba91753bf82eead436a40837c251e82c",
"second_sha256": "fd8b14f70a6a14978e65f9899b694164ba91753bf82eead436a40837c251e82c"
},
"block": {
"all_exact": true,
"fields": {
"diagnostic": true,
"environment": true,
"evaluations": true,
"hashes": true,
"manifest": true,
"model": true,
"optimizer": true,
"training_history": true
},
"first_sha256": "980a4a534e458199d95b5864b008a81f51b565e05a7b2f24a644b36d2134eccc",
"second_sha256": "980a4a534e458199d95b5864b008a81f51b565e05a7b2f24a644b36d2134eccc"
},
"full": {
"all_exact": true,
"fields": {
"diagnostic": true,
"environment": true,
"evaluations": true,
"hashes": true,
"manifest": true,
"model": true,
"optimizer": true,
"training_history": true
},
"first_sha256": "c6e32b737c7bf7fdc9b647650ebd7c9d2f9f8550f66a5b84f168f16fa7d2eacb",
"second_sha256": "c6e32b737c7bf7fdc9b647650ebd7c9d2f9f8550f66a5b84f168f16fa7d2eacb"
}
}
}
research/K3_ATTNRES_REDUCED_AUDIT.md
+568
View File
@@ -0,0 +1,568 @@
# Kimi K3 第四轮：Attention Residuals 缩小版独立机制实验审计
> 协议：`llm-atlas-k3-attnres-reduced-v1`
>
> 预注册：`research/K3_ATTNRES_REDUCED_PROTOCOL.md`
>
> 数据清单：`experiments/k3/attnres/manifest.json`
>
> 执行日期：2026-07-30
>
> 执行设备：NVIDIA GeForce RTX 5090；PyTorch `2.11.0+cu128`
## 0. 先说结论
这一轮没有伪装成“跑通了 K3”。它做的是一个刻意缩小、从零训练、可公开复查的
Attention Residuals（AttnRes）机制实验：
```text
3 个结构
× 3 个预先冻结的初始化 seed
× 每格 2,000 steps
× 每步 32 × 256 target bytes
= 9 个正式训练格，147,456,000 target bytes
```
三个结构共享完全相同的 16 个 Transformer blocks、32 个残差子层、初始公共参数、
训练窗口、优化器与验证集。唯一设计变量是“前面产生的 residual states 怎样供下一层读取”：
```text
Baseline：只读上一残差状态
Full AttnRes：对全部历史残差状态做按维 softmax 混合
Block AttnRes：每 4 个残差子层形成一个局部块，块间再混合
```
在冻结的 64 个验证窗口上，最终 bits per byte（BPC，越低越好）为：
| seed | Baseline | Full | Block | Full − Base | Block − Base |
|---:|---:|---:|---:|---:|---:|
| 2026073001 | 2.00054 | 1.98454 | 1.94788 | −0.01600 | −0.05266 |
| 2026073002 | 1.99844 | 1.98523 | 1.95709 | −0.01321 | −0.04135 |
| 2026073003 | 1.99842 | 1.98392 | 1.96503 | −0.01450 | −0.03339 |
| 三 seed 均值 | 1.99913 | 1.98457 | 1.95667 | **−0.01457** | **−0.04247** |
按照看结果前冻结的判据——三个 seed 同为负，并且均值不高于 `−0.010 BPC`——Full 与
Block 都得到：
> **directional support in this reduced protocol**
中文应该读成：
> 在这套缩小训练合同中，允许子层重新读取更早的残差状态，方向一致地改善了验证 BPC。
它**不应该**读成：
- 已复现 AttnRes 论文的大模型收益；
- 已运行 Kimi K3 checkpoint；
- 已证明 Block 一般优于 Full；
- 已得到同参数、同 FLOPs 或同 wall time 的优势；
- 三个 seed 可以支持总体显著性、置信区间或 scaling-law 外推。
本轮还有一个同样重要的反结果：预注册的“16 个 Transformer blocks 的核心参数梯度 RMS
变异系数”在本实验中，Baseline 为 `0.3447`，Full 为 `0.5087`，Block 为 `0.6306`。
在这个定义和尺度下，AttnRes **没有**表现出更均匀的跨深度梯度。这个结果与论文的大模型
梯度叙述不能直接对齐，网站必须把它作为边界而不是藏起来。
---
## 1. 为什么不是直接跑 K3
### 1.1 公开 checkpoint 的未决形状冲突
第三轮工件审计在 K3 第一层 KDA 中发现：
| 官方工件 | `A_log` 所要求或实际给出的形状 |
|---|---|
| `config.json` | `num_heads = 96` |
| Hugging Face remote code | 按 `num_heads` 构造，即 `[96]` |
| FlashKDA kernel API | `[H]`，K3 中应为 `[96]` |
| checkpoint safetensors header | `[128]` |
本轮再次检查了当前官方 Hugging Face / GitHub 工件，并额外检查当前 vLLM 与 SGLang
Kimi K3 loader。两者都按 local head 维构造和切分 `A_log`，没有公开 `128 → 96` 的转换规则。
因此，以下做法都会越过证据：
```text
裁掉最后 32 个值
把 128 强行 reshape 成别的语义
把 128 解释成 head_dim
绕过 loader 后把输出叫作“K3 forward”
```
在 Moonshot 给出转换合同、修订权重，或一个官方 loader 明确处理这 32 个额外值之前，
本站不制造“真实 K3 前向结果”。
### 1.2 为什么缩小版仍有价值
不能诚实执行 1.56 TB checkpoint，不等于只能停在架构示意图。AttnRes 的核心问题可以被
缩成一个更小、但仍可被证伪的问题：
> 在相同 Transformer 主干、相同输入窗口与相同初始化下，把“固定单位 residual
> connection”替换成“学习的历史 residual 混合”，短预算训练是否出现一致方向？
这个问题不依赖 KDA、MLA、MoE、MXFP4、视觉塔或完整 K3 参数。它只检验 AttnRes
的局部机制方向，并且可以把全部代码、清单、聚合指标和复现哈希开源。
---
## 2. 先冻结什么
首版协议在正式结果产生前冻结以下项目：
- 数据仓库、revision、config、split 拼接规则与 byte tokenizer；
- 模型层数、宽度、head 数、FFN 宽度、位置编码、RMSNorm 与激活函数；
- Baseline / Full / Block 的精确定义；
- 三个初始化 seed；
- 每格训练步数、batch、context 与总 target bytes；
- AdamW、学习率曲线、weight decay、gradient clipping；
- 验证步、固定验证窗口与主指标；
- 主对比、方向判据和“不能宣称什么”；
- 深度 RMS、混合权重与梯度诊断；
- smoke 与正式独立进程 replay 合同。
正式协议不是从本轮曲线倒推的。关键提交顺序为：
```text
9039de1 research: preregister reduced AttnRes study
e361753 research: lock reduced AttnRes model contract
f998720 research: add reduced AttnRes runner
1f20f81 research: freeze reduced AttnRes corpus
```
### 2.1 一次被公开保留的预训练故障
第一次 smoke 在 step 0 的验证 forward 后停止：
```text
RuntimeError:
view size is not compatible with input tensor's size and stride
```
原因是切片后的 target tensor 不连续，而 loss 路径用了 `.view()`。当时：
- 尚未执行一个 optimizer step；
- 没有正式结果文件；
- 没有任何条件的训练或最终 BPC 可供选择。
修复仅把 `.view()` 改为语义等价且支持非连续输入的 `.reshape()`：
```text
5f49906 fix: flatten noncontiguous AttnRes targets
```
随后三个结构分别完成两次独立 smoke，冻结字段逐字段 exact。这个故障不改变实验设计，
但应留在审计链中，避免“第一次就完美运行”的虚假叙事。
---
## 3. 数据合同：文本怎样变成 byte 任务
### 3.1 固定数据源
| 项目 | 固定值 |
|---|---|
| repository | `Salesforce/wikitext` |
| revision | `b08601e04326c79dfdd32d625aee71d232d685c3` |
| config | `wikitext-2-raw-v1` |
| 行处理 | `(text or "") + "\n"` |
| 编码 | UTF-8 |
| tokenizer | byte ID `0..255` |
| vocabulary | 256 |
拼接后的 split：
| split | bytes | SHA-256 |
|---|---:|---|
| train | 10,951,563 | `0ca7d3e74dbe44564ea5942b85232f1bbcb525c9cd481cd5d28a87ee90e7e9b4` |
| validation | 1,148,008 | `a42356f6a8ff1d25daf25ec9db49e10a537c265581b61c74604bb63231dee719` |
| test | 1,292,014 | `bfe9eb16ab9987fb88bde4ea9a30a00f2a45db01dfc14bad78d05325789c4f12` |
这里的 byte tokenizer 不是为了追求最佳语言模型性能，而是为了移除另一个潜在变量：
不同 BPE 模型、词表和 normalization。BPC 也因此可以直接比较，而不受 tokenization
长度变化影响。
### 3.2 无状态窗口计划
训练窗口不靠进程内 RNG 顺序产生。对每个：
```text
architecture + seed + step + batch row
```
协议用 SHA-256 派生 train start offset。三种结构在同一 seed 下使用相同窗口；结构名不进入
窗口选择的有效随机盐。冻结计划包含 192,000 个起点，SHA-256 为：
```text
81521a70ec61f3717968f160cb711e50c5f52a665a6961538d339360cb695f48
```
固定 64 个验证窗口 tensor hash：
```text
5f71fda757fc75010ed16e7636bc394c69f55b34a3713b3b5a7ef8e03eae3c20
```
固定 16 个诊断窗口 hash：
```text
d970af9b0c656c9826f369b5fe6e3869a6f6cfeccfa5a922fe94ed1d24b86818
```
---
## 4. 模型合同：只改变 residual 读取拓扑
### 4.1 公共主干
| 项目 | 值 |
|---|---:|
| Transformer blocks | 16 |
| residual sublayers | 32（每 block attention + FFN） |
| model width | 192 |
| attention heads | 6 |
| head dimension | 32 |
| SwiGLU hidden | 768 |
| context | 256 bytes |
| dropout | 0 |
| position | learned absolute embedding |
| norm | pre-RMSNorm + final RMSNorm |
| attention softmax | causal, float32 |
| embedding / LM head | tied |
Attention projection 与 SwiGLU 都不使用 bias。三个结构的公共 core 参数均为
`9,541,824`，每个 seed 的公共参数初始化哈希在三种结构间 exact。
### 4.2 Baseline
普通 residual 子层：
```text
x_(l+1) = x_l + F_l(RMSNorm(x_l))
```
它只保留一个随深度持续累积的 residual stream。
### 4.3 Full AttnRes
第 `l` 个子层先对从 embedding 到当前深度的全部 residual states 做学习混合：
```text
α_l = softmax(q_l · RMSNorm(states))
x̃_l = Σ_i α_(l,i) state_i
state_(l+1) = F_l(RMSNorm(x̃_l))
```
`q_l` 是按 hidden dimension 学习的 pseudoquery，初始化为 0，所以初始 softmax 为均匀
读取。最后还有一个 output mixer，把 33 个可见 sources 混成 LM head 的输入。
### 4.4 Block AttnRes
32 个 residual sublayers 被分成 8 块，每块 4 层：
```text
块内：新 branch state 做普通局部累加
块边界：对历史块状态做学习混合，产生下一块输入
```
它保留“可以回读历史”的机制，同时把 Full 随深度增长的状态集合限制在块级。
### 4.5 参数公平与计算不公平
| 结构 | core | mixer | total | 相对 Baseline mixer overhead |
|---|---:|---:|---:|---:|
| Baseline | 9,541,824 | 0 | 9,541,824 | 0 |
| Full | 9,541,824 | 12,672 | 9,554,496 | 0.1328% |
| Block | 9,541,824 | 12,672 | 9,554,496 | 0.1328% |
这是近似同参数，不是同 FLOPs。当前教学实现用 PyTorch eager 保存和混合历史 states，
没有使用论文的大模型优化 kernel，所以它适合机制观察，不适合推断生产吞吐。
---
## 5. 训练与主指标
### 5.1 每格预算
```text
2,000 steps
× batch 32
× context 256 target bytes
= 16,384,000 target bytes / run
```
优化器：
```text
AdamW β=(0.9, 0.95), ε=1e-8
peak LR=3e-4, min LR=3e-5
100-step warmup + cosine decay
weight decay=0.1 for ndim>=2
global grad clip=1.0
BF16 autocast
```
验证发生在 `0, 100, 250, 500, 1000, 1500, 2000` steps。主指标只使用 step 2000
的固定验证 BPC；曲线用于帮助理解，不用于重新选择终点。
### 5.2 判据为什么这么保守
只有三个 seed，不能可靠估计总体方差或给出有意义的 population confidence interval。
所以预注册不用 p-value，而只问两个简单问题：
```text
三个 paired deltas 是否同方向？
mean delta 是否至少达到 0.010 BPC？
```
如果答案都是“是”，只写作本协议内的 directional support / concern。它是一道防止
夸大结论的阈值，不是一个通用显著性标准。
### 5.3 正式结果
Full 的三组配对差：
```text
−0.015996
−0.013207
−0.014498
mean = −0.014567 BPC
```
Block 的三组配对差：
```text
−0.052660
−0.041349
−0.033388
mean = −0.042466 BPC
```
两组都满足预注册的方向支持规则。Block 相对 Full 的均值差为 `−0.027898 BPC`，三个
seed 也同为负；但“Block − Full”不是预注册主判据，而且本实现的计算图与优化效率不同，
所以它只适合描述，不升级成一般性排名。
---
## 6. 它花了多少计算与显存
排除每次前 20 个计时 warmup steps 后，三个 seed 的均值：
| 结构 | mean step | 相对 Baseline | peak allocated | 相对 Baseline |
|---|---:|---:|---:|---:|
| Baseline | 21.62 ms | 1.00× | 3.04 GB | 1.00× |
| Full | 146.81 ms | 6.79× | 14.26 GB | 4.70× |
| Block | 54.63 ms | 2.53× | 6.52 GB | 2.15× |
这张表应该怎样读：
- Full 在教学实现中为每层保留并读取更多历史 states，因此最贵；
- Block 把可见历史限制在块级，成本明显下降；
- 时间与显存是“这份 PyTorch eager 实现 + RTX 5090”的观测；
- 不能把 6.79× / 2.53× 外推到论文 kernel、K3 训练系统或生产推理；
- BPC 改善不能被写成同 FLOPs 改善。
---
## 7. 看 residual stream：Block 为什么呈现锯齿
Baseline 的 stream-state RMS 从平均 `0.0661` 增到 `0.2110`：普通 residual
connection 把分支输出一路累积。
Block 的前 12 个 partial-state RMS 是：
```text
0.0334 0.0598 0.0967 0.1181
0.0509 0.0569 0.0745 0.0815
0.0532 0.0581 0.1079 0.1158
```
每四个值形成一个局部块：
```text
块内：逐层累积，RMS 通常上升
块间：重新从历史块状态混合，partial stream 被重置
```
所以曲线不是训练不稳定造成的随机锯齿，而是 Block 拓扑的直接几何痕迹。完整 32 点向量、
三个 seed 的 min / mean / max 均进入公开 JSON。
Full 的 layer-input RMS 则从 `0.0544` 降到 `0.00866`。这不等于信息“消失”：
每层输入是多个经过 RMSNorm 的历史 states 的学习加权和，混合可以通过方向抵消改变
合成向量的 RMS。只看单个标量不能判断信息保留量。
---
## 8. mixer 权重：一个事后但有解释力的观察
Full 在第 31 个 residual sublayer 的 branch-output RMS 平均达到 `1.4964`，是 32 层中
最大值。但最终 output mixer 给这个 source 的平均权重只有：
```text
observed = 0.002794
uniform = 1 / 33 = 0.030303
ratio = 0.0922× uniform
```
一个直观解释是：输出混合器学会了压低这个幅值突增的 source，而不是被迫把它以单位
residual 权重传到输出。
必须同时保留两个限制：
1. “最大 spike 对应低权重”是看完完整 trace 后挑出的描述；
2. 它不是预注册 endpoint，不能作为独立确认性证据。
网站会明确标注 **post-hoc descriptive callout**，并展示完整深度权重图，让读者看到它
不是从被隐藏的其他 source 中挑出的孤立数字。
---
## 9. 梯度结果没有复现论文叙述
预注册诊断对每个 Transformer block 的公共 core 参数计算 gradient RMS，再求 16 个
block 间的 coefficient of variation：
| 结构 | seed 1 | seed 2 | seed 3 | mean CV |
|---|---:|---:|---:|---:|
| Baseline | 0.3537 | 0.3409 | 0.3396 | **0.3447** |
| Full | 0.4890 | 0.5100 | 0.5272 | **0.5087** |
| Block | 0.6622 | 0.6849 | 0.5448 | **0.6306** |
CV 越低，按这个特定定义才越均匀。因此本轮观察是：
```text
Baseline < Full < Block
```
这和 AttnRes 论文在大模型训练中报告的、更平坦的跨深度梯度幅值叙述不是同一个结果。
合理边界包括：
- 本实验只有 width 192、16 blocks、2,000 steps；
- 本指标是“按 block 汇总的核心参数 gradient RMS”；
- 论文图可能观察 activation / residual-output gradients，聚合对象并不相同；
- byte-level WikiText-2 与论文的大规模训练数据、优化器状态和训练阶段不同。
正确表述是：
> 本缩小实验的主 BPC 对比支持 AttnRes 的方向，但预注册的核心参数梯度均匀性指标不支持
> 论文式叙述；这提示该解释可能依赖尺度、指标定义或训练阶段，需要后续专门实验。
不正确的做法是改换一个看起来更漂亮的梯度统计后，只展示新指标。
---
## 10. 复现链
### 10.1 smoke
三个结构各执行两次独立的 20-step smoke。以下字段对每个结构都 exact：
```text
manifest
model
optimizer
hashes
evaluations
training_history
diagnostic
environment
```
### 10.2 正式独立进程 replay
预先指定：
```text
architecture = block
seed = 2026073001
steps = 2000
```
正式格与 fresh-process replay 的最终 BPC 都是：
```text
1.947877975922777
```
上面的八组冻结字段全部 exact。计时不要求 exact，实际也不相等，因为 wall time 受系统
调度影响。
| 文件 | SHA-256 |
|---|---|
| formal block / seed 1 | `5df870369d9a86ccb4ba4191fbd1d6f3642893dd47a60f8f6d1143006bdfbdaf` |
| fresh replay | `e74d3323e5fe31378bb8aad7a8efa2fb91995cd7224c158cf03006466cdea2a7` |
### 10.3 公开产物
| 产物 | 内容 | SHA-256 |
|---|---|---|
| `src/data/k3-attnres-reduced.json` | 9 个完整 run + 聚合 + 复现记录 | `44f8622654d32485f8d6e698c02ba1365ddffb10cbd73db0294136d0bd93ce88` |
| `src/data/k3-attnres-reduced-compact.json` | 网站所需完整曲线与诊断 | `44864d48eddb2ae5887fba4b74f63b5a3d6a23497886ee307decf5b4f45d9faf` |
| `experiments/k3/attnres/reproduction.json` | smoke、初始化与 replay audit | `545543b7e4a970ca3bc0e6246610a32fb53ec3d9546a98e0f17f38d7121918a2` |
聚合器在同一批只读 run 文件上再次运行后，三个文件 SHA-256 全部不变。
---
## 11. 证据等级
### A. 本轮可直接主张
- 冻结协议下 9 个训练格的最终 BPC 与完整验证曲线；
- Full / Block 相对 Baseline 的三 seed 配对方向；
- 当前实现的参数量、实测 step time 与 peak allocated memory；
- 固定诊断窗口上的 residual RMS、mixer 权重与参数梯度统计；
- smoke 与指定正式格的独立进程 exact replay；
- 数据、窗口、初始化和产物 SHA-256。
### B. 只能作为机制解释
- Block 的四层锯齿与块边界重混合一致；
- Full mixer 可能通过降低权重抑制高 RMS source；
- Full / Block 的短预算优势可能来自更灵活的深度路由。
这些解释与观测相容，但不是唯一因果解释。
### C. 本轮明确不主张
- K3 checkpoint 已成功 forward 或训练；
- 论文表格、Figure 4–8 或 paper-scale scaling 已复现；
- AttnRes 在任意模型、数据和预算上都降低 loss；
- Block 一般优于 Full；
- 梯度在 AttnRes 中更均匀；
- 同 FLOPs、同 wall time 或生产系统的性价比优势；
- 三个 seed 支持总体统计显著性。
---
## 12. 下一步
这个实验把“能运行的独立机制 probe”完成了，但真实 K3 仍有两道门：
1. `A_log [128]` 的官方转换或权重修订；
2. 能加载完整或官方切分 K3 的受支持执行环境。
AttnRes 本身的下一轮也不应只增加 seed。优先级更高的是：
- 对齐论文实际使用的 activation / output-gradient 诊断定义；
- 增加 depth 与训练预算，检验梯度结论是否随尺度翻转；
- 做 mixer 计算的优化实现，再讨论同 wall-time 或近似同-FLOP 对比；
- 冻结一个更强 tokenizer / corpus 后检查 byte-level 结论是否保持；
- 将 Block size 作为预注册变量，而不是看完结果后挑 4。
---
## 13. 一手来源
- [Kimi K3 Technical Report](https://arxiv.org/abs/2607.24653)
- [Kimi K3 official checkpoint](https://huggingface.co/moonshotai/Kimi-K3)
- [Kimi K3 official code repository](https://github.com/MoonshotAI/Kimi-K3)
- [Attention Residuals](https://arxiv.org/abs/2603.15031)
- [Official Attention Residuals implementation](https://github.com/MoonshotAI/Attention-Residuals)
- [WikiText dataset repository](https://huggingface.co/datasets/Salesforce/wikitext)
- [vLLM Kimi K3 implementation](https://github.com/vllm-project/vllm)
- [SGLang Kimi K3 implementation](https://github.com/sgl-project/sglang)
Grok CLI 在协议冻结前只承担一次对抗式方法审阅：它提出锁定参数容量、残差拓扑、数据顺序、
指标定义和复现合同的检查项。所有论文事实和实验结论仍由一手来源、冻结代码与本地运行产物
支持；Grok 输出不作为证据来源。
src/data/k3-attnres-reduced-compact.json
+3106
View File
File diff suppressed because it is too large Load Diff
src/data/k3-attnres-reduced.json
+21498
View File
File diff suppressed because it is too large Load Diff