diff --git a/PROGRESS.md b/PROGRESS.md index 4c741fd..a8aad76 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -8,7 +8,7 @@ |---|---:|---:|---| | 研究框架与规范 | 进行中 | 83% | Scaling Laws 二轮拟合复现与逐图精读 | | 网站设计系统 | 进行中 | 89% | 打印样式与更多通用可视化组件 | -| Kimi K3 深读 | 完成二轮 | 88% | 第三轮加入官方权重 traces、独立复现与逐图数值重绘 | +| Kimi K3 深读 | 三轮实证进行中 | 92% | 匹配 CUDA 12.9+ 执行 FlashKDA,并接入真实 hidden-state / expert-load traces | | 语言模型前史 | 完成首版 | 78% | Kneser–Ney、LSTM、Bahdanau 逐图精读与真实小语料复现 | | Transformer 基础 | 完成首版 | 79% | 多头电路、归一化 traces 与真实 kernel / KV 配置 | | 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 | @@ -41,7 +41,7 @@ - [x] 完成 486 篇关键论文索引,覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。 - [x] 完成可检索、可按专题筛选的论文库页面。 - [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。 -- [x] 完成 K3 三轴架构与八联报告实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 四联实验、长上下文、MoE 路由、推理三页签,以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等六十七个原创交互视图。 +- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 四联实验、长上下文、MoE 路由、推理三页签,以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等七十一个原创交互视图。 - [x] 完成长上下文首版:五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。 - [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文,并建立长上下文研究账本。 - [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文,并建立 MoE 研究账本。 @@ -156,10 +156,17 @@ - [x] 完成 K3 八联交互实验:Delta Rule、bounded decay、Block AttnRes、LatentMoE payload、SiTU-GLU、Quantile Balancing、MOPD/partial rollout、hybrid prefix cache。 - [x] K3 专属 Chrome 断言通过:32/21/100 内容计数、八个实验计算、键盘 tabs、事实纠错、桌面与 390px 移动端均无异常。 - [x] K3 二轮以源提交 `b669615`、不可变镜像 `20260729T040336Z-b669615` 发布;OCI digest `sha256:498b7e43…31cdb3c`,NAS、VPS/Tailscale、NPM、DNS、HTTPS、证书、门户、公开 Forgejo 与十六套生产 Chrome 回归全链路通过;保留 `20260729T031901Z-cd96dab` 回滚。 +- [x] 启动 K3 三轮开放工件审计:固定 HF revision `9f62e4e9…b3569` 与 FlashKDA revision `1ce47ea3…ffb0b`,通过 config、index、safetensors headers 与 HTTP Range 避免把 1.56 TB 全量下载写成必要前提。 +- [x] 闭合 93 层配置与 checkpoint 拓扑:69 KDA / 24 MLA、1 dense / 92 MoE、96 shards、497,220 tensor entries、247,296 expert packed tensors 与同数 scales、187 组 AttnRes projection / norm。 +- [x] 完成 K3 四联开放工件实验:93 层真实配置条带、routed expert / MLA / MoonViT tensor anatomy、小参数范围审计,以及 FlashKDA 作者 benchmark / 本机编译 / synthetic router 探针边界。 +- [x] 发现并保留官方工件的 `A_log [128]` vs config / remote code / FlashKDA API expected `[96]` 形状不一致;不宣布 checkpoint 损坏,也不把非标准 channel-wise 假设冒充真实 forward。 +- [x] FlashKDA 本机编译边界已实测:RTX 5090 架构受支持,但当前 CUDA 12.8 低于官方 12.9+;g++ 13 已推进到 nvcc,随后因 CUDA headers / glibc declarations 冲突停止,kernel 尚未执行。 +- [x] 第三轮证据快照、可复现探针脚本与正式审计账本已进入开源树;原始权重字节不提交,真实观测 O、推导 D、执行 X、合成 S 与未决 U 分开标记。 +- [x] K3 新四视图本地真实 Chrome 回归通过:93 层条带、tensor group、参数分布、benchmark/router 切换、键盘 tabs、桌面与 390px 移动端均无异常。 ## 正在进行 -- [ ] K3 三轮:使用开放权重与官方实现加入 KDA/AttnRes/MoE 真实 traces、FlashKDA kernel 对照、逐图数值重绘与独立复现。 +- [ ] K3 三轮下一闸门:在匹配 CUDA 12.9+ 环境执行 FlashKDA correctness / benchmark,获得真实 token hidden states、expert load 与 cache traces,再做逐图数值重绘和独立小模型复现。 - [ ] DeepSeek 三轮:真实专家负载、MLA kernel、FP8 / pipeline 与 R1-like RL traces,外加独立小模型复现。 - [ ] 表示、位置与残差二轮:真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。 - [ ] 评测安全二轮:真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。 @@ -278,6 +285,9 @@ | 2026-07-29 | K3 Figure 1–16 / Table 1–5 全部建立课程视觉契约 | 每张图同时写支持范围与不可外推项;作者报告、论文、推导与 toy model 使用 R/P/D/T 标签 | | 2026-07-29 | K3 二轮用八个独立实验闭环 | Delta memory、BF16 decay、AttnRes、LatentMoE、SiTU、QB、MOPD/RL 与 prefix cache 分开操作,不合成伪“架构总分” | | 2026-07-29 | K3 二轮用不可变镜像 `20260729T040336Z-b669615` 发布 | OCI digest `sha256:498b7e43…31cdb3c`;复用 `12010→8080`、NPM host 31 / cert 41、门户 order 180 与公开 Forgejo;十六套生产 Chrome 回归通过,保留 `20260729T031901Z-cd96dab` 回滚 | +| 2026-07-29 | K3 三轮先审计开放工件,不要求加载 1.56 TB | 固定官方 revisions;用 config、index、96 个 shard headers 与两个小范围权重切片闭合真实 topology、shape 和参数统计 | +| 2026-07-29 | 开放工件证据使用 O / D / X / S / U 五种身份 | 观测、推导、本机执行、合成探针与未决矛盾不互相冒充;FlashKDA 作者 benchmark 不写成本机 benchmark | +| 2026-07-29 | `A_log [128]` 与 expected `[96]` 保持未决 | 并列报告 checkpoint、config、remote code 与 kernel API;等待官方 loader / 修订解释,不擅自 reshape | ## 未决问题 diff --git a/README.md b/README.md index 00c159e..6403cdd 100644 --- a/README.md +++ b/README.md @@ -19,8 +19,13 @@ 当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读, 语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化,以及评测与安全深度专题, -以及 67 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、 +以及 71 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、 8 个交互实验和 100 个一手/官方节点,完整覆盖架构、预训练、后训练、系统、评测、案例与附录。 +第三轮已完成首个开放工件里程碑:固定官方模型与 FlashKDA revisions,审计 96 个 checkpoint shards、 +497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计,并用 4 个新视图 +明确区分官方观测、确定性推导、本机执行、合成探针和未决矛盾。详见 +[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与 +[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)。 DeepSeek 二轮专题以 24 张问题账、10 次技术转向、 4 个交互实验和 60 个一手/官方节点,串起 Dense、MoE、MLA、V3 协同、R1 与 V4。 其余专题按进度账本持续扩建。 diff --git a/ROADMAP.md b/ROADMAP.md index c6b9e23..8c25305 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -154,7 +154,8 @@ pass^k、校准、动态基准、代码 Verifier、LLM Judge、Arena、Agent 最 ## 三条贯穿式案例 1. **Kimi K3 解剖**:二轮已完成 32 张问题账、Figure 1–16 / Table 1–5 审计、 - 8 个交互实验与 100 节点阅读链,把全部专题重新汇入架构—预训练—后训练—系统—评测因果链。 + 8 个报告实验与 100 节点阅读链;三轮首个里程碑进一步固定官方 revisions,审计 96 个 checkpoint shards、 + 497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes,并用 4 个工件视图公开复现边界。 2. **DeepSeek 技术谱系**:DeepSeek LLM → DeepSeekMoE → V2/MLA → V3/FP8/MTP/DualPipe → Math/GRPO → R1 → V3.2/DSA → V4 长上下文。 3. **“一个 Token 的旅行”**:从文本分词,经注意力、MoE、GPU 集群、后训练,再到线上推理与工具调用。 diff --git a/experiments/k3/checkpoint_probe.py b/experiments/k3/checkpoint_probe.py new file mode 100644 index 0000000..52de01b --- /dev/null +++ b/experiments/k3/checkpoint_probe.py @@ -0,0 +1,388 @@ +#!/usr/bin/env python3 +"""Build a small, auditable snapshot from Kimi K3's public model artifacts. + +The script intentionally does not download a checkpoint. It consumes: + +1. the public config and safetensors index; +2. safetensors JSON headers fetched with HTTP Range; +3. two small byte ranges containing one KDA parameter prefix and one MoE + router prefix; +4. a local checkout of the official FlashKDA repository. + +Raw model bytes stay local. The generated JSON contains only aggregate +statistics, public shapes, revisions, checksums, and a clearly labelled +synthetic-input router stress probe. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import platform +import re +import subprocess +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +import torch + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--config", type=Path, required=True) + parser.add_argument("--index", type=Path, required=True) + parser.add_argument("--hf-model", type=Path, required=True) + parser.add_argument("--kda-slice", type=Path, required=True) + parser.add_argument("--router-prefix", type=Path, required=True) + parser.add_argument("--mla-header", type=Path, required=True) + parser.add_argument("--vision-header", type=Path, required=True) + parser.add_argument("--flashkda-dir", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--synthetic-tokens", type=int, default=2048) + parser.add_argument("--seed", type=int, default=20260729) + parser.add_argument("--captured-at", default=None) + return parser.parse_args() + + +def read_json(path: Path) -> dict[str, Any]: + return json.loads(path.read_text()) + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for block in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def summarize(values: torch.Tensor) -> dict[str, Any]: + flat = values.detach().float().flatten().cpu() + points = torch.tensor([0, 0.01, 0.1, 0.25, 0.5, 0.75, 0.9, 0.99, 1]) + quantiles = torch.quantile(flat, points).tolist() + labels = ["min", "p01", "p10", "p25", "p50", "p75", "p90", "p99", "max"] + return { + "count": flat.numel(), + "mean": flat.mean().item(), + "std": flat.std().item(), + "quantiles": dict(zip(labels, quantiles, strict=True)), + } + + +def tensor_fact(header: dict[str, Any], name: str) -> dict[str, Any]: + entry = header[name] + return { + "name": name, + "dtype": entry["dtype"], + "shape": entry["shape"], + "bytes": entry["data_offsets"][1] - entry["data_offsets"][0], + } + + +def parse_benchmark(path: Path, heads: int = 96) -> dict[str, Any]: + text = path.read_text() + section = text.split(f"### `T=8192`, `H={heads}`, `D=128`", 1)[1].split("###", 1)[0] + rows = {} + for label, flash, chunk, speedup, gdn, gdn_speedup in re.findall( + r"\| ([^|]+?) \| ([0-9.]+) \| ([0-9.]+) \| ([0-9.]+)× \| ([0-9.]+) \| ([0-9.]+)× \|", + section, + ): + rows[label.strip()] = { + "flash_kda_ms": float(flash), + "fla_chunk_kda_ms": float(chunk), + "speedup_vs_chunk_kda": float(speedup), + "fla_chunk_gdn_ms": float(gdn), + "speedup_vs_gdn": float(gdn_speedup), + } + return {"sequence": 8192, "heads": heads, "dimension": 128, "rows": rows} + + +def load_metrics(load: torch.Tensor) -> dict[str, Any]: + mean = load.mean() + ordered = load.sort().values + count = load.numel() + indices = torch.arange(1, count + 1, device=load.device, dtype=torch.float32) + gini = ((2 * indices - count - 1) * ordered).sum() / (count * ordered.sum()) + quantiles = torch.quantile( + load, + torch.tensor([0, 0.1, 0.25, 0.5, 0.75, 0.9, 0.99, 1], device=load.device), + ).tolist() + labels = ["min", "p10", "p25", "p50", "p75", "p90", "p99", "max"] + return { + "mean": mean.item(), + "std": load.std().item(), + "cv": (load.std() / mean).item(), + "gini": gini.item(), + "zero_experts": int((load == 0).sum()), + "quantiles": dict(zip(labels, quantiles, strict=True)), + } + + +def main() -> None: + args = parse_args() + config = read_json(args.config) + text = config["text_config"] + index = read_json(args.index) + hf_model = read_json(args.hf_model) + mla_header = read_json(args.mla_header) + vision_header = read_json(args.vision_header) + + names = list(index["weight_map"]) + shard_files = [ + item + for item in hf_model["siblings"] + if re.fullmatch(r"model-\d+-of-\d+\.safetensors", item["rfilename"]) + ] + shard_sizes = [item["size"] for item in shard_files] + + kda_raw = args.kda_slice.read_bytes() + if len(kda_raw) != 49_664: + raise ValueError(f"unexpected KDA slice length: {len(kda_raw)}") + a_log = torch.frombuffer(bytearray(kda_raw[:512]), dtype=torch.float32).clone() + dt_bias = torch.frombuffer(bytearray(kda_raw[512:]), dtype=torch.float32).clone().view(96, 128) + + router_raw = args.router_prefix.read_bytes() + if len(router_raw) != 13_488_640: + raise ValueError(f"unexpected router prefix length: {len(router_raw)}") + correction_bias = torch.frombuffer( + bytearray(router_raw[:3584]), dtype=torch.float32 + ).clone() + router_weight = torch.frombuffer( + bytearray(router_raw[643_584:13_488_640]), dtype=torch.bfloat16 + ).clone().view(896, 7168) + + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + torch.manual_seed(args.seed) + if device.type == "cuda": + torch.cuda.manual_seed_all(args.seed) + router_device = router_weight.to(device) + bias_device = correction_bias.to(device) + synthetic = torch.randn( + args.synthetic_tokens, 7168, device=device, dtype=torch.float32 + ) + synthetic *= torch.rsqrt(synthetic.square().mean(-1, keepdim=True) + 1e-6) + scores = torch.sigmoid(synthetic.to(torch.bfloat16) @ router_device.T).float() + unbiased_ids = scores.topk(16, dim=-1).indices + biased_ids = (scores + bias_device).topk(16, dim=-1).indices + + def loads(ids: torch.Tensor) -> torch.Tensor: + return torch.bincount(ids.flatten(), minlength=896).float() + + unbiased_load = loads(unbiased_ids) + biased_load = loads(biased_ids) + overlap = torch.tensor( + [ + len(set(unbiased_ids[row].tolist()) & set(biased_ids[row].tolist())) + for row in range(args.synthetic_tokens) + ], + dtype=torch.float32, + ) + router_norms = router_weight.float().norm(dim=1) + + # This is deliberately a hypothesis probe, not a canonical forward pass: + # the checkpoint stores A_log[128], while public code/API expect A_log[H=96]. + channelwise_log_decay = -5.0 * torch.sigmoid(torch.exp(a_log).view(1, 128) * dt_bias) + channelwise_retention = torch.exp(channelwise_log_decay) + + per_expert_bytes = 3 * (5_505_024 + 344_064) + all_routed_expert_bytes = per_expert_bytes * 896 * 92 + kda_layers = text["linear_attn_config"]["kda_layers"] + mla_layers = text["linear_attn_config"]["full_attn_layers"] + + flash_revision = subprocess.check_output( + ["git", "-C", str(args.flashkda_dir), "rev-parse", "HEAD"], + text=True, + ).strip() + + captured_at = args.captured_at or datetime.now(timezone.utc).isoformat() + result = { + "schema_version": 1, + "captured_at": captured_at, + "evidence_boundary": { + "checkpoint_forward_run": False, + "raw_weights_committed": False, + "router_inputs": "deterministic synthetic RMS-normalized vectors, not token hidden states", + "kda_retention_probe": "noncanonical channel-wise interpretation used only to expose the A_log shape ambiguity", + }, + "provenance": { + "huggingface_model": "moonshotai/Kimi-K3", + "huggingface_revision": hf_model["sha"], + "flashkda_revision": flash_revision, + "sha256": { + "config": sha256(args.config), + "index": sha256(args.index), + "kda_slice": sha256(args.kda_slice), + "router_prefix": sha256(args.router_prefix), + }, + }, + "checkpoint": { + "tensor_data_bytes": index["metadata"]["total_size"], + "tensor_data_tb": index["metadata"]["total_size"] / 1e12, + "tensor_data_tib": index["metadata"]["total_size"] / 2**40, + "shards": len(shard_files), + "shard_file_bytes": { + "sum": sum(shard_sizes), + "min": min(shard_sizes), + "max": max(shard_sizes), + "mean": sum(shard_sizes) / len(shard_sizes), + }, + "tensor_entries": len(names), + "tensor_counts": { + "expert_packed": sum( + bool(re.search(r"experts\.\d+\.w[123]\.weight_packed$", name)) + for name in names + ), + "expert_scales": sum( + bool(re.search(r"experts\.\d+\.w[123]\.weight_scale$", name)) + for name in names + ), + "router_weight": sum( + name.endswith("block_sparse_moe.gate.weight") for name in names + ), + "router_correction_bias": sum( + name.endswith("gate.e_score_correction_bias") for name in names + ), + "attnres_proj": sum( + bool(re.search(r"(_res_proj|output_attn_res_proj)\.weight$", name)) + for name in names + ), + "attnres_norm": sum( + bool(re.search(r"(_res_norm|output_attn_res_norm)\.weight$", name)) + for name in names + ), + "kda_a_log": sum(name.endswith("self_attn.A_log") for name in names), + "kda_dt_bias": sum(name.endswith("self_attn.dt_bias") for name in names), + "vision": sum(name.startswith("vision_tower.") for name in names), + "projector": sum(name.startswith("mm_projector.") for name in names), + }, + "derived_routed_expert_bytes": all_routed_expert_bytes, + "derived_routed_expert_share": all_routed_expert_bytes + / index["metadata"]["total_size"], + }, + "configuration": { + "layers": text["num_hidden_layers"], + "dense_layers": text["first_k_dense_replace"], + "hidden": text["hidden_size"], + "vocabulary": text["vocab_size"], + "context": text["max_position_embeddings"], + "kda_layers": kda_layers, + "mla_layers": mla_layers, + "heads": text["num_attention_heads"], + "head_dim": text["linear_attn_config"]["head_dim"], + "attnres_block": text["attn_res_block_size"], + "experts": text["num_experts"], + "active_experts": text["num_experts_per_token"], + "shared_experts": text["num_shared_experts"], + "latent_width": text["routed_expert_hidden_size"], + "expert_intermediate": text["moe_intermediate_size"], + "situ_beta": text["activation_situ_beta"], + "situ_linear_beta": text["activation_situ_linear_beta"], + "mla_nope": text["mla_use_nope"], + "mla_output_gate": text["mla_use_output_gate"], + }, + "tensor_examples": { + "mla_layer_4": [ + tensor_fact( + mla_header, + "language_model.model.layers.3.self_attn.kv_a_proj_with_mqa.weight", + ), + tensor_fact( + mla_header, + "language_model.model.layers.3.self_attn.kv_b_proj.weight", + ), + tensor_fact( + mla_header, + "language_model.model.layers.3.self_attn.q_a_proj.weight", + ), + tensor_fact( + mla_header, + "language_model.model.layers.3.self_attn.q_b_proj.weight", + ), + tensor_fact( + mla_header, + "language_model.model.layers.3.self_attn.g_proj.weight", + ), + ], + "vision": [ + tensor_fact(vision_header, "vision_tower.patch_embed.proj.weight"), + tensor_fact(vision_header, "vision_tower.patch_embed.pos_emb.weight"), + tensor_fact(vision_header, "vision_tower.encoder.blocks.0.wqkv.weight"), + tensor_fact(vision_header, "vision_tower.encoder.blocks.26.wqkv.weight"), + tensor_fact(vision_header, "vision_tower.encoder.final_layernorm.weight"), + ], + "routed_expert_0": [ + {"name": "w1.weight_packed", "dtype": "U8", "shape": [3072, 1792], "bytes": 5_505_024}, + {"name": "w1.weight_scale", "dtype": "U8", "shape": [3072, 112], "bytes": 344_064}, + {"name": "w2.weight_packed", "dtype": "U8", "shape": [3584, 1536], "bytes": 5_505_024}, + {"name": "w2.weight_scale", "dtype": "U8", "shape": [3584, 96], "bytes": 344_064}, + {"name": "w3.weight_packed", "dtype": "U8", "shape": [3072, 1792], "bytes": 5_505_024}, + {"name": "w3.weight_scale", "dtype": "U8", "shape": [3072, 112], "bytes": 344_064}, + ], + }, + "parameter_audit": { + "a_log_checkpoint_shape": [128], + "a_log_public_code_shape": [96], + "dt_bias_shape": [96, 128], + "beta_projection_shape": [96, 7168], + "status": "observed shape inconsistency; runtime meaning unresolved", + "a_log": summarize(a_log), + "a_rate_exp": summarize(torch.exp(a_log)), + "dt_bias": summarize(dt_bias), + "channelwise_hypothesis": { + "log_decay": summarize(channelwise_log_decay), + "one_step_retention": summarize(channelwise_retention), + "retention_after_64_steps": summarize(channelwise_retention.pow(64)), + }, + "router_correction_bias": summarize(correction_bias), + "router_row_l2": summarize(router_norms), + "router_bias_norm_correlation": torch.corrcoef( + torch.stack([correction_bias, router_norms]) + )[0, 1].item(), + }, + "router_stress_probe": { + "seed": args.seed, + "synthetic_tokens": args.synthetic_tokens, + "hidden_rms": synthetic.square().mean().sqrt().item(), + "without_correction_bias": load_metrics(unbiased_load), + "with_correction_bias": load_metrics(biased_load), + "membership_overlap_mean": overlap.mean().item(), + "tokens_changed": int((overlap < 16).sum()), + "changed_fraction": (overlap < 16).float().mean().item(), + "mean_replacements_per_token": (16 - overlap).mean().item(), + }, + "flashkda": { + "supported_architectures": ["90a", "100a", "103a", "120a"], + "requirements": {"cuda": ">=12.9", "pytorch": ">=2.4", "gpu": "SM90+"}, + "official_benchmarks": { + "h20": parse_benchmark(args.flashkda_dir / "BENCHMARK_H20.md"), + "gb200": parse_benchmark(args.flashkda_dir / "BENCHMARK_GB200.md"), + }, + "local_environment": { + "python": platform.python_version(), + "torch": torch.__version__, + "torch_cuda": torch.version.cuda, + "gpu": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None, + "capability": list(torch.cuda.get_device_capability(0)) + if torch.cuda.is_available() + else None, + "libc": list(platform.libc_ver()), + }, + "local_build": { + "status": "blocked_before_kernel execution", + "attempt_1": "system g++ 15 exceeds CUDA 12.8 host compiler range", + "attempt_2": "temporary g++ 13 reaches nvcc, then CUDA 12.8 headers conflict with current glibc math declarations", + "interpretation": "GPU architecture is listed by the repository, but the local CUDA 12.8 stack is below the official CUDA 12.9 requirement", + }, + }, + } + + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(result, ensure_ascii=False, indent=2) + "\n") + print(json.dumps({"output": str(args.output), "bytes": args.output.stat().st_size})) + + +if __name__ == "__main__": + main() diff --git a/research/K3_ARTIFACT_AUDIT.md b/research/K3_ARTIFACT_AUDIT.md new file mode 100644 index 0000000..2c1b686 --- /dev/null +++ b/research/K3_ARTIFACT_AUDIT.md @@ -0,0 +1,367 @@ +# Kimi K3 第三轮:开放工件、权重元数据与可复现实验审计 + +> 研究截点:2026-07-29 +> 官方模型修订:`moonshotai/Kimi-K3@9f62e4e9fffbd0a83ddd60e1c209d828994b3569` +> 官方 FlashKDA 修订:`MoonshotAI/FlashKDA@1ce47ea3bb22c84eb9cc665028399cf35e8ffb0b` +> 原则:模型卡、配置、远程代码、safetensors header、kernel API、作者 benchmark 和本机实验分开记账。 + +## 1. 本轮究竟要推进什么 + +第二轮已经把 47 页报告拆成问题、公式、图表和教学实验。第三轮不能只是再写一遍报告,而要回答: + +1. 开放权重仓库实际包含哪些文件、分片和张量? +2. 报告的 `69 KDA + 24 MLA`、`1 dense + 92 MoE` 是否能在配置和 tensor names 中闭合? +3. Stable LatentMoE 与 MXFP4 在真实 checkpoint 中怎样落成 packed tensors 与 scales? +4. `NoPE` 在公开实现中究竟是“删掉位置相关通道”,还是“不施加 rotary transform”? +5. AttnRes 在 checkpoint 中留下多少 projection / norm 参数? +6. FlashKDA 的公开接口、支持架构、作者 benchmark 和本机可运行边界分别是什么? +7. 哪些可称为真实工件 trace,哪些仍只是确定性推导或 synthetic probe? + +本轮不声称完成以下事情: + +- 没有在单张 RTX 5090 上加载 1.56 TB checkpoint; +- 没有获得真实 token hidden states、线上 expert load 或生产 cache trace; +- 没有把随机向量上的 router 行为写成真实数据分布; +- 没有把未成功执行的 FlashKDA kernel 写成“本机 benchmark”; +- 没有替官方解释下面发现的 `A_log` 形状不一致。 + +## 2. 一手工件与校验 + +| 工件 | canonical source | 本地校验 | +|---|---|---| +| 模型配置 | `https://huggingface.co/moonshotai/Kimi-K3/blob/main/config.json` | `9710e121…379213` | +| HF configuration code | `configuration_kimi_k3.py` | `735eb9eb…b416ae` | +| 多模态 modeling code | `modeling_kimi_k3.py` | `b9171c96…65ea2` | +| 文本 backbone code | `modeling_kimi_linear.py` | `9e3564c7…ff44a` | +| tensor index | `model.safetensors.index.json` | `a1c52106…7febd` | +| layer-1 KDA 小范围字节 | shard 1 HTTP Range | `dccb10e7…d9b2` | +| layer-2 router prefix | shard 2 HTTP Range | `dbe66ff8…23ee` | +| FlashKDA | `https://github.com/MoonshotAI/FlashKDA` | git `1ce47ea3…f0b` | + +完整校验值保存在 `src/data/k3-artifact-snapshot.json`。原始权重字节不进入公开仓库。 + +### 2.1 为什么只读 Range + +模型 tensor data 为 `1,560,860,324,864` bytes,即: + +- `1.56086 TB`(十进制); +- `1.41959 TiB`(二进制); +- 96 个 safetensors shards; +- 497,220 个 tensor entries。 + +当前机器是 32 GB RTX 5090、123 GiB RAM。完整下载和加载既不必要,也不能支持“单卡复现”。 +safetensors 把 JSON header 放在每个分片开头;header 给出 tensor name、dtype、shape 与 byte offsets。 +因此可以只读: + +```text +8-byte header length +→ JSON header +→ selected tensor byte ranges +``` + +这足以审计结构,也能读取少量参数做统计,同时避免把 1.56 TB 下载行为伪装成研究必要条件。 + +## 3. checkpoint 拓扑:配置、索引与 header 闭合 + +### 3.1 层型 + +配置精确给出: + +- 93 层; +- KDA layers:69; +- full-attention / Gated MLA layers:24; +- MLA 位于 1-based `4, 8, …, 92, 93`; +- 第一层 dense,后 92 层为 MoE; +- AttnRes block size 为 12。 + +因此真实层条带是: + +```text +L01 KDA + dense FFN +L02 KDA + MoE +L03 KDA + MoE +L04 MLA + MoE +... +L92 MLA + MoE +L93 MLA + MoE +``` + +最后连续两个 MLA 不是排版错误:`L92` 是最后一个 3:1 block 的 MLA,`L93` 是报告所述额外末层 MLA。 + +### 3.2 tensor counts + +| 对象 | index 实际计数 | 为什么是这个数 | +|---|---:|---| +| KDA `A_log` | 69 | 每个 KDA layer 一份 | +| KDA `dt_bias` | 69 | 每个 KDA layer 一份 | +| Router weight | 92 | 每个 MoE layer 一份 | +| Router correction bias | 92 | 每个 MoE layer 一份 | +| Routed expert packed weights | 247,296 | `92 × 896 × 3` | +| Routed expert scales | 247,296 | `92 × 896 × 3` | +| AttnRes projections | 187 | `93 × 2 + output 1` | +| AttnRes norms | 187 | `93 × 2 + output 1` | +| Vision tensors | 165 | 27-layer MoonViT-V2 与输入/输出项 | +| Projector tensors | 3 | 视觉塔到 7168 text hidden | + +`497,220` 不是“参数数量”,而是 safetensors 中的命名 tensor entry 数。不能和 `2.78T parameters` +混为一个口径。 + +## 4. 真实 tensor shape:架构不再只靠报告表格 + +### 4.1 Dense layer 1 的 KDA + +shard 1 header 给出: + +| tensor | dtype | shape | +|---|---|---| +| `q/k/v/g_proj.weight` | BF16 | `12288 × 7168` | +| `o_proj.weight` | BF16 | `7168 × 12288` | +| `q/k/v_conv1d.weight` | F32 | `12288 × 1 × 4` | +| `b_proj.weight` | BF16 | `96 × 7168` | +| `f_a_proj.weight` | BF16 | `128 × 7168` | +| `f_b_proj.weight` | BF16 | `12288 × 128` | +| dense FFN gate/up | BF16 | `33792 × 7168` | +| dense FFN down | BF16 | `7168 × 33792` | + +`12288 = 96 heads × 128 dims`;ShortConv kernel size 4 也直接出现在 checkpoint shape 中。 + +### 4.2 MLA layer 4 + +shard 4 header 给出: + +| tensor | shape | +|---|---| +| `q_a_proj` | `1536 × 7168` | +| `q_b_proj` | `18432 × 1536` | +| `kv_a_proj_with_mqa` | `576 × 7168` | +| `kv_b_proj` | `24576 × 512` | +| `g_proj` | `12288 × 7168` | +| `o_proj` | `7168 × 12288` | + +关键解释: + +- `576 = 512 KV latent + 64 auxiliary q/k channel`; +- `q_b` 输出 `96 × (128 + 64) = 18432`; +- 公开 forward 没有对这 64 维施加 RoPE,但仍投影、拼接并进入 attention; +- 因此 K3 的 `mla_use_nope=true` 应解释为“不施加显式 rotary transform”,不能改写成“MLA 中完全不存在任何额外 q/k 通道”。 + +### 4.3 一个 routed expert 的 MXFP4 表示 + +layer 2 / expert 0 的 header: + +| tensor | dtype | packed shape | bytes | +|---|---|---:|---:| +| `w1.weight_packed` | U8 | `3072 × 1792` | 5,505,024 | +| `w1.weight_scale` | U8 | `3072 × 112` | 344,064 | +| `w2.weight_packed` | U8 | `3584 × 1536` | 5,505,024 | +| `w2.weight_scale` | U8 | `3584 × 96` | 344,064 | +| `w3.weight_packed` | U8 | `3072 × 1792` | 5,505,024 | +| `w3.weight_scale` | U8 | `3072 × 112` | 344,064 | + +这里能直接读出两个结构: + +1. routed expert 工作在 `3584` latent width,而不是 `7168` full hidden; +2. scale group size 为 32:`3584 / 32 = 112`,`3072 / 32 = 96`。 + +每个 routed expert 的 packed weights + scales 为 `17,547,264` bytes。按配置与 index 的统一形状推导, +92 层全部 routed experts 约占 tensor data 的 `92.67%`。这是 header + 配置的确定性推导,不是运行显存。 + +### 4.4 MoonViT-V2 + +shard 96 header: + +| tensor | shape | +|---|---| +| patch projection | `1024 × 3 × 14 × 14` | +| learned 2D position table | `64 × 64 × 1024` | +| block 0 / 26 QKV | `4608 × 1024` | +| block 0 / 26 MLP up | `4096 × 1024` | +| block 0 / 26 MLP down | `1024 × 4096` | +| final norm | `1024` | + +配置、首层和末层 tensor 同时支持“27 layers、patch 14、vision hidden 1024”;这比只引用模型卡更强。 + +## 5. 小参数 Range audit + +### 5.1 KDA layer 1 + +只读 shard 1 开头的 49,664 bytes: + +- checkpoint `A_log`: F32 `[128]`; +- `dt_bias`: F32 `[12288]`,可按公开配置写成 `[96, 128]`。 + +真实参数统计: + +| 参数 | median | p10–p90 | min–max | +|---|---:|---:|---:| +| `A_log` | −0.1533 | −0.5367–0.0743 | −0.7531–2.4661 | +| `exp(A_log)` | 0.8579 | 0.5847–1.0771 | 0.4709–11.7764 | +| `dt_bias` | −4.6220 | −6.4912–−2.5690 | −7.8938–0.1792 | + +### 5.2 一个必须公开保留的形状不一致 + +四份官方工件目前给出: + +| 工件 | 观测 | +|---|---| +| `config.json` | `num_heads=96`, `head_dim=128` | +| HF remote code | `A_log` 初始化 shape 为 `[num_heads]`,即 `[96]` | +| FlashKDA API / C++ check | `A_log` 必须 `[H]` | +| checkpoint shard header | `A_log` 实际为 `[128]` | + +同时: + +- `dt_bias` 可闭合为 `[96,128]`; +- `b_proj.weight` 为 `[96,7168]`,β 显然按 96 heads; +- q/k/v projection 为 `12288 = 96×128`。 + +因此,**公开工件中存在可复现的 `A_log [128]` vs expected `[96]` 形状不一致**。 + +当前可下的结论只有: + +- 这是 header / code / config / kernel API 四方直接观测,不是 Grok 猜测; +- 公开 remote code 按字面构造时会期待 `[96]`; +- 需要 Moonshot、实际 vLLM/SGLang loader 或后续权重修订解释转换规则。 + +当前不能下的结论: + +- 不能直接宣布 checkpoint 损坏; +- 不能擅自把 `[128]` 解释为 per-channel `A_log`; +- 不能用某个猜测 reshape 得到的曲线冒充模型真实 retention。 + +数据快照保留了一个明确标成 `noncanonical channel-wise hypothesis` 的数值探针,只用于说明: +若把 `[128]` 当 channel 参数,能得到怎样的 retention 分布;它不进入正式 forward 结论。 + +## 6. Router:真实权重与 synthetic input 必须分层 + +读取 layer 2 的: + +- correction bias:896 个 F32; +- router weight:`896 × 7168` BF16,约 12.85 MB。 + +真实参数统计: + +- correction bias median `0.00493`,min `−0.08442`,max `0.02861`; +- router row L2 median `5.0364`,min `2.6019`,max `7.0232`; +- bias 与 row norm 的 Pearson correlation `0.4567`。 + +为了测试“只拿真实 router weights 是否就能评价 Quantile Balancing”,脚本生成 2,048 个固定 seed、 +RMS=1 的各向同性随机 hidden vectors,再比较 top-16: + +| 条件 | load CV | Gini | zero-load experts | +|---|---:|---:|---:| +| 不加 correction bias | 2.085 | 0.831 | 558 | +| 加 checkpoint correction bias | 2.529 | 0.879 | 673 | + +两组 top-16 平均只重合 `2.17 / 16`。 + +这不是在证明 QB 让真实负载更差,反而证明: + +1. router weights 与 hidden-state distribution 是共同训练的; +2. sigmoid top scores 在随机 RMS=1 输入下容易饱和; +3. 小 correction bias 会在饱和的近并列区域强烈改写名次; +4. 没有真实 hidden traces,就不能用随机向量评价真实负载均衡。 + +因此网站把这组结果命名为 **counterexample / synthetic stress probe**,不是“真实 expert load trace”。 + +## 7. FlashKDA:作者 benchmark 与本机实验分开 + +### 7.1 官方仓库事实 + +FlashKDA `1ce47ea3`: + +- CUTLASS kernels; +- 支持 `90a / 100a / 103a / 120a`; +- README 要求 SM90+、CUDA 12.9+、PyTorch 2.4+; +- kernel API 固定 `K=V=128`; +- q/k/v/g 为 BF16,`A_log` / `dt_bias` 为 F32; +- 支持 fixed length、variable length、initial / final recurrent state。 + +官方报告的 `T=8192, H=96, D=128`: + +| device | case | FlashKDA | FLA chunk KDA | 作者报告 speedup | +|---|---|---:|---:|---:| +| H20 | fixed | 2.6220 ms | 4.8388 ms | 1.85× | +| H20 | 8×1024 varlen | 2.0432 ms | 4.6723 ms | 2.29× | +| GB200 | fixed | 1.0087 ms | 2.3271 ms | 2.31× | +| GB200 | 8×1024 varlen | 0.7064 ms | 2.3105 ms | 3.27× | + +这些是作者仓库 benchmark,不是本站复跑值。 + +### 7.2 本机真实构建边界 + +本机: + +- RTX 5090,compute capability `12.0`; +- PyTorch `2.11.0+cu128`; +- PyTorch CUDA `12.8`; +- 官方源码明确包含 `sm_120a`,所以不是 GPU architecture 缺失。 + +两次可复现构建: + +1. 系统 `g++ 15.2`:PyTorch extension 在编译前拒绝,CUDA 12.8 要求 host compiler `<14`; +2. 临时解包 `g++ 13.4`:成功进入 nvcc,但 CUDA 12.8 headers 与当前 glibc math declarations + 在 `cospi / sinpi / rsqrt` exception specification 处冲突。 + +结论: + +- kernel 尚未在本站机器执行; +- 失败与 README 的 CUDA 12.9+ 要求一致; +- 不能把 `sm_120a` 支持写成本机已经跑通; +- 下一次应使用匹配 PyTorch 的 CUDA 12.9+ toolchain 或官方容器后再复跑 correctness + benchmark。 + +## 8. 可复现实验入口 + +脚本: + +```text +experiments/k3/checkpoint_probe.py +``` + +提交的数据快照: + +```text +src/data/k3-artifact-snapshot.json +``` + +脚本会: + +1. 解析 config、index 与 selected headers; +2. 校验小范围字节长度和 SHA-256; +3. 统计真实 KDA / router 参数; +4. 运行明确标注的 synthetic router counterexample; +5. 解析 FlashKDA 官方 H20 / GB200 benchmark; +6. 输出本机环境与构建边界; +7. 不提交原始权重。 + +## 9. 网站实现合同 + +第三轮开放工件实验室必须有四个视图: + +1. **Layer map**:93 层真实 config 条带;显示 KDA/MLA、dense/MoE、AttnRes block。 +2. **Tensor anatomy**:checkpoint / expert / vision / MLA tensor shape 与数量。 +3. **Parameter audit**:真实 Range statistics,并把 `A_log` mismatch 放在主视区。 +4. **Reproduction boundary**:官方 benchmark、本站构建失败点、synthetic router counterexample。 + +每个视图必须显示证据类型: + +- `O` = official artifact observation; +- `D` = deterministic derivation; +- `X` = executed local experiment; +- `S` = synthetic stress probe; +- `U` = unresolved inconsistency。 + +## 10. 下一轮证据闸门 + +- [x] 官方 config / code / index revision 固定; +- [x] 96 个分片与 497,220 tensor entries 审计; +- [x] KDA / MLA / MoE / AttnRes / Vision tensor shapes 入账; +- [x] selected open-weight ranges 做真实参数统计; +- [x] 发现并限定 `A_log` shape inconsistency; +- [x] FlashKDA RTX 5090 构建尝试留下可复现边界; +- [ ] 使用 CUDA 12.9+ 匹配环境跑 FlashKDA exact correctness; +- [ ] 取得真实 hidden-state / router load trace; +- [ ] 取得可加载的 reduced checkpoint、官方 trace 或多机资源; +- [ ] 对 Figure 3 / 4 / 5 做真实数值重绘; +- [ ] 对 AttnRes 读取分布做真实 token / layer trace。 + diff --git a/scripts/check-k3-browser.mjs b/scripts/check-k3-browser.mjs index 0a2c90d..54ec668 100644 --- a/scripts/check-k3-browser.mjs +++ b/scripts/check-k3-browser.mjs @@ -75,6 +75,10 @@ const overview = await evaluate(`(() => ({ paperGroups: document.querySelectorAll("#papers .paper-group").length, labTabs: document.querySelectorAll("[data-k3-tab]").length, labPanels: document.querySelectorAll("[data-k3-panel]").length, + artifactTabs: document.querySelectorAll("[data-artifact-tab]").length, + artifactPanels: document.querySelectorAll("[data-artifact-panel]").length, + artifactLayers: document.querySelectorAll("[data-layer-cell]").length, + artifactMismatch: document.querySelector("#artifacts")?.textContent.includes("A_log [128] ≠ expected [96]"), nativeVisionCorrected: document.body.textContent.includes("MoonViT‑V2 从头训练") && document.body.textContent.includes("同一个 next-token prediction objective"), staleVisionClaim: document.body.textContent.includes("先固定语言模型训练视觉组件"), @@ -159,12 +163,100 @@ const labs = await evaluate(`(() => { }; })()`); +const artifacts = await evaluate(`(() => { + const root = document.querySelector("[data-k3-artifact-lab]"); + const panel = () => root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel; + const text = (selector) => root.querySelector(selector).textContent.trim(); + const input = (selector, value) => { + const node = root.querySelector(selector); + node.value = value; + node.dispatchEvent(new Event("input", { bubbles: true })); + }; + + const initial = { + panel: panel(), + layer: text("[data-layer-number]"), + attention: text("[data-layer-attention]"), + ffn: text("[data-layer-ffn]"), + block: text("[data-layer-block]"), + }; + input("[data-layer-slider]", "93"); + const terminal = { + layer: text("[data-layer-number]"), + attention: text("[data-layer-attention]"), + ffn: text("[data-layer-ffn]"), + copy: text("[data-layer-special]"), + }; + + root.querySelector('[data-artifact-tab="tensors"]').click(); + const tensors = { + panel: panel(), + groups: root.querySelectorAll("[data-tensor-tab]").length, + expertRows: root.querySelector('[data-tensor-panel="expert"]').querySelectorAll(":scope > div").length, + entries: root.textContent.includes("497,220"), + share: root.textContent.includes("92.67%"), + }; + root.querySelector('[data-tensor-tab="mla"]').click(); + const mla = { + visible: !root.querySelector('[data-tensor-panel="mla"]').hidden, + rows: root.querySelector('[data-tensor-panel="mla"]').querySelectorAll(":scope > div").length, + has576: root.querySelector('[data-tensor-panel="mla"]').textContent.includes("576 × 7168"), + }; + + root.querySelector('[data-artifact-tab="parameters"]').click(); + const parameterInitial = { + panel: panel(), + shape: text("[data-parameter-shape]"), + conflict: root.textContent.includes("A_log [128]") && root.textContent.includes("A_log [H] = [96]"), + }; + input("[data-parameter-select]", "dt"); + const parameterChanged = { + shape: text("[data-parameter-shape]"), + count: text("[data-parameter-count]"), + }; + + root.querySelector('[data-artifact-tab="reproduction"]').click(); + const reproductionInitial = { + panel: panel(), + flash: text("[data-benchmark-flash]"), + fla: text("[data-benchmark-fla]"), + speedup: text("[data-benchmark-speedup]"), + cv: text("[data-router-cv]"), + zero: text("[data-router-zero]"), + }; + input("[data-benchmark-device]", "gb200"); + input("[data-benchmark-case]", "Varlen, \\\`seq_lens\\\`=\\\`1024 x 8\\\`"); + input("[data-router-mode]", "bias"); + const reproductionChanged = { + flash: text("[data-benchmark-flash]"), + speedup: text("[data-benchmark-speedup]"), + cv: text("[data-router-cv]"), + zero: text("[data-router-zero]"), + }; + + const first = root.querySelector('[data-artifact-tab="layers"]'); + first.focus(); + first.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowRight", bubbles: true })); + return { + initial, terminal, tensors, mla, parameterInitial, parameterChanged, + reproductionInitial, reproductionChanged, + keyboardSelected: root.querySelector('[data-artifact-tab][aria-selected="true"]').dataset.artifactTab, + keyboardVisible: panel(), + }; +})()`); + await evaluate(`(() => { document.querySelector("[data-k3-lab]").scrollIntoView({ block: "start", behavior: "instant" }); window.scrollBy(0, -82); })()`); await pause(180); await screenshot("/tmp/llm-atlas-k3-lab-desktop.png"); +await evaluate(`(() => { + document.querySelector("[data-k3-artifact-lab]").scrollIntoView({ block: "start", behavior: "instant" }); + window.scrollBy(0, -82); +})()`); +await pause(180); +await screenshot("/tmp/llm-atlas-k3-artifact-desktop.png"); await command("Emulation.setDeviceMetricsOverride", { width: 390, @@ -183,8 +275,10 @@ const mobile = await evaluate(`(() => { menuVisible: getComputedStyle(toggle).display !== "none", menuOpen: toggle.getAttribute("aria-expanded"), tabs: root.querySelectorAll("[data-k3-tab]").length, + artifactTabs: document.querySelectorAll("[data-artifact-tab]").length, + artifactLayers: document.querySelectorAll("[data-layer-cell]").length, offenders: [...document.querySelectorAll("body *")] - .filter((node) => !node.closest(".paper-chain, .spec-table-wrap, .cache-strip, .architecture-explorer, [data-k3-lab]")) + .filter((node) => !node.closest(".paper-chain, .spec-table-wrap, .cache-strip, .architecture-explorer, [data-k3-lab], [data-k3-artifact-lab]")) .filter((node) => node.getBoundingClientRect().right > document.documentElement.clientWidth + 1) .slice(0, 15) .map((node) => ({ @@ -201,17 +295,24 @@ await evaluate(`(() => { })()`); await pause(180); await screenshot("/tmp/llm-atlas-k3-mobile.png"); +await evaluate(`(() => { + document.querySelector("[data-k3-artifact-lab]").scrollIntoView({ block: "start", behavior: "instant" }); + window.scrollBy(0, -64); +})()`); +await pause(180); +await screenshot("/tmp/llm-atlas-k3-artifact-mobile.png"); -const report = { overview, labs, mobile, exceptions }; +const report = { overview, labs, artifacts, mobile, exceptions }; console.log(JSON.stringify(report, null, 2)); const numeric = (text) => Number.parseFloat(text.replaceAll(",", "").replace("−", "-")); const failures = []; if (!overview.title.includes("因果环节")) failures.push("K3 二轮标题异常"); -if (overview.sections !== 31 || overview.tocLinks !== 31) failures.push("30 个编号专题加阅读链的目录结构异常"); +if (overview.sections !== 32 || overview.tocLinks !== 32) failures.push("31 个编号专题加阅读链的目录结构异常"); if (overview.ledgers !== 32 || overview.reportMap !== 9) failures.push("32 张问题账或报告地图异常"); if (overview.figureAtlas !== 21 || overview.paperLinks !== 100 || overview.paperGroups < 12) failures.push("图表审计或 100 节点阅读链异常"); if (overview.labTabs !== 8 || overview.labPanels !== 8) failures.push("八联实验结构异常"); +if (overview.artifactTabs !== 4 || overview.artifactPanels !== 4 || overview.artifactLayers !== 93 || !overview.artifactMismatch) failures.push("开放工件四视图、93 层条带或形状冲突异常"); if (!overview.nativeVisionCorrected || overview.staleVisionClaim) failures.push("原生多模态纠错未生效或旧错误残留"); if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出"); if (labs.memoryInitial.panel !== "memory" || numeric(labs.memoryInitial.additiveError) <= numeric(labs.memoryInitial.deltaError)) failures.push("Delta memory 初始递推异常"); @@ -227,7 +328,16 @@ if (!labs.rlOver.budget.includes("reward 改为 −1")) failures.push("Reasoning if (numeric(labs.cacheInitial.hit) !== 2560 || !labs.cacheInitial.recompute.includes("256")) failures.push("Hybrid prefix cache 默认命中异常"); if (numeric(labs.cacheSparse.hit) >= numeric(labs.cacheInitial.hit) || numeric(labs.cacheSparse.recompute) <= numeric(labs.cacheInitial.recompute)) failures.push("稀疏 KDA checkpoint 未降低 joint hit"); if (labs.keyboardSelected !== "decay" || labs.keyboardVisible !== "decay") failures.push("实验键盘 tab 导航异常"); -if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 8) failures.push("移动端导航或实验异常"); +if (artifacts.initial.panel !== "layers" || numeric(artifacts.initial.layer) !== 1 || artifacts.initial.attention !== "KDA" || artifacts.initial.ffn !== "DENSE") failures.push("开放工件初始层视图异常"); +if (numeric(artifacts.terminal.layer) !== 93 || artifacts.terminal.attention !== "MLA" || artifacts.terminal.ffn !== "MOE" || !artifacts.terminal.copy.includes("L92 / L93")) failures.push("K3 末层真实配置条带异常"); +if (artifacts.tensors.panel !== "tensors" || artifacts.tensors.groups !== 3 || artifacts.tensors.expertRows !== 6 || !artifacts.tensors.entries || !artifacts.tensors.share) failures.push("checkpoint tensor anatomy 异常"); +if (!artifacts.mla.visible || artifacts.mla.rows !== 5 || !artifacts.mla.has576) failures.push("MLA header shape 视图异常"); +if (artifacts.parameterInitial.panel !== "parameters" || artifacts.parameterInitial.shape !== "[128] F32" || !artifacts.parameterInitial.conflict) failures.push("A_log 工件冲突审计异常"); +if (artifacts.parameterChanged.shape !== "[96,128] F32" || !artifacts.parameterChanged.count.includes("12,288")) failures.push("真实 dt_bias 参数切换异常"); +if (artifacts.reproductionInitial.panel !== "reproduction" || numeric(artifacts.reproductionInitial.speedup) !== 1.85 || numeric(artifacts.reproductionInitial.cv) < 2) failures.push("FlashKDA H20 或 router 初始探针异常"); +if (numeric(artifacts.reproductionChanged.speedup) !== 3.27 || numeric(artifacts.reproductionChanged.flash) !== 0.7064 || numeric(artifacts.reproductionChanged.cv) <= numeric(artifacts.reproductionInitial.cv) || numeric(artifacts.reproductionChanged.zero) <= numeric(artifacts.reproductionInitial.zero)) failures.push("GB200 benchmark 或 synthetic router counterexample 未更新"); +if (artifacts.keyboardSelected !== "tensors" || artifacts.keyboardVisible !== "tensors") failures.push("开放工件键盘 tab 导航异常"); +if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 8 || mobile.artifactTabs !== 4 || mobile.artifactLayers !== 93) failures.push("移动端导航或实验异常"); if (mobile.offenders.length) failures.push(`移动端越界元素:${JSON.stringify(mobile.offenders)}`); if (exceptions.length) failures.push(`浏览器异常:${exceptions.join(" | ")}`); diff --git a/src/components/K3ArtifactLab.astro b/src/components/K3ArtifactLab.astro new file mode 100644 index 0000000..3465d2d --- /dev/null +++ b/src/components/K3ArtifactLab.astro @@ -0,0 +1,791 @@ +--- +import { + k3ArtifactEvidence, + k3ArtifactLayers, + k3ArtifactSnapshot as snapshot, + k3ArtifactViews, +} from "@/data/k3Artifacts"; + +const checkpoint = snapshot.checkpoint; +const audit = snapshot.parameter_audit; +const probe = snapshot.router_stress_probe; +const flash = snapshot.flashkda; + +const bytes = (value: number) => { + if (value >= 2 ** 40) return `${(value / 2 ** 40).toFixed(3)} TiB`; + if (value >= 2 ** 30) return `${(value / 2 ** 30).toFixed(2)} GiB`; + if (value >= 2 ** 20) return `${(value / 2 ** 20).toFixed(2)} MiB`; + return `${value.toLocaleString("en-US")} B`; +}; + +const parameterRows = [ + { + id: "alog", + label: "A_log / checkpoint", + shape: "[128] F32", + source: audit.a_log, + }, + { + id: "dt", + label: "dt_bias / checkpoint", + shape: "[96,128] F32", + source: audit.dt_bias, + }, + { + id: "router-bias", + label: "router correction bias", + shape: "[896] F32", + source: audit.router_correction_bias, + }, + { + id: "router-norm", + label: "router row L2", + shape: "[896] derived", + source: audit.router_row_l2, + }, +]; + +const tensorGroups = [ + { + id: "expert", + label: "ROUTED EXPERT 0", + boundary: "O / 一个真实 expert 的 packed tensors", + rows: snapshot.tensor_examples.routed_expert_0, + }, + { + id: "mla", + label: "LAYER 4 / GATED MLA", + boundary: "O / 真实 shard 4 header", + rows: snapshot.tensor_examples.mla_layer_4, + }, + { + id: "vision", + label: "MOONVIT-V2", + boundary: "O / 真实 shard 96 header", + rows: snapshot.tensor_examples.vision, + }, +]; + +const benchmarkDevices = [ + ["h20", "H20", flash.official_benchmarks.h20], + ["gb200", "GB200", flash.official_benchmarks.gb200], +] as const; +--- + +
+
+
+

ROUND 03 / OPEN ARTIFACT FORENSICS

+

不加载 1.56 TB,也能从 config、tensor header 与小范围权重读出真实结构

+
+

+ HF revision {snapshot.provenance.huggingface_revision.slice(0, 9)} · + FlashKDA {snapshot.provenance.flashkda_revision.slice(0, 9)}。 + 原始权重不进仓库;真实观测、推导、执行、合成探针与未决矛盾分别标记。 +

+
+ +
+ {k3ArtifactViews.map(([id, number, title, subtitle], index) => ( + + ))} +
+ +
+
+
O / CONFIG → EXECUTION STRIP

拖动一层:同时看 attention、FFN 与 AttnRes block

+

层号按人类阅读使用 1–93;公开 Python 实现内部使用 0-based index。L92 与 L93 连续两个 MLA 是真实配置。

+
+
+ +
+ KDA · 69 + MLA · 24 + AttnRes source · 8 +
+
+
+ {k3ArtifactLayers.map((layer) => ( + + ))} +
+
+
LAYER01

首层 dense;后续 92 层进入 MoE。

+
SEQUENCE MIXERKDA

96 heads × 128 dims · recurrent state

+
WIDTH MIXERDENSE

33792 intermediate · BF16

+
DEPTH SOURCEB1 · OPEN

这一层把 prefix sum 写入 block-level source。

+
+
O / exact public config

条带不代表每层 FLOPs 相同;KDA、MLA、dense 与 896→16 MoE 的状态和执行代价不同。

+
+ + + + + + + + +
+ + + + diff --git a/src/data/k3-artifact-snapshot.json b/src/data/k3-artifact-snapshot.json new file mode 100644 index 0000000..baf136b --- /dev/null +++ b/src/data/k3-artifact-snapshot.json @@ -0,0 +1,600 @@ +{ + "schema_version": 1, + "captured_at": "2026-07-29T12:21:00+08:00", + "evidence_boundary": { + "checkpoint_forward_run": false, + "raw_weights_committed": false, + "router_inputs": "deterministic synthetic RMS-normalized vectors, not token hidden states", + "kda_retention_probe": "noncanonical channel-wise interpretation used only to expose the A_log shape ambiguity" + }, + "provenance": { + "huggingface_model": "moonshotai/Kimi-K3", + "huggingface_revision": "9f62e4e9fffbd0a83ddd60e1c209d828994b3569", + "flashkda_revision": "1ce47ea3bb22c84eb9cc665028399cf35e8ffb0b", + "sha256": { + "config": "9710e121a58d03ac92c8d6da287a19541994319afbbe6d6202af001ffd379213", + "index": "a1c5210650ce71d2d3ae9ec5a101ac4afd3cf4b10091be589853437eb967febd", + "kda_slice": "dccb10e734a1b6756fe24233ef86c6417b704133a2ec4e56c5c5ddf87274d9b2", + "router_prefix": "dbe66ff82e58bbd273514c0a51b74043c519f64df4586166321cf6266e1723ee" + } + }, + "checkpoint": { + "tensor_data_bytes": 1560860324864, + "tensor_data_tb": 1.560860324864, + "tensor_data_tib": 1.4195941956713796, + "shards": 96, + "shard_file_bytes": { + "sum": 1560936091448, + "min": 92289328, + "max": 16990916912, + "mean": 16259750952.583334 + }, + "tensor_entries": 497220, + "tensor_counts": { + "expert_packed": 247296, + "expert_scales": 247296, + "router_weight": 92, + "router_correction_bias": 92, + "attnres_proj": 187, + "attnres_norm": 187, + "kda_a_log": 69, + "kda_dt_bias": 69, + "vision": 165, + "projector": 3 + }, + "derived_routed_expert_bytes": 1446456066048, + "derived_routed_expert_share": 0.9267043584915465 + }, + "configuration": { + "layers": 93, + "dense_layers": 1, + "hidden": 7168, + "vocabulary": 163840, + "context": 1048576, + "kda_layers": [ + 1, + 2, + 3, + 5, + 6, + 7, + 9, + 10, + 11, + 13, + 14, + 15, + 17, + 18, + 19, + 21, + 22, + 23, + 25, + 26, + 27, + 29, + 30, + 31, + 33, + 34, + 35, + 37, + 38, + 39, + 41, + 42, + 43, + 45, + 46, + 47, + 49, + 50, + 51, + 53, + 54, + 55, + 57, + 58, + 59, + 61, + 62, + 63, + 65, + 66, + 67, + 69, + 70, + 71, + 73, + 74, + 75, + 77, + 78, + 79, + 81, + 82, + 83, + 85, + 86, + 87, + 89, + 90, + 91 + ], + "mla_layers": [ + 4, + 8, + 12, + 16, + 20, + 24, + 28, + 32, + 36, + 40, + 44, + 48, + 52, + 56, + 60, + 64, + 68, + 72, + 76, + 80, + 84, + 88, + 92, + 93 + ], + "heads": 96, + "head_dim": 128, + "attnres_block": 12, + "experts": 896, + "active_experts": 16, + "shared_experts": 2, + "latent_width": 3584, + "expert_intermediate": 3072, + "situ_beta": 4.0, + "situ_linear_beta": 25.0, + "mla_nope": true, + "mla_output_gate": true + }, + "tensor_examples": { + "mla_layer_4": [ + { + "name": "language_model.model.layers.3.self_attn.kv_a_proj_with_mqa.weight", + "dtype": "BF16", + "shape": [ + 576, + 7168 + ], + "bytes": 8257536 + }, + { + "name": "language_model.model.layers.3.self_attn.kv_b_proj.weight", + "dtype": "BF16", + "shape": [ + 24576, + 512 + ], + "bytes": 25165824 + }, + { + "name": "language_model.model.layers.3.self_attn.q_a_proj.weight", + "dtype": "BF16", + "shape": [ + 1536, + 7168 + ], + "bytes": 22020096 + }, + { + "name": "language_model.model.layers.3.self_attn.q_b_proj.weight", + "dtype": "BF16", + "shape": [ + 18432, + 1536 + ], + "bytes": 56623104 + }, + { + "name": "language_model.model.layers.3.self_attn.g_proj.weight", + "dtype": "BF16", + "shape": [ + 12288, + 7168 + ], + "bytes": 176160768 + } + ], + "vision": [ + { + "name": "vision_tower.patch_embed.proj.weight", + "dtype": "BF16", + "shape": [ + 1024, + 3, + 14, + 14 + ], + "bytes": 1204224 + }, + { + "name": "vision_tower.patch_embed.pos_emb.weight", + "dtype": "BF16", + "shape": [ + 64, + 64, + 1024 + ], + "bytes": 8388608 + }, + { + "name": "vision_tower.encoder.blocks.0.wqkv.weight", + "dtype": "BF16", + "shape": [ + 4608, + 1024 + ], + "bytes": 9437184 + }, + { + "name": "vision_tower.encoder.blocks.26.wqkv.weight", + "dtype": "BF16", + "shape": [ + 4608, + 1024 + ], + "bytes": 9437184 + }, + { + "name": "vision_tower.encoder.final_layernorm.weight", + "dtype": "BF16", + "shape": [ + 1024 + ], + "bytes": 2048 + } + ], + "routed_expert_0": [ + { + "name": "w1.weight_packed", + "dtype": "U8", + "shape": [ + 3072, + 1792 + ], + "bytes": 5505024 + }, + { + "name": "w1.weight_scale", + "dtype": "U8", + "shape": [ + 3072, + 112 + ], + "bytes": 344064 + }, + { + "name": "w2.weight_packed", + "dtype": "U8", + "shape": [ + 3584, + 1536 + ], + "bytes": 5505024 + }, + { + "name": "w2.weight_scale", + "dtype": "U8", + "shape": [ + 3584, + 96 + ], + "bytes": 344064 + }, + { + "name": "w3.weight_packed", + "dtype": "U8", + "shape": [ + 3072, + 1792 + ], + "bytes": 5505024 + }, + { + "name": "w3.weight_scale", + "dtype": "U8", + "shape": [ + 3072, + 112 + ], + "bytes": 344064 + } + ] + }, + "parameter_audit": { + "a_log_checkpoint_shape": [ + 128 + ], + "a_log_public_code_shape": [ + 96 + ], + "dt_bias_shape": [ + 96, + 128 + ], + "beta_projection_shape": [ + 96, + 7168 + ], + "status": "observed shape inconsistency; runtime meaning unresolved", + "a_log": { + "count": 128, + "mean": -0.16894778609275818, + "std": 0.3667531907558441, + "quantiles": { + "min": -0.7530547380447388, + "p01": -0.7163181900978088, + "p10": -0.5367198586463928, + "p25": -0.4014696776866913, + "p50": -0.1533127874135971, + "p75": 0.0, + "p90": 0.07427694648504257, + "p99": 0.7512762546539307, + "max": 2.4661004543304443 + } + }, + "a_rate_exp": { + "count": 128, + "mean": 0.9474229216575623, + "std": 1.0007015466690063, + "quantiles": { + "min": 0.47092580795288086, + "p01": 0.48882821202278137, + "p10": 0.5846699476242065, + "p25": 0.6693384051322937, + "p50": 0.8578758239746094, + "p75": 1.0, + "p90": 1.0771057605743408, + "p99": 2.1350440979003906, + "max": 11.776434898376465 + } + }, + "dt_bias": { + "count": 12288, + "mean": -4.57477331161499, + "std": 1.423690915107727, + "quantiles": { + "min": -7.8938117027282715, + "p01": -6.962827682495117, + "p10": -6.4912495613098145, + "p25": -5.797565460205078, + "p50": -4.622031211853027, + "p75": -3.4063913822174072, + "p90": -2.568972587585449, + "p99": -1.895374059677124, + "max": 0.179231196641922 + } + }, + "channelwise_hypothesis": { + "log_decay": { + "count": 12288, + "mean": -0.22606194019317627, + "std": 0.26349756121635437, + "quantiles": { + "min": -2.685786724090576, + "p01": -1.122031569480896, + "p10": -0.6017768383026123, + "p25": -0.3287131190299988, + "p50": -0.12320166081190109, + "p75": -0.035499051213264465, + "p90": -0.008835914544761181, + "p99": -8.967738722276408e-06, + "max": 0.0 + } + }, + "one_step_retention": { + "count": 12288, + "mean": 0.821946918964386, + "std": 0.17656706273555756, + "quantiles": { + "min": 0.068167544901371, + "p01": 0.32561764121055603, + "p10": 0.5478373765945435, + "p25": 0.7198495268821716, + "p50": 0.8840853571891785, + "p75": 0.9651236534118652, + "p90": 0.9912030100822449, + "p99": 0.9999910593032837, + "max": 1.0 + } + }, + "retention_after_64_steps": { + "count": 12288, + "mean": 0.13120755553245544, + "std": 0.253229558467865, + "quantiles": { + "min": 0.0, + "p01": 6.506081464145866e-32, + "p10": 1.8781597374798298e-17, + "p25": 7.302480842241721e-10, + "p50": 0.00037638185312971473, + "p75": 0.10311195999383926, + "p90": 0.5680769681930542, + "p99": 0.9994263052940369, + "max": 1.0 + } + } + }, + "router_correction_bias": { + "count": 896, + "mean": 6.285762310653809e-07, + "std": 0.02034943550825119, + "quantiles": { + "min": -0.08442370593547821, + "p01": -0.06806425005197525, + "p10": -0.0273849219083786, + "p25": -0.009373871609568596, + "p50": 0.004928342066705227, + "p75": 0.014873139560222626, + "p90": 0.020546497777104378, + "p99": 0.02667994052171707, + "max": 0.028605474159121513 + } + }, + "router_row_l2": { + "count": 896, + "mean": 4.949629783630371, + "std": 0.895085871219635, + "quantiles": { + "min": 2.6018569469451904, + "p01": 2.9133214950561523, + "p10": 3.7345614433288574, + "p25": 4.310352325439453, + "p50": 5.036390781402588, + "p75": 5.6203179359436035, + "p90": 6.068215370178223, + "p99": 6.65656042098999, + "max": 7.023199081420898 + } + }, + "router_bias_norm_correlation": 0.456669420003891 + }, + "router_stress_probe": { + "seed": 20260729, + "synthetic_tokens": 2048, + "hidden_rms": 0.999999463558197, + "without_correction_bias": { + "mean": 36.57143020629883, + "std": 76.2339096069336, + "cv": 2.0845208168029785, + "gini": 0.8310886025428772, + "zero_experts": 558, + "quantiles": { + "min": 0.0, + "p10": 0.0, + "p25": 0.0, + "p50": 0.0, + "p75": 16.0, + "p90": 166.0, + "p99": 310.0999755859375, + "max": 369.0 + } + }, + "with_correction_bias": { + "mean": 36.57143020629883, + "std": 92.48487854003906, + "cv": 2.528883218765259, + "gini": 0.878814697265625, + "zero_experts": 673, + "quantiles": { + "min": 0.0, + "p10": 0.0, + "p25": 0.0, + "p50": 0.0, + "p75": 0.0, + "p90": 167.0, + "p99": 398.0999755859375, + "max": 451.0 + } + }, + "membership_overlap_mean": 2.1708984375, + "tokens_changed": 2048, + "changed_fraction": 1.0, + "mean_replacements_per_token": 13.8291015625 + }, + "flashkda": { + "supported_architectures": [ + "90a", + "100a", + "103a", + "120a" + ], + "requirements": { + "cuda": ">=12.9", + "pytorch": ">=2.4", + "gpu": "SM90+" + }, + "official_benchmarks": { + "h20": { + "sequence": 8192, + "heads": 96, + "dimension": 128, + "rows": { + "Fixed": { + "flash_kda_ms": 2.622, + "fla_chunk_kda_ms": 4.8388, + "speedup_vs_chunk_kda": 1.85, + "fla_chunk_gdn_ms": 3.1985, + "speedup_vs_gdn": 1.22 + }, + "Varlen, `seq_lens`=[1300, 547, 2048, 963, 271, 3063]": { + "flash_kda_ms": 2.3449, + "fla_chunk_kda_ms": 4.8291, + "speedup_vs_chunk_kda": 2.06, + "fla_chunk_gdn_ms": 3.0541, + "speedup_vs_gdn": 1.3 + }, + "Varlen, `seq_lens`=`1024 x 8`": { + "flash_kda_ms": 2.0432, + "fla_chunk_kda_ms": 4.6723, + "speedup_vs_chunk_kda": 2.29, + "fla_chunk_gdn_ms": 2.9117, + "speedup_vs_gdn": 1.43 + } + } + }, + "gb200": { + "sequence": 8192, + "heads": 96, + "dimension": 128, + "rows": { + "Fixed": { + "flash_kda_ms": 1.0087, + "fla_chunk_kda_ms": 2.3271, + "speedup_vs_chunk_kda": 2.31, + "fla_chunk_gdn_ms": 1.2792, + "speedup_vs_gdn": 1.27 + }, + "Varlen, `seq_lens`=[1300, 547, 2048, 963, 271, 3063]": { + "flash_kda_ms": 0.8597, + "fla_chunk_kda_ms": 2.334, + "speedup_vs_chunk_kda": 2.71, + "fla_chunk_gdn_ms": 1.2962, + "speedup_vs_gdn": 1.51 + }, + "Varlen, `seq_lens`=`1024 x 8`": { + "flash_kda_ms": 0.7064, + "fla_chunk_kda_ms": 2.3105, + "speedup_vs_chunk_kda": 3.27, + "fla_chunk_gdn_ms": 1.2744, + "speedup_vs_gdn": 1.8 + } + } + } + }, + "local_environment": { + "python": "3.10.14", + "torch": "2.11.0+cu128", + "torch_cuda": "12.8", + "gpu": "NVIDIA GeForce RTX 5090", + "capability": [ + 12, + 0 + ], + "libc": [ + "glibc", + "2.43" + ] + }, + "local_build": { + "status": "blocked_before_kernel execution", + "attempt_1": "system g++ 15 exceeds CUDA 12.8 host compiler range", + "attempt_2": "temporary g++ 13 reaches nvcc, then CUDA 12.8 headers conflict with current glibc math declarations", + "interpretation": "GPU architecture is listed by the repository, but the local CUDA 12.8 stack is below the official CUDA 12.9 requirement" + } + } +} diff --git a/src/data/k3Artifacts.ts b/src/data/k3Artifacts.ts new file mode 100644 index 0000000..241d1e3 --- /dev/null +++ b/src/data/k3Artifacts.ts @@ -0,0 +1,36 @@ +import snapshot from "./k3-artifact-snapshot.json"; + +export { snapshot as k3ArtifactSnapshot }; + +const mlaLayerSet = new Set(snapshot.configuration.mla_layers); + +export const k3ArtifactLayers = Array.from( + { length: snapshot.configuration.layers }, + (_, index) => { + const layer = index + 1; + const attention = mlaLayerSet.has(layer) ? "MLA" : "KDA"; + return { + layer, + attention, + feedForward: layer === 1 ? "DENSE" : "MOE", + block: Math.floor(index / snapshot.configuration.attnres_block) + 1, + opensResidualBlock: index % snapshot.configuration.attnres_block === 0, + terminalGlobal: layer === snapshot.configuration.layers, + }; + }, +); + +export const k3ArtifactViews = [ + ["layers", "01", "LAYER MAP", "93 层配置"], + ["tensors", "02", "TENSOR ANATOMY", "497,220 entries"], + ["parameters", "03", "PARAMETER AUDIT", "真实权重小切片"], + ["reproduction", "04", "REPRODUCTION", "作者值与本机边界"], +] as const; + +export const k3ArtifactEvidence = [ + ["O", "OFFICIAL ARTIFACT", "官方 config、code、index、header 或参数字节直接观测。"], + ["D", "DERIVATION", "由公开 shape 与计数做确定性算术。"], + ["X", "EXECUTED", "本站机器实际执行并留下环境与结果。"], + ["S", "SYNTHETIC", "真实权重加合成输入;只测试边界,不代表真实 token。"], + ["U", "UNRESOLVED", "工件之间存在不一致,当前不擅自补解释。"], +] as const; diff --git a/src/pages/index.astro b/src/pages/index.astro index 2440bd6..12af072 100644 --- a/src/pages/index.astro +++ b/src/pages/index.astro @@ -128,19 +128,19 @@ const paths = [
-

NEW / K3 ROUND 02 REPORT · FORMULA · SYSTEM · EVIDENCE

-

47 页不再压成摘要:把 K3 的每个因果环节重新展开

+

NEW / K3 ROUND 03 REPORT · CHECKPOINT · KERNEL · BOUNDARY

+

47 页不再压成摘要:再把 1.56 TB 开放工件接回报告

- 用三十二张问题账逐节读完 KDA、Gated MLA、AttnRes、Stable LatentMoE、原生视觉、 - 预训练、九专家 MOPD、Agent 环境、FlashKDA / MoonEP、混合 prefix cache、评测与案例边界。 + 在三十二张报告问题账之外,继续审计 96 个 safetensors 分片、497,220 个 tensor entries、 + 真实 KDA / MLA / MoE / MoonViT shape、小范围权重统计、FlashKDA 编译边界与未决形状矛盾。

REPORT
16 Figures · 5 Tables
-
NODES
100 个一手 / 官方节点
-
LAB
Delta · Decay · AttnRes · MoE · QB · RL · Cache
+
ARTIFACTS
96 shards · 497,220 entries
+
LAB
8 报告实验 + 4 工件视图
- +
diff --git a/src/pages/k3/index.astro b/src/pages/k3/index.astro index 1b64c6b..c405837 100644 --- a/src/pages/k3/index.astro +++ b/src/pages/k3/index.astro @@ -1,6 +1,7 @@ --- import BaseLayout from "@/layouts/BaseLayout.astro"; import ArchitectureExplorer from "@/components/ArchitectureExplorer.astro"; +import K3ArtifactLab from "@/components/K3ArtifactLab.astro"; import K3ReportLab from "@/components/K3ReportLab.astro"; import { k3FigureAtlas, k3Ledgers, k3PaperChain, k3ReportMap } from "@/data/k3"; @@ -34,7 +35,8 @@ const toc = [ ["26", "cases", "案例边界"], ["27", "xtml", "XTML 协议"], ["28", "lab", "八联交互实验"], - ["29", "audit", "21 张图表审计"], + ["29", "artifacts", "开放权重工件审计"], + ["30", "audit", "21 张图表审计"], ["↳", "papers", "100 节点阅读链"], ]; @@ -103,13 +105,13 @@ const paperGroups = [
-

ANCHOR REPORT / ROUND 02 KIMI K3 · 47 PAGES

+

ANCHOR REPORT / ROUND 03 KIMI K3 · REPORT → OPEN ARTIFACTS

不把报告压成摘要
把每个因果环节
重新展开

K3 同时扩展序列、深度、宽度、视觉与 Agent 轨迹。真正值得读的不是 2.8T 这个最大数字, @@ -119,10 +121,11 @@ const paperGroups = [

QUESTIONS
32 张问题账
REPORT
16 Figures · 5 Tables
-
LABS
8 个可操作实验
+
LABS
8 个机制实验 + 4 个工件视图
READING
100 个一手 / 官方节点
MODEL
2.78T total / 104.2B active
-
STATUS
K3 二轮深读
+
ARTIFACTS
96 shards · 497,220 tensors
+
STATUS
K3 三轮进行中
@@ -137,7 +140,8 @@ const paperGroups = [
证据约定 - R = K3 报告;P = 原论文 / 官方实现;D = 确定性推导;T = 教学模型。四者不互相冒充。 + 报告层用 R / P / D / T 区分报告、原始来源、推导与教学模型; + 工件层用 O / D / X / S / U 区分观测、推导、本站执行、合成探针与未决矛盾。
@@ -844,8 +848,29 @@ const paperGroups = [ +
+

29 OPEN ARTIFACT FORENSICS

+

从“报告说了什么”走到“公开 checkpoint 实际长什么样”

+

+ 第三轮固定到官方 Hugging Face revision,读取 config、remote code、60 MB tensor index、 + 四个 safetensors headers 和两个小范围参数切片。原始权重不进入本站仓库; + 结构、shape、计数、参数统计与本机编译边界都可以从公开脚本重复生成。 +

+
+
O / OBSERVED1.4196 TiB tensor data

96 shards、497,220 entries;不是运行显存,也不是参数量口径。

+
D / CLOSED LOOP69 KDA · 24 MLA · 92 MoE

配置、tensor names 与 header shape 三方闭合。

+
U / UNRESOLVEDA_log [128] ≠ expected [96]

checkpoint 与公开代码 / kernel API 的形状冲突保留在主视区,不擅自解释。

+
+ +
+
+
-

29 FIGURE & TABLE AUDIT

+

30 FIGURE & TABLE AUDIT

Figure 1–16、Table 1–5:每张图究竟支持什么,不能支持什么

{k3FigureAtlas.map(([id, report, title, contract]) => ( @@ -900,6 +925,28 @@ const paperGroups = [