wuyang/llm-atlas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: audit K3 open model artifacts

Browse Source
This commit is contained in:
wuyang
2026-07-29 12:44:37 +08:00
parent c3d332680c
commit be2b291a04
13 changed files with 2387 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/components/K3ArtifactLab.astro
+791
View File
@@ -0,0 +1,791 @@
---
import {
k3ArtifactEvidence,
k3ArtifactLayers,
k3ArtifactSnapshot as snapshot,
k3ArtifactViews,
} from "@/data/k3Artifacts";
const checkpoint = snapshot.checkpoint;
const audit = snapshot.parameter_audit;
const probe = snapshot.router_stress_probe;
const flash = snapshot.flashkda;
const bytes = (value: number) => {
if (value >= 2 ** 40) return `${(value / 2 ** 40).toFixed(3)} TiB`;
if (value >= 2 ** 30) return `${(value / 2 ** 30).toFixed(2)} GiB`;
if (value >= 2 ** 20) return `${(value / 2 ** 20).toFixed(2)} MiB`;
return `${value.toLocaleString("en-US")} B`;
};
const parameterRows = [
{
id: "alog",
label: "A_log / checkpoint",
shape: "[128] F32",
source: audit.a_log,
},
{
id: "dt",
label: "dt_bias / checkpoint",
shape: "[96,128] F32",
source: audit.dt_bias,
},
{
id: "router-bias",
label: "router correction bias",
shape: "[896] F32",
source: audit.router_correction_bias,
},
{
id: "router-norm",
label: "router row L2",
shape: "[896] derived",
source: audit.router_row_l2,
},
];
const tensorGroups = [
{
id: "expert",
label: "ROUTED EXPERT 0",
boundary: "O / 一个真实 expert 的 packed tensors",
rows: snapshot.tensor_examples.routed_expert_0,
},
{
id: "mla",
label: "LAYER 4 / GATED MLA",
boundary: "O / 真实 shard 4 header",
rows: snapshot.tensor_examples.mla_layer_4,
},
{
id: "vision",
label: "MOONVIT-V2",
boundary: "O / 真实 shard 96 header",
rows: snapshot.tensor_examples.vision,
},
];
const benchmarkDevices = [
["h20", "H20", flash.official_benchmarks.h20],
["gb200", "GB200", flash.official_benchmarks.gb200],
] as const;
---
<figure class="artifact-lab" data-k3-artifact-lab>
<header class="artifact-head">
<div>
<p>ROUND 03 / OPEN ARTIFACT FORENSICS</p>
<h3>不加载 1.56 TB，也能从 config、tensor header 与小范围权重读出真实结构</h3>
</div>
<p>
HF revision <code>{snapshot.provenance.huggingface_revision.slice(0, 9)}</code> ·
FlashKDA <code>{snapshot.provenance.flashkda_revision.slice(0, 9)}</code>。
原始权重不进仓库；真实观测、推导、执行、合成探针与未决矛盾分别标记。
</p>
</header>
<div class="artifact-tabs" role="tablist" aria-label="选择 K3 开放工件实验">
{k3ArtifactViews.map(([id, number, title, subtitle], index) => (
<button
type="button"
role="tab"
data-artifact-tab={id}
aria-selected={index === 0 ? "true" : "false"}
tabindex={index === 0 ? "0" : "-1"}
>
<span>{number}</span><b>{title}</b><small>{subtitle}</small>
</button>
))}
</div>
<section class="artifact-panel" data-artifact-panel="layers">
<div class="panel-intro">
<div><span>O / CONFIG → EXECUTION STRIP</span><h4>拖动一层：同时看 attention、FFN 与 AttnRes block</h4></div>
<p>层号按人类阅读使用 1–93；公开 Python 实现内部使用 0-based index。L92 与 L93 连续两个 MLA 是真实配置。</p>
</div>
<div class="layer-control">
<label>
<span>SELECT LAYER <output data-layer-label>1 / 93</output></span>
<input data-layer-slider type="range" min="1" max="93" value="1" />
</label>
<div class="layer-legend">
<i class="kda"></i><span>KDA · 69</span>
<i class="mla"></i><span>MLA · 24</span>
<i class="open"></i><span>AttnRes source · 8</span>
</div>
</div>
<div class="layer-strip" aria-label="K3 真实 93 层配置条带">
{k3ArtifactLayers.map((layer) => (
<button
type="button"
data-layer-cell
data-layer={layer.layer}
data-attention={layer.attention}
data-ffn={layer.feedForward}
data-block={layer.block}
data-open={String(layer.opensResidualBlock)}
data-terminal={String(layer.terminalGlobal)}
class:list={[
layer.attention.toLowerCase(),
{ open: layer.opensResidualBlock, terminal: layer.terminalGlobal },
]}
aria-label={`第 ${layer.layer} 层，${layer.attention}，${layer.feedForward}，AttnRes block ${layer.block}`}
title={`L${layer.layer} · ${layer.attention} · ${layer.feedForward} · B${layer.block}`}
>
<span>{layer.layer}</span>
</button>
))}
</div>
<div class="layer-readout">
<article><span>LAYER</span><b data-layer-number>01</b><p data-layer-special>首层 dense；后续 92 层进入 MoE。</p></article>
<article><span>SEQUENCE MIXER</span><b data-layer-attention>KDA</b><p data-layer-attention-copy>96 heads × 128 dims · recurrent state</p></article>
<article><span>WIDTH MIXER</span><b data-layer-ffn>DENSE</b><p data-layer-ffn-copy>33792 intermediate · BF16</p></article>
<article class="dark"><span>DEPTH SOURCE</span><b data-layer-block>B1 · OPEN</b><p data-layer-block-copy>这一层把 prefix sum 写入 block-level source。</p></article>
</div>
<div class="boundary"><b>O / exact public config</b><p>条带不代表每层 FLOPs 相同；KDA、MLA、dense 与 896→16 MoE 的状态和执行代价不同。</p></div>
</section>
<section class="artifact-panel" data-artifact-panel="tensors" hidden>
<div class="panel-intro">
<div><span>O + D / SAFETENSORS INDEX</span><h4>497,220 个 entries 里，绝大多数为什么来自 experts？</h4></div>
<p>entry count、dtype、shape 与 byte offsets 来自公开 index/header；占比是确定性算术，不是运行显存。</p>
</div>
<div class="artifact-metrics">
<article><span>SHARDS</span><b>{checkpoint.shards}</b><p>96 个公开 safetensors</p></article>
<article><span>TENSOR DATA</span><b>{checkpoint.tensor_data_tib.toFixed(3)} TiB</b><p>{checkpoint.tensor_data_tb.toFixed(3)} TB decimal</p></article>
<article><span>ENTRIES</span><b>{checkpoint.tensor_entries.toLocaleString("en-US")}</b><p>不是 parameter count</p></article>
<article class="dark"><span>ROUTED WEIGHT SHARE</span><b>{(checkpoint.derived_routed_expert_share * 100).toFixed(2)}%</b><p>D / packed weights + scales</p></article>
</div>
<div class="tensor-ledger">
<div>
<span>247,296</span><b>packed expert tensors</b><p>92 layers × 896 experts × w1/w2/w3</p>
</div>
<div>
<span>247,296</span><b>scale tensors</b><p>每个 packed matrix 独立保存 group scales</p>
</div>
<div>
<span>187 + 187</span><b>AttnRes proj / norm</b><p>每层 attention + MLP 两次读取，再加 output</p>
</div>
<div>
<span>165 + 3</span><b>vision / projector</b><p>MoonViT-V2 与 shared embedding bridge</p>
</div>
</div>
<div class="tensor-selector">
<span>INSPECT HEADER GROUP</span>
{tensorGroups.map((group, index) => (
<button
type="button"
data-tensor-tab={group.id}
aria-pressed={index === 0 ? "true" : "false"}
>
{group.label}
</button>
))}
</div>
{tensorGroups.map((group, index) => (
<div class="tensor-table" data-tensor-panel={group.id} hidden={index !== 0}>
<header><span>{group.label}</span><b>{group.boundary}</b></header>
{group.rows.map((row) => (
<div>
<code>{row.name.replace("language_model.model.layers.3.self_attn.", "").replace("vision_tower.", "")}</code>
<span>{row.dtype}</span>
<b>{row.shape.join(" × ")}</b>
<small>{bytes(row.bytes)}</small>
</div>
))}
</div>
))}
<div class="boundary"><b>O header / D aggregate</b><p>MXFP4 packed shape 不是原始逻辑矩阵 shape；必须结合 latent width、group size 与 loader 格式解释。</p></div>
</section>
<section class="artifact-panel" data-artifact-panel="parameters" hidden>
<div class="panel-intro">
<div><span>O + U / SELECTED WEIGHT RANGES</span><h4>几十 KB 真实参数，首先暴露的是一个不能擅自修掉的矛盾</h4></div>
<p>只读 shard 1 的 49,664 bytes 与 shard 2 的 13.49 MB prefix；统计可复现，原始参数不再分发。</p>
</div>
<div class="shape-conflict">
<div><span>CONFIG</span><b>96 heads × 128 dims</b><p>q/k/v projection = 12,288</p></div>
<i>≠</i>
<div class="warn"><span>CHECKPOINT</span><b>A_log [128]</b><p>真实 shard header</p></div>
<i>≠</i>
<div><span>CODE + KERNEL API</span><b>A_log [H] = [96]</b><p>remote code 与 C++ TORCH_CHECK</p></div>
</div>
<div class="parameter-control">
<label>
<span>PARAMETER VIEW</span>
<select data-parameter-select>
{parameterRows.map((row) => <option value={row.id}>{row.label}</option>)}
</select>
</label>
<p><b data-parameter-shape>[128] F32</b><span data-parameter-count>128 values</span></p>
</div>
<div class="distribution">
{parameterRows.map((row, index) => (
<div
data-parameter-row={row.id}
hidden={index !== 0}
data-min={row.source.quantiles.min}
data-p10={row.source.quantiles.p10}
data-p50={row.source.quantiles.p50}
data-p90={row.source.quantiles.p90}
data-max={row.source.quantiles.max}
data-mean={row.source.mean}
data-std={row.source.std}
data-count={row.source.count}
data-shape={row.shape}
>
<div class="axis">
<i style="left:10%"></i><i style="left:50%"></i><i style="left:90%"></i>
</div>
<div class="distribution-stats">
<article><span>MIN</span><b>{row.source.quantiles.min.toFixed(4)}</b></article>
<article><span>P10</span><b>{row.source.quantiles.p10.toFixed(4)}</b></article>
<article><span>MEDIAN</span><b>{row.source.quantiles.p50.toFixed(4)}</b></article>
<article><span>P90</span><b>{row.source.quantiles.p90.toFixed(4)}</b></article>
<article><span>MAX</span><b>{row.source.quantiles.max.toFixed(4)}</b></article>
</div>
<p>mean {row.source.mean.toFixed(5)} · std {row.source.std.toFixed(5)}</p>
</div>
))}
</div>
<div class="hypothesis-card">
<span>U / NONCANONICAL HYPOTHESIS ONLY</span>
<h5>若把 checkpoint `[128]` 临时当作 channel-wise 参数</h5>
<div>
<p><b>{audit.channelwise_hypothesis.one_step_retention.quantiles.p50.toFixed(4)}</b><span>one-step median retention</span></p>
<p><b>{audit.channelwise_hypothesis.retention_after_64_steps.quantiles.p50.toExponential(2)}</b><span>64-step median retention</span></p>
<p><b>不能定案</b><span>公开 loader / 官方解释仍缺失</span></p>
</div>
</div>
<div class="boundary danger"><b>U / unresolved inconsistency</b><p>本站只报告形状冲突；不宣布 checkpoint 损坏，也不把 channel-wise 猜测冒充真实 K3 forward。</p></div>
</section>
<section class="artifact-panel" data-artifact-panel="reproduction" hidden>
<div class="panel-intro">
<div><span>X + S / WHAT ACTUALLY RAN</span><h4>作者 benchmark、本站编译尝试与合成反例，三者不能写成同一种实测</h4></div>
<p>RTX 5090 是 sm_120，但本机 PyTorch CUDA 12.8 低于 FlashKDA README 的 12.9+；kernel 尚未执行。</p>
</div>
<div class="repro-controls">
<label><span>OFFICIAL DEVICE</span><select data-benchmark-device>
{benchmarkDevices.map(([id, label]) => <option value={id}>{label}</option>)}
</select></label>
<label><span>OFFICIAL CASE</span><select data-benchmark-case>
<option value="Fixed">Fixed T=8192</option>
<option value="Varlen, `seq_lens`=`1024 x 8`">8 × 1024 varlen</option>
</select></label>
<label><span>ROUTER STRESS</span><select data-router-mode>
<option value="raw">without correction bias</option>
<option value="bias">with checkpoint bias</option>
</select></label>
</div>
<div class="benchmark-readout">
<article><span>FLASHKDA</span><b data-benchmark-flash>2.6220 ms</b><p>O / author repository</p></article>
<article><span>FLA CHUNK KDA</span><b data-benchmark-fla>4.8388 ms</b><p>O / same author table</p></article>
<article class="accent"><span>AUTHOR SPEEDUP</span><b data-benchmark-speedup>1.85×</b><p>不能外推到 RTX 5090</p></article>
<article class="dark"><span>LOCAL KERNEL</span><b>NOT RUN</b><p>CUDA 12.8 &lt; official 12.9+</p></article>
</div>
<div class="local-build">
<article><span>ATTEMPT 01</span><b>g++ 15 rejected</b><p>CUDA 12.8 host compiler range要求 &lt;14。</p></article>
<i>→</i>
<article><span>ATTEMPT 02</span><b>g++ 13 reached nvcc</b><p>随后在 glibc math declarations 处与 CUDA 12.8 headers 冲突。</p></article>
<i>→</i>
<article class="warn"><span>NEXT GATE</span><b>CUDA 12.9+ matched env</b><p>再跑 exact correctness 与本机 benchmark。</p></article>
</div>
<div class="router-counterexample">
<div>
<span>S / REAL WEIGHTS, SYNTHETIC HIDDEN</span>
<h5>随机 RMS=1 输入为什么不能评价 Quantile Balancing</h5>
<p>2,048 个固定 seed 向量通过真实 `896×7168` router；它们不是模型 token hidden states。</p>
</div>
<div class="router-stats">
<p><span>LOAD CV</span><b data-router-cv>{probe.without_correction_bias.cv.toFixed(3)}</b></p>
<p><span>GINI</span><b data-router-gini>{probe.without_correction_bias.gini.toFixed(3)}</b></p>
<p><span>ZERO EXPERTS</span><b data-router-zero>{probe.without_correction_bias.zero_experts}</b></p>
<p><span>TOP-16 OVERLAP</span><b>{probe.membership_overlap_mean.toFixed(2)} / 16</b></p>
</div>
</div>
<div class="boundary"><b>X/S boundary</b><p>编译失败是本站真实执行结果；router counterexample 只证明 hidden distribution 不可省略，不证明真实 QB 变好或变坏。</p></div>
</section>
<footer class="evidence-strip">
{k3ArtifactEvidence.map(([code, title, copy]) => (
<div><span>{code}</span><b>{title}</b><p>{copy}</p></div>
))}
</footer>
</figure>
<script>
const roots = document.querySelectorAll<HTMLElement>("[data-k3-artifact-lab]");
roots.forEach((root) => {
const $ = <T extends HTMLElement = HTMLElement>(selector: string) => root.querySelector<T>(selector)!;
const $$ = <T extends HTMLElement = HTMLElement>(selector: string) => [...root.querySelectorAll<T>(selector)];
const put = (selector: string, value: string | number) => {
const node = $(selector);
if (node) node.textContent = String(value);
};
const tabs = $$<HTMLButtonElement>("[data-artifact-tab]");
const panels = $$<HTMLElement>("[data-artifact-panel]");
const selectTab = (id: string) => {
tabs.forEach((tab) => {
const active = tab.dataset.artifactTab === id;
tab.setAttribute("aria-selected", String(active));
tab.tabIndex = active ? 0 : -1;
});
panels.forEach((panel) => panel.hidden = panel.dataset.artifactPanel !== id);
};
tabs.forEach((tab, index) => {
tab.addEventListener("click", () => selectTab(tab.dataset.artifactTab ?? "layers"));
tab.addEventListener("keydown", (event: KeyboardEvent) => {
if (!["ArrowLeft", "ArrowRight", "Home", "End"].includes(event.key)) return;
event.preventDefault();
let next = index;
if (event.key === "ArrowLeft") next = (index - 1 + tabs.length) % tabs.length;
if (event.key === "ArrowRight") next = (index + 1) % tabs.length;
if (event.key === "Home") next = 0;
if (event.key === "End") next = tabs.length - 1;
selectTab(tabs[next].dataset.artifactTab ?? "layers");
tabs[next].focus();
});
});
const layerSlider = $<HTMLInputElement>("[data-layer-slider]");
const cells = $$<HTMLButtonElement>("[data-layer-cell]");
const renderLayer = () => {
const selected = Number(layerSlider.value);
const cell = cells.find((item) => Number(item.dataset.layer) === selected)!;
cells.forEach((item) => item.classList.toggle("active", item === cell));
const attention = cell.dataset.attention ?? "KDA";
const ffn = cell.dataset.ffn ?? "MOE";
const block = cell.dataset.block ?? "1";
const opens = cell.dataset.open === "true";
const terminal = cell.dataset.terminal === "true";
put("[data-layer-label]", `${selected} / 93`);
put("[data-layer-number]", String(selected).padStart(2, "0"));
put("[data-layer-attention]", attention);
put("[data-layer-ffn]", ffn);
put("[data-layer-block]", `B${block} · ${opens ? "OPEN" : "READ"}`);
put("[data-layer-special]", terminal
? "额外末层 Gated MLA；因此 L92 / L93 连续全局 attention。"
: selected === 1
? "首层 dense；后续 92 层进入 MoE。"
: `第 ${Math.ceil(selected / 4)} 个 hybrid 节奏位置。`);
put("[data-layer-attention-copy]", attention === "KDA"
? "96 heads × 128 dims · recurrent state"
: "512 latent + 64 auxiliary · No rotary transform");
put("[data-layer-ffn-copy]", ffn === "DENSE"
? "33792 intermediate · BF16"
: "7168 → 3584 latent → 896 choose 16");
put("[data-layer-block-copy]", opens
? "这一层把 prefix sum 写入新的 block-level source。"
: `读取 embedding 与此前 ${opens ? block : Math.min(Number(block), 8)} 个 block source。`);
};
layerSlider.addEventListener("input", renderLayer);
cells.forEach((cell) => cell.addEventListener("click", () => {
layerSlider.value = cell.dataset.layer ?? "1";
renderLayer();
}));
renderLayer();
const tensorTabs = $$<HTMLButtonElement>("[data-tensor-tab]");
const tensorPanels = $$<HTMLElement>("[data-tensor-panel]");
tensorTabs.forEach((button) => button.addEventListener("click", () => {
const id = button.dataset.tensorTab;
tensorTabs.forEach((item) => item.setAttribute("aria-pressed", String(item === button)));
tensorPanels.forEach((panel) => panel.hidden = panel.dataset.tensorPanel !== id);
}));
const parameterSelect = $<HTMLSelectElement>("[data-parameter-select]");
const parameterRows = $$<HTMLElement>("[data-parameter-row]");
const renderParameter = () => {
const row = parameterRows.find((item) => item.dataset.parameterRow === parameterSelect.value)!;
parameterRows.forEach((item) => item.hidden = item !== row);
put("[data-parameter-shape]", row.dataset.shape ?? "");
put("[data-parameter-count]", `${Number(row.dataset.count).toLocaleString("en-US")} values`);
};
parameterSelect.addEventListener("input", renderParameter);
renderParameter();
const benchmarks = {
h20: {
Fixed: { flash: 2.6220, fla: 4.8388, speedup: 1.85 },
"Varlen, `seq_lens`=`1024 x 8`": { flash: 2.0432, fla: 4.6723, speedup: 2.29 },
},
gb200: {
Fixed: { flash: 1.0087, fla: 2.3271, speedup: 2.31 },
"Varlen, `seq_lens`=`1024 x 8`": { flash: .7064, fla: 2.3105, speedup: 3.27 },
},
};
const device = $<HTMLSelectElement>("[data-benchmark-device]");
const benchmarkCase = $<HTMLSelectElement>("[data-benchmark-case]");
const renderBenchmark = () => {
const row = benchmarks[device.value as keyof typeof benchmarks][benchmarkCase.value as "Fixed" | "Varlen, `seq_lens`=`1024 x 8`"];
put("[data-benchmark-flash]", `${row.flash.toFixed(4)} ms`);
put("[data-benchmark-fla]", `${row.fla.toFixed(4)} ms`);
put("[data-benchmark-speedup]", `${row.speedup.toFixed(2)}×`);
};
device.addEventListener("input", renderBenchmark);
benchmarkCase.addEventListener("input", renderBenchmark);
const routerMode = $<HTMLSelectElement>("[data-router-mode]");
const routerRows = {
raw: { cv: 2.0845208168, gini: .8310886025, zero: 558 },
bias: { cv: 2.5288832188, gini: .8788146973, zero: 673 },
};
const renderRouter = () => {
const row = routerRows[routerMode.value as keyof typeof routerRows];
put("[data-router-cv]", row.cv.toFixed(3));
put("[data-router-gini]", row.gini.toFixed(3));
put("[data-router-zero]", row.zero);
};
routerMode.addEventListener("input", renderRouter);
});
</script>
<style>
.artifact-lab {
max-width: 1080px;
margin: 42px 0;
border: 1px solid var(--ink);
background: var(--paper-raised);
}
.artifact-head,
.panel-intro {
display: grid;
grid-template-columns: minmax(0, 1.15fr) minmax(280px, .85fr);
gap: 30px;
padding: 28px;
}
.artifact-head { color: var(--paper); background: var(--ink); }
.artifact-head > div > p,
.panel-intro span,
.tensor-selector > span {
color: var(--copper);
font: .6rem/1.2 var(--mono);
letter-spacing: .12em;
}
.artifact-head h3,
.panel-intro h4 { margin: 10px 0 0; font-size: 1.23rem; line-height: 1.35; }
.artifact-head h3 { color: var(--paper); }
.artifact-head > p,
.panel-intro > p { margin: 0; font-size: .72rem; line-height: 1.75; }
.artifact-head > p { color: rgba(255,255,255,.66); }
.artifact-head code { color: var(--paper); font-size: .66rem; }
.artifact-tabs {
display: grid;
grid-template-columns: repeat(4, 1fr);
border-bottom: 1px solid var(--line);
}
.artifact-tabs button {
min-height: 76px;
padding: 13px 16px;
border: 0;
border-right: 1px solid var(--line);
color: var(--ink);
background: transparent;
text-align: left;
cursor: pointer;
}
.artifact-tabs button:last-child { border-right: 0; }
.artifact-tabs button[aria-selected="true"] { color: var(--paper); background: var(--copper); }
.artifact-tabs span,
.artifact-tabs b,
.artifact-tabs small { display: block; }
.artifact-tabs span { opacity: .65; font: .55rem/1 var(--mono); }
.artifact-tabs b { margin-top: 8px; font: 700 .67rem/1 var(--mono); letter-spacing: .05em; }
.artifact-tabs small { margin-top: 5px; opacity: .72; font-size: .61rem; }
.artifact-panel { padding-bottom: 26px; }
.panel-intro { padding: 24px 28px; border-bottom: 1px solid var(--line); }
.panel-intro > p { color: var(--muted); }
.layer-control,
.parameter-control {
display: grid;
grid-template-columns: 1.15fr .85fr;
gap: 24px;
align-items: center;
margin: 24px 28px;
padding: 16px;
border: 1px solid var(--line);
background: var(--paper);
}
.layer-control label,
.parameter-control label,
.repro-controls label {
color: var(--muted);
font: .62rem/1.4 var(--mono);
}
.layer-control label > span { display: flex; justify-content: space-between; }
.layer-control output { color: var(--copper); }
.layer-control input { width: 100%; margin-top: 12px; accent-color: var(--copper); }
.layer-legend { display: grid; grid-template-columns: 12px 1fr 12px 1fr 12px 1fr; gap: 8px; align-items: center; font: .58rem/1.3 var(--mono); }
.layer-legend i { width: 12px; height: 12px; }
.layer-legend .kda { background: var(--ink); }
.layer-legend .mla { background: var(--copper); }
.layer-legend .open { border: 2px solid var(--copper); background: transparent; }
.layer-strip {
display: grid;
grid-template-columns: repeat(31, minmax(7px, 1fr));
gap: 3px;
margin: 0 28px 24px;
}
.layer-strip button {
position: relative;
min-width: 0;
height: 32px;
padding: 0;
border: 1px solid transparent;
color: transparent;
background: var(--ink);
cursor: pointer;
}
.layer-strip button.mla { background: var(--copper); }
.layer-strip button.open::after {
content: "";
position: absolute;
inset: 3px;
border: 1px solid var(--paper);
}
.layer-strip button.terminal { box-shadow: 0 0 0 2px var(--copper); }
.layer-strip button.active { outline: 3px solid var(--signal); outline-offset: 1px; z-index: 2; }
.layer-strip span { font: .44rem/1 var(--mono); }
.layer-readout,
.artifact-metrics,
.benchmark-readout {
display: grid;
grid-template-columns: repeat(4, 1fr);
gap: 12px;
margin: 0 28px 24px;
}
.layer-readout article,
.artifact-metrics article,
.benchmark-readout article {
min-width: 0;
min-height: 128px;
padding: 18px;
border: 1px solid var(--line);
background: var(--paper);
}
.layer-readout span,
.artifact-metrics span,
.benchmark-readout span,
.local-build span,
.router-counterexample span,
.hypothesis-card > span {
color: var(--copper);
font: .57rem/1.2 var(--mono);
letter-spacing: .07em;
}
.layer-readout b,
.artifact-metrics b,
.benchmark-readout b { display: block; margin-top: 12px; font: 800 1.08rem/1.1 var(--mono); }
.layer-readout p,
.artifact-metrics p,
.benchmark-readout p,
.local-build p { margin: 12px 0 0; color: var(--muted); font-size: .66rem; line-height: 1.55; }
.layer-readout article.dark,
.artifact-metrics article.dark,
.benchmark-readout article.dark { color: var(--paper); background: var(--ink); }
.layer-readout article.dark p,
.artifact-metrics article.dark p,
.benchmark-readout article.dark p { color: rgba(255,255,255,.62); }
article.accent { border-color: rgba(173,100,69,.55); background: var(--copper-pale); }
.tensor-ledger {
display: grid;
grid-template-columns: repeat(4, 1fr);
margin: 0 28px 24px;
border-top: 1px solid var(--line);
border-left: 1px solid var(--line);
}
.tensor-ledger > div { padding: 16px; border-right: 1px solid var(--line); border-bottom: 1px solid var(--line); }
.tensor-ledger span { color: var(--copper); font: 800 .88rem/1 var(--mono); }
.tensor-ledger b { display: block; margin-top: 9px; font-size: .73rem; }
.tensor-ledger p { margin: 8px 0 0; color: var(--muted); font-size: .62rem; line-height: 1.5; }
.tensor-selector {
display: flex;
gap: 8px;
align-items: center;
margin: 0 28px 14px;
}
.tensor-selector > span { margin-right: auto; }
.tensor-selector button {
padding: 8px 10px;
border: 1px solid var(--line);
color: var(--ink);
background: var(--paper);
font: .57rem var(--mono);
cursor: pointer;
}
.tensor-selector button[aria-pressed="true"] { color: var(--paper); border-color: var(--copper); background: var(--copper); }
.tensor-table { margin: 0 28px 24px; border: 1px solid var(--line); background: var(--paper); }
.tensor-table header,
.tensor-table > div {
display: grid;
grid-template-columns: minmax(0, 1.8fr) 70px minmax(150px, 1fr) 100px;
gap: 14px;
align-items: center;
padding: 12px 14px;
border-bottom: 1px solid var(--line);
}
.tensor-table > div:last-child { border-bottom: 0; }
.tensor-table header { color: var(--paper); background: var(--ink); }
.tensor-table header span,
.tensor-table header b { font: .57rem/1.3 var(--mono); }
.tensor-table header b { grid-column: 2 / -1; }
.tensor-table code { overflow-wrap: anywhere; color: var(--ink); font-size: .59rem; }
.tensor-table > div span,
.tensor-table > div b,
.tensor-table > div small { font: .59rem/1.35 var(--mono); }
.tensor-table > div span { color: var(--copper); }
.tensor-table > div small { color: var(--muted); text-align: right; }
.shape-conflict {
display: grid;
grid-template-columns: 1fr 24px 1fr 24px 1fr;
gap: 10px;
align-items: center;
margin: 24px 28px;
}
.shape-conflict > div { min-height: 116px; padding: 18px; border: 1px solid var(--line); background: var(--paper); }
.shape-conflict > i,
.local-build > i { color: var(--copper); text-align: center; font-style: normal; }
.shape-conflict span { color: var(--copper); font: .57rem var(--mono); }
.shape-conflict b { display: block; margin-top: 12px; font: 800 .86rem var(--mono); }
.shape-conflict p { margin: 10px 0 0; color: var(--muted); font-size: .63rem; }
.shape-conflict .warn,
.local-build .warn { border-color: var(--signal); background: color-mix(in srgb, var(--signal) 8%, var(--paper)); }
.parameter-control label { display: grid; grid-template-columns: 1fr; gap: 9px; }
.parameter-control select,
.repro-controls select { width: 100%; padding: 9px; border: 1px solid var(--line); color: var(--ink); background: var(--paper-raised); font: .62rem var(--mono); }
.parameter-control > p { display: flex; justify-content: space-between; gap: 16px; margin: 0; font: .62rem var(--mono); }
.parameter-control > p b { color: var(--copper); }
.distribution { margin: 0 28px 24px; padding: 20px; border: 1px solid var(--line); background: var(--paper); }
.axis { position: relative; height: 30px; margin: 4px 4px 18px; border-top: 4px solid var(--ink); }
.axis::before,
.axis::after { content: ""; position: absolute; top: -8px; width: 3px; height: 12px; background: var(--copper); }
.axis::before { left: 0; }
.axis::after { right: 0; }
.axis i { position: absolute; top: -7px; width: 3px; height: 10px; background: var(--signal); }
.distribution-stats { display: grid; grid-template-columns: repeat(5, 1fr); gap: 8px; }
.distribution-stats article { padding: 12px; border: 1px solid var(--line); }
.distribution-stats span,
.distribution-stats b { display: block; font: .57rem/1.2 var(--mono); }
.distribution-stats span { color: var(--muted); }
.distribution-stats b { margin-top: 8px; color: var(--copper); }
.distribution [data-parameter-row] > p { margin: 16px 0 0; color: var(--muted); font: .6rem var(--mono); }
.hypothesis-card { margin: 0 28px 24px; padding: 20px; border: 1px dashed var(--signal); background: color-mix(in srgb, var(--signal) 6%, var(--paper)); }
.hypothesis-card h5,
.router-counterexample h5 { margin: 10px 0 16px; font-size: .9rem; }
.hypothesis-card > div { display: grid; grid-template-columns: repeat(3, 1fr); gap: 12px; }
.hypothesis-card p { margin: 0; padding: 14px; border: 1px solid var(--line); background: var(--paper); }
.hypothesis-card p b,
.hypothesis-card p span { display: block; }
.hypothesis-card p b { font: 800 .9rem var(--mono); }
.hypothesis-card p span { margin-top: 8px; color: var(--muted); font-size: .6rem; }
.repro-controls {
display: grid;
grid-template-columns: repeat(3, 1fr);
gap: 12px;
margin: 24px 28px;
}
.repro-controls label { display: grid; gap: 9px; padding: 14px; border: 1px solid var(--line); background: var(--paper); }
.local-build {
display: grid;
grid-template-columns: 1fr 24px 1fr 24px 1fr;
gap: 10px;
align-items: center;
margin: 0 28px 24px;
}
.local-build article { min-height: 138px; padding: 18px; border: 1px solid var(--line); background: var(--paper); }
.local-build b { display: block; margin-top: 12px; font: 800 .78rem/1.35 var(--mono); }
.router-counterexample {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 22px;
margin: 0 28px 24px;
padding: 20px;
border: 1px solid var(--line);
background: var(--paper);
}
.router-counterexample > div > p { color: var(--muted); font-size: .67rem; line-height: 1.6; }
.router-stats { display: grid; grid-template-columns: repeat(2, 1fr); gap: 8px; }
.router-stats p { margin: 0; padding: 12px; border: 1px solid var(--line); }
.router-stats span,
.router-stats b { display: block; }
.router-stats b { margin-top: 8px; font: 800 .87rem var(--mono); }
.boundary {
margin: 0 28px;
padding: 14px 16px;
border-left: 3px solid var(--copper);
background: var(--copper-pale);
}
.boundary b { font: .61rem var(--mono); }
.boundary p { margin: 7px 0 0; color: var(--muted); font-size: .65rem; line-height: 1.55; }
.boundary.danger { border-color: var(--signal); background: color-mix(in srgb, var(--signal) 7%, var(--paper)); }
.evidence-strip {
display: grid;
grid-template-columns: repeat(5, 1fr);
margin-top: 26px;
border-top: 1px solid var(--line);
}
.evidence-strip > div { padding: 15px; border-right: 1px solid var(--line); }
.evidence-strip > div:last-child { border-right: 0; }
.evidence-strip span { color: var(--paper); background: var(--copper); padding: 3px 5px; font: .58rem var(--mono); }
.evidence-strip b { display: block; margin-top: 10px; font: .58rem var(--mono); }
.evidence-strip p { margin: 8px 0 0; color: var(--muted); font-size: .58rem; line-height: 1.45; }
@media (max-width: 760px) {
.artifact-head,
.panel-intro,
.layer-control,
.parameter-control,
.router-counterexample { grid-template-columns: 1fr; gap: 16px; }
.artifact-tabs { grid-template-columns: repeat(2, 1fr); }
.artifact-tabs button:nth-child(2) { border-right: 0; }
.artifact-tabs button:nth-child(-n+2) { border-bottom: 1px solid var(--line); }
.layer-strip { grid-template-columns: repeat(31, minmax(5px, 1fr)); gap: 2px; }
.layer-strip button { height: 24px; }
.layer-readout,
.artifact-metrics,
.benchmark-readout,
.tensor-ledger { grid-template-columns: repeat(2, 1fr); }
.tensor-selector { align-items: stretch; flex-wrap: wrap; }
.tensor-selector > span { width: 100%; }
.tensor-table { overflow-x: auto; }
.tensor-table header,
.tensor-table > div { min-width: 690px; }
.shape-conflict,
.local-build { grid-template-columns: 1fr; }
.shape-conflict > i,
.local-build > i { transform: rotate(90deg); }
.distribution-stats { grid-template-columns: repeat(2, 1fr); }
.hypothesis-card > div,
.repro-controls { grid-template-columns: 1fr; }
.evidence-strip { grid-template-columns: 1fr; }
.evidence-strip > div { border-right: 0; border-bottom: 1px solid var(--line); }
}
@media (max-width: 440px) {
.artifact-head,
.panel-intro { padding: 21px; }
.layer-control,
.layer-strip,
.layer-readout,
.artifact-metrics,
.benchmark-readout,
.tensor-ledger,
.tensor-selector,
.tensor-table,
.shape-conflict,
.parameter-control,
.distribution,
.hypothesis-card,
.repro-controls,
.local-build,
.router-counterexample,
.boundary { margin-left: 16px; margin-right: 16px; }
.layer-readout,
.artifact-metrics,
.benchmark-readout,
.tensor-ledger,
.router-stats { grid-template-columns: 1fr; }
}
</style>
src/data/k3-artifact-snapshot.json
+600
View File
@@ -0,0 +1,600 @@
{
"schema_version": 1,
"captured_at": "2026-07-29T12:21:00+08:00",
"evidence_boundary": {
"checkpoint_forward_run": false,
"raw_weights_committed": false,
"router_inputs": "deterministic synthetic RMS-normalized vectors, not token hidden states",
"kda_retention_probe": "noncanonical channel-wise interpretation used only to expose the A_log shape ambiguity"
},
"provenance": {
"huggingface_model": "moonshotai/Kimi-K3",
"huggingface_revision": "9f62e4e9fffbd0a83ddd60e1c209d828994b3569",
"flashkda_revision": "1ce47ea3bb22c84eb9cc665028399cf35e8ffb0b",
"sha256": {
"config": "9710e121a58d03ac92c8d6da287a19541994319afbbe6d6202af001ffd379213",
"index": "a1c5210650ce71d2d3ae9ec5a101ac4afd3cf4b10091be589853437eb967febd",
"kda_slice": "dccb10e734a1b6756fe24233ef86c6417b704133a2ec4e56c5c5ddf87274d9b2",
"router_prefix": "dbe66ff82e58bbd273514c0a51b74043c519f64df4586166321cf6266e1723ee"
}
},
"checkpoint": {
"tensor_data_bytes": 1560860324864,
"tensor_data_tb": 1.560860324864,
"tensor_data_tib": 1.4195941956713796,
"shards": 96,
"shard_file_bytes": {
"sum": 1560936091448,
"min": 92289328,
"max": 16990916912,
"mean": 16259750952.583334
},
"tensor_entries": 497220,
"tensor_counts": {
"expert_packed": 247296,
"expert_scales": 247296,
"router_weight": 92,
"router_correction_bias": 92,
"attnres_proj": 187,
"attnres_norm": 187,
"kda_a_log": 69,
"kda_dt_bias": 69,
"vision": 165,
"projector": 3
},
"derived_routed_expert_bytes": 1446456066048,
"derived_routed_expert_share": 0.9267043584915465
},
"configuration": {
"layers": 93,
"dense_layers": 1,
"hidden": 7168,
"vocabulary": 163840,
"context": 1048576,
"kda_layers": [
1,
2,
3,
5,
6,
7,
9,
10,
11,
13,
14,
15,
17,
18,
19,
21,
22,
23,
25,
26,
27,
29,
30,
31,
33,
34,
35,
37,
38,
39,
41,
42,
43,
45,
46,
47,
49,
50,
51,
53,
54,
55,
57,
58,
59,
61,
62,
63,
65,
66,
67,
69,
70,
71,
73,
74,
75,
77,
78,
79,
81,
82,
83,
85,
86,
87,
89,
90,
91
],
"mla_layers": [
4,
8,
12,
16,
20,
24,
28,
32,
36,
40,
44,
48,
52,
56,
60,
64,
68,
72,
76,
80,
84,
88,
92,
93
],
"heads": 96,
"head_dim": 128,
"attnres_block": 12,
"experts": 896,
"active_experts": 16,
"shared_experts": 2,
"latent_width": 3584,
"expert_intermediate": 3072,
"situ_beta": 4.0,
"situ_linear_beta": 25.0,
"mla_nope": true,
"mla_output_gate": true
},
"tensor_examples": {
"mla_layer_4": [
{
"name": "language_model.model.layers.3.self_attn.kv_a_proj_with_mqa.weight",
"dtype": "BF16",
"shape": [
576,
7168
],
"bytes": 8257536
},
{
"name": "language_model.model.layers.3.self_attn.kv_b_proj.weight",
"dtype": "BF16",
"shape": [
24576,
512
],
"bytes": 25165824
},
{
"name": "language_model.model.layers.3.self_attn.q_a_proj.weight",
"dtype": "BF16",
"shape": [
1536,
7168
],
"bytes": 22020096
},
{
"name": "language_model.model.layers.3.self_attn.q_b_proj.weight",
"dtype": "BF16",
"shape": [
18432,
1536
],
"bytes": 56623104
},
{
"name": "language_model.model.layers.3.self_attn.g_proj.weight",
"dtype": "BF16",
"shape": [
12288,
7168
],
"bytes": 176160768
}
],
"vision": [
{
"name": "vision_tower.patch_embed.proj.weight",
"dtype": "BF16",
"shape": [
1024,
3,
14,
14
],
"bytes": 1204224
},
{
"name": "vision_tower.patch_embed.pos_emb.weight",
"dtype": "BF16",
"shape": [
64,
64,
1024
],
"bytes": 8388608
},
{
"name": "vision_tower.encoder.blocks.0.wqkv.weight",
"dtype": "BF16",
"shape": [
4608,
1024
],
"bytes": 9437184
},
{
"name": "vision_tower.encoder.blocks.26.wqkv.weight",
"dtype": "BF16",
"shape": [
4608,
1024
],
"bytes": 9437184
},
{
"name": "vision_tower.encoder.final_layernorm.weight",
"dtype": "BF16",
"shape": [
1024
],
"bytes": 2048
}
],
"routed_expert_0": [
{
"name": "w1.weight_packed",
"dtype": "U8",
"shape": [
3072,
1792
],
"bytes": 5505024
},
{
"name": "w1.weight_scale",
"dtype": "U8",
"shape": [
3072,
112
],
"bytes": 344064
},
{
"name": "w2.weight_packed",
"dtype": "U8",
"shape": [
3584,
1536
],
"bytes": 5505024
},
{
"name": "w2.weight_scale",
"dtype": "U8",
"shape": [
3584,
96
],
"bytes": 344064
},
{
"name": "w3.weight_packed",
"dtype": "U8",
"shape": [
3072,
1792
],
"bytes": 5505024
},
{
"name": "w3.weight_scale",
"dtype": "U8",
"shape": [
3072,
112
],
"bytes": 344064
}
]
},
"parameter_audit": {
"a_log_checkpoint_shape": [
128
],
"a_log_public_code_shape": [
96
],
"dt_bias_shape": [
96,
128
],
"beta_projection_shape": [
96,
7168
],
"status": "observed shape inconsistency; runtime meaning unresolved",
"a_log": {
"count": 128,
"mean": -0.16894778609275818,
"std": 0.3667531907558441,
"quantiles": {
"min": -0.7530547380447388,
"p01": -0.7163181900978088,
"p10": -0.5367198586463928,
"p25": -0.4014696776866913,
"p50": -0.1533127874135971,
"p75": 0.0,
"p90": 0.07427694648504257,
"p99": 0.7512762546539307,
"max": 2.4661004543304443
}
},
"a_rate_exp": {
"count": 128,
"mean": 0.9474229216575623,
"std": 1.0007015466690063,
"quantiles": {
"min": 0.47092580795288086,
"p01": 0.48882821202278137,
"p10": 0.5846699476242065,
"p25": 0.6693384051322937,
"p50": 0.8578758239746094,
"p75": 1.0,
"p90": 1.0771057605743408,
"p99": 2.1350440979003906,
"max": 11.776434898376465
}
},
"dt_bias": {
"count": 12288,
"mean": -4.57477331161499,
"std": 1.423690915107727,
"quantiles": {
"min": -7.8938117027282715,
"p01": -6.962827682495117,
"p10": -6.4912495613098145,
"p25": -5.797565460205078,
"p50": -4.622031211853027,
"p75": -3.4063913822174072,
"p90": -2.568972587585449,
"p99": -1.895374059677124,
"max": 0.179231196641922
}
},
"channelwise_hypothesis": {
"log_decay": {
"count": 12288,
"mean": -0.22606194019317627,
"std": 0.26349756121635437,
"quantiles": {
"min": -2.685786724090576,
"p01": -1.122031569480896,
"p10": -0.6017768383026123,
"p25": -0.3287131190299988,
"p50": -0.12320166081190109,
"p75": -0.035499051213264465,
"p90": -0.008835914544761181,
"p99": -8.967738722276408e-06,
"max": 0.0
}
},
"one_step_retention": {
"count": 12288,
"mean": 0.821946918964386,
"std": 0.17656706273555756,
"quantiles": {
"min": 0.068167544901371,
"p01": 0.32561764121055603,
"p10": 0.5478373765945435,
"p25": 0.7198495268821716,
"p50": 0.8840853571891785,
"p75": 0.9651236534118652,
"p90": 0.9912030100822449,
"p99": 0.9999910593032837,
"max": 1.0
}
},
"retention_after_64_steps": {
"count": 12288,
"mean": 0.13120755553245544,
"std": 0.253229558467865,
"quantiles": {
"min": 0.0,
"p01": 6.506081464145866e-32,
"p10": 1.8781597374798298e-17,
"p25": 7.302480842241721e-10,
"p50": 0.00037638185312971473,
"p75": 0.10311195999383926,
"p90": 0.5680769681930542,
"p99": 0.9994263052940369,
"max": 1.0
}
}
},
"router_correction_bias": {
"count": 896,
"mean": 6.285762310653809e-07,
"std": 0.02034943550825119,
"quantiles": {
"min": -0.08442370593547821,
"p01": -0.06806425005197525,
"p10": -0.0273849219083786,
"p25": -0.009373871609568596,
"p50": 0.004928342066705227,
"p75": 0.014873139560222626,
"p90": 0.020546497777104378,
"p99": 0.02667994052171707,
"max": 0.028605474159121513
}
},
"router_row_l2": {
"count": 896,
"mean": 4.949629783630371,
"std": 0.895085871219635,
"quantiles": {
"min": 2.6018569469451904,
"p01": 2.9133214950561523,
"p10": 3.7345614433288574,
"p25": 4.310352325439453,
"p50": 5.036390781402588,
"p75": 5.6203179359436035,
"p90": 6.068215370178223,
"p99": 6.65656042098999,
"max": 7.023199081420898
}
},
"router_bias_norm_correlation": 0.456669420003891
},
"router_stress_probe": {
"seed": 20260729,
"synthetic_tokens": 2048,
"hidden_rms": 0.999999463558197,
"without_correction_bias": {
"mean": 36.57143020629883,
"std": 76.2339096069336,
"cv": 2.0845208168029785,
"gini": 0.8310886025428772,
"zero_experts": 558,
"quantiles": {
"min": 0.0,
"p10": 0.0,
"p25": 0.0,
"p50": 0.0,
"p75": 16.0,
"p90": 166.0,
"p99": 310.0999755859375,
"max": 369.0
}
},
"with_correction_bias": {
"mean": 36.57143020629883,
"std": 92.48487854003906,
"cv": 2.528883218765259,
"gini": 0.878814697265625,
"zero_experts": 673,
"quantiles": {
"min": 0.0,
"p10": 0.0,
"p25": 0.0,
"p50": 0.0,
"p75": 0.0,
"p90": 167.0,
"p99": 398.0999755859375,
"max": 451.0
}
},
"membership_overlap_mean": 2.1708984375,
"tokens_changed": 2048,
"changed_fraction": 1.0,
"mean_replacements_per_token": 13.8291015625
},
"flashkda": {
"supported_architectures": [
"90a",
"100a",
"103a",
"120a"
],
"requirements": {
"cuda": ">=12.9",
"pytorch": ">=2.4",
"gpu": "SM90+"
},
"official_benchmarks": {
"h20": {
"sequence": 8192,
"heads": 96,
"dimension": 128,
"rows": {
"Fixed": {
"flash_kda_ms": 2.622,
"fla_chunk_kda_ms": 4.8388,
"speedup_vs_chunk_kda": 1.85,
"fla_chunk_gdn_ms": 3.1985,
"speedup_vs_gdn": 1.22
},
"Varlen, `seq_lens`=[1300, 547, 2048, 963, 271, 3063]": {
"flash_kda_ms": 2.3449,
"fla_chunk_kda_ms": 4.8291,
"speedup_vs_chunk_kda": 2.06,
"fla_chunk_gdn_ms": 3.0541,
"speedup_vs_gdn": 1.3
},
"Varlen, `seq_lens`=`1024 x 8`": {
"flash_kda_ms": 2.0432,
"fla_chunk_kda_ms": 4.6723,
"speedup_vs_chunk_kda": 2.29,
"fla_chunk_gdn_ms": 2.9117,
"speedup_vs_gdn": 1.43
}
}
},
"gb200": {
"sequence": 8192,
"heads": 96,
"dimension": 128,
"rows": {
"Fixed": {
"flash_kda_ms": 1.0087,
"fla_chunk_kda_ms": 2.3271,
"speedup_vs_chunk_kda": 2.31,
"fla_chunk_gdn_ms": 1.2792,
"speedup_vs_gdn": 1.27
},
"Varlen, `seq_lens`=[1300, 547, 2048, 963, 271, 3063]": {
"flash_kda_ms": 0.8597,
"fla_chunk_kda_ms": 2.334,
"speedup_vs_chunk_kda": 2.71,
"fla_chunk_gdn_ms": 1.2962,
"speedup_vs_gdn": 1.51
},
"Varlen, `seq_lens`=`1024 x 8`": {
"flash_kda_ms": 0.7064,
"fla_chunk_kda_ms": 2.3105,
"speedup_vs_chunk_kda": 3.27,
"fla_chunk_gdn_ms": 1.2744,
"speedup_vs_gdn": 1.8
}
}
}
},
"local_environment": {
"python": "3.10.14",
"torch": "2.11.0+cu128",
"torch_cuda": "12.8",
"gpu": "NVIDIA GeForce RTX 5090",
"capability": [
12,
0
],
"libc": [
"glibc",
"2.43"
]
},
"local_build": {
"status": "blocked_before_kernel execution",
"attempt_1": "system g++ 15 exceeds CUDA 12.8 host compiler range",
"attempt_2": "temporary g++ 13 reaches nvcc, then CUDA 12.8 headers conflict with current glibc math declarations",
"interpretation": "GPU architecture is listed by the repository, but the local CUDA 12.8 stack is below the official CUDA 12.9 requirement"
}
}
}
src/data/k3Artifacts.ts
+36
View File
@@ -0,0 +1,36 @@
import snapshot from "./k3-artifact-snapshot.json";
export { snapshot as k3ArtifactSnapshot };
const mlaLayerSet = new Set(snapshot.configuration.mla_layers);
export const k3ArtifactLayers = Array.from(
{ length: snapshot.configuration.layers },
(_, index) => {
const layer = index + 1;
const attention = mlaLayerSet.has(layer) ? "MLA" : "KDA";
return {
layer,
attention,
feedForward: layer === 1 ? "DENSE" : "MOE",
block: Math.floor(index / snapshot.configuration.attnres_block) + 1,
opensResidualBlock: index % snapshot.configuration.attnres_block === 0,
terminalGlobal: layer === snapshot.configuration.layers,
};
},
);
export const k3ArtifactViews = [
["layers", "01", "LAYER MAP", "93 层配置"],
["tensors", "02", "TENSOR ANATOMY", "497,220 entries"],
["parameters", "03", "PARAMETER AUDIT", "真实权重小切片"],
["reproduction", "04", "REPRODUCTION", "作者值与本机边界"],
] as const;
export const k3ArtifactEvidence = [
["O", "OFFICIAL ARTIFACT", "官方 config、code、index、header 或参数字节直接观测。"],
["D", "DERIVATION", "由公开 shape 与计数做确定性算术。"],
["X", "EXECUTED", "本站机器实际执行并留下环境与结果。"],
["S", "SYNTHETIC", "真实权重加合成输入；只测试边界，不代表真实 token。"],
["U", "UNRESOLVED", "工件之间存在不一致，当前不擅自补解释。"],
] as const;
src/pages/index.astro
+7 -7
View File
@@ -128,19 +128,19 @@ const paths = [
<div class="release-grid">
<a class="release-card k3-release" href="/k3/">
<div>
<p class="eyebrow"><span>NEW / K3 ROUND 02</span> REPORT · FORMULA · SYSTEM · EVIDENCE</p>
<h2>47 页不再压成摘要：把 K3 的每个因果环节重新展开</h2>
<p class="eyebrow"><span>NEW / K3 ROUND 03</span> REPORT · CHECKPOINT · KERNEL · BOUNDARY</p>
<h2>47 页不再压成摘要：再把 1.56 TB 开放工件接回报告</h2>
<p>
用三十二张问题账逐节读完 KDA、Gated MLA、AttnRes、Stable LatentMoE、原生视觉、
预训练、九专家 MOPD、Agent 环境、FlashKDA / MoonEP、混合 prefix cache、评测与案例边界。
在三十二张报告问题账之外，继续审计 96 个 safetensors 分片、497,220 个 tensor entries、
真实 KDA / MLA / MoE / MoonViT shape、小范围权重统计、FlashKDA 编译边界与未决形状矛盾。
</p>
</div>
<dl>
<div><dt>REPORT</dt><dd>16 Figures · 5 Tables</dd></div>
<div><dt>NODES</dt><dd>100 个一手 / 官方节点</dd></div>
<div><dt>LAB</dt><dd>Delta · Decay · AttnRes · MoE · QB · RL · Cache</dd></div>
<div><dt>ARTIFACTS</dt><dd>96 shards · 497,220 entries</dd></div>
<div><dt>LAB</dt><dd>8 报告实验 + 4 工件视图</dd></div>
</dl>
<span class="release-arrow" aria-hidden="true">从报告目录进入完整因果链 →</span>
<span class="release-arrow" aria-hidden="true">从报告目录进入开放工件证据链 →</span>
</a>
<a class="release-card deepseek-release" href="/deepseek/">
<div>
src/pages/k3/index.astro
+55 -8
View File
@@ -1,6 +1,7 @@
---
import BaseLayout from "@/layouts/BaseLayout.astro";
import ArchitectureExplorer from "@/components/ArchitectureExplorer.astro";
import K3ArtifactLab from "@/components/K3ArtifactLab.astro";
import K3ReportLab from "@/components/K3ReportLab.astro";
import { k3FigureAtlas, k3Ledgers, k3PaperChain, k3ReportMap } from "@/data/k3";
@@ -34,7 +35,8 @@ const toc = [
["26", "cases", "案例边界"],
["27", "xtml", "XTML 协议"],
["28", "lab", "八联交互实验"],
["29", "audit", "21 张图表审计"],
["29", "artifacts", "开放权重工件审计"],
["30", "audit", "21 张图表审计"],
["↳", "papers", "100 节点阅读链"],
];
@@ -103,13 +105,13 @@ const paperGroups = [
<BaseLayout
title="Kimi K3 技术报告完整深读：架构、训练、RL、系统与评测"
description="用三十二张问题账、二十一张图表审计、八个交互实验与一百个一手阅读节点，逐节读懂 Kimi K3 技术报告。"
description="用三十二张问题账、二十一张图表审计、八个机制实验、四个开放工件视图与一百个一手阅读节点，逐节读懂 Kimi K3。"
section="k3"
>
<header class="page-hero k3-hero">
<div class="page-hero-inner">
<div>
<p class="eyebrow"><span>ANCHOR REPORT / ROUND 02</span> KIMI K3 · 47 PAGES</p>
<p class="eyebrow"><span>ANCHOR REPORT / ROUND 03</span> KIMI K3 · REPORT → OPEN ARTIFACTS</p>
<h1>不把报告压成摘要<br />把每个因果环节<br />重新展开</h1>
<p class="lead">
K3 同时扩展序列、深度、宽度、视觉与 Agent 轨迹。真正值得读的不是 2.8T 这个最大数字，
@@ -119,10 +121,11 @@ const paperGroups = [
<dl class="page-facts">
<div><dt>QUESTIONS</dt><dd>32 张问题账</dd></div>
<div><dt>REPORT</dt><dd>16 Figures · 5 Tables</dd></div>
<div><dt>LABS</dt><dd>8 个可操作实验</dd></div>
<div><dt>LABS</dt><dd>8 个机制实验 + 4 个工件视图</dd></div>
<div><dt>READING</dt><dd>100 个一手 / 官方节点</dd></div>
<div><dt>MODEL</dt><dd>2.78T total / 104.2B active</dd></div>
<div><dt>STATUS</dt><dd>K3 二轮深读</dd></div>
<div><dt>ARTIFACTS</dt><dd>96 shards · 497,220 tensors</dd></div>
<div><dt>STATUS</dt><dd>K3 三轮进行中</dd></div>
</dl>
</div>
</header>
@@ -137,7 +140,8 @@ const paperGroups = [
</ol>
<div class="rail-note">
<b>证据约定</b>
R = K3 报告；P = 原论文 / 官方实现；D = 确定性推导；T = 教学模型。四者不互相冒充。
报告层用 R / P / D / T 区分报告、原始来源、推导与教学模型；
工件层用 O / D / X / S / U 区分观测、推导、本站执行、合成探针与未决矛盾。
</div>
</aside>
@@ -844,8 +848,29 @@ const paperGroups = [
<K3ReportLab />
</section>
<section class="article-section" id="artifacts">
<p class="eyebrow"><span>29</span> OPEN ARTIFACT FORENSICS</p>
<h2>从“报告说了什么”走到“公开 checkpoint 实际长什么样”</h2>
<p class="lede">
第三轮固定到官方 Hugging Face revision，读取 config、remote code、60 MB tensor index、
四个 safetensors headers 和两个小范围参数切片。原始权重不进入本站仓库；
结构、shape、计数、参数统计与本机编译边界都可以从公开脚本重复生成。
</p>
<div class="artifact-callout">
<article><span>O / OBSERVED</span><b>1.4196 TiB tensor data</b><p>96 shards、497,220 entries；不是运行显存，也不是参数量口径。</p></article>
<article><span>D / CLOSED LOOP</span><b>69 KDA · 24 MLA · 92 MoE</b><p>配置、tensor names 与 header shape 三方闭合。</p></article>
<article class="warning"><span>U / UNRESOLVED</span><b>A_log [128] ≠ expected [96]</b><p>checkpoint 与公开代码 / kernel API 的形状冲突保留在主视区，不擅自解释。</p></article>
</div>
<K3ArtifactLab />
<div class="hero-actions">
<a class="button primary" href="https://huggingface.co/moonshotai/Kimi-K3">打开官方开放权重</a>
<a class="button" href="https://github.com/MoonshotAI/FlashKDA">打开 FlashKDA 官方实现</a>
<a class="button" href="https://github.com/MoonshotAI/FlashKDA/blob/master/BENCHMARK_GB200.md">核对作者 GB200 benchmark</a>
</div>
</section>
<section class="article-section" id="audit">
<p class="eyebrow"><span>29</span> FIGURE & TABLE AUDIT</p>
<p class="eyebrow"><span>30</span> FIGURE & TABLE AUDIT</p>
<h2>Figure 1–16、Table 1–5：每张图究竟支持什么，不能支持什么</h2>
<div class="figure-atlas">
{k3FigureAtlas.map(([id, report, title, contract]) => (
@@ -900,6 +925,28 @@ const paperGroups = [
<style>
.k3-hero { border-bottom-color: var(--copper); }
.anchor-alias { position: relative; top: -88px; display: block; visibility: hidden; }
.artifact-callout {
display: grid;
grid-template-columns: repeat(3, minmax(0, 1fr));
max-width: 1080px;
margin: 32px 0;
border-top: 1px solid var(--line);
border-left: 1px solid var(--line);
}
.artifact-callout article {
min-height: 145px;
padding: 20px;
border-right: 1px solid var(--line);
border-bottom: 1px solid var(--line);
background: var(--paper-raised);
}
.artifact-callout article.warning {
border-color: var(--signal);
background: color-mix(in srgb, var(--signal) 7%, var(--paper));
}
.artifact-callout span { color: var(--copper); font: .58rem/1.2 var(--mono); letter-spacing: .08em; }
.artifact-callout b { display: block; margin-top: 13px; font: 800 .85rem/1.3 var(--mono); }
.artifact-callout p { margin: 12px 0 0; color: var(--muted); font-size: .68rem; line-height: 1.6; }
.ledger-grid {
display: grid;
grid-template-columns: repeat(3, minmax(0, 1fr));
@@ -1200,7 +1247,7 @@ const paperGroups = [
.xtml-stage > i:nth-of-type(n+3) { display: none; }
}
@media (max-width: 720px) {
.ledger-grid, .axis-grid, .domain-grid, .agentenv-grid, .serving-grid, .eval-axes, .case-grid,
.artifact-callout, .ledger-grid, .axis-grid, .domain-grid, .agentenv-grid, .serving-grid, .eval-axes, .case-grid,
.mechanism-steps, .protocol-grid, .comparison-grid, .layer-separation, .precision-contract,
.two-column, .environment-list, .figure-atlas, .audit-legend, .harness-parts {
grid-template-columns: 1fr;
src/pages/progress/index.astro
+6 -3
View File
@@ -9,7 +9,7 @@ const researching = chapters.filter((chapter) => ["researching", "drafting"].inc
const workstreams = [
{ label: "研究框架与规范", value: 83, next: "给 Scaling 与推理专题补逐篇图表/实验精读层级" },
{ label: "网站设计系统", value: 89, next: "打印样式与更多通用可视化组件" },
{ label: "Kimi K3 深读", value: 88, next: "第三轮加入官方权重 traces、独立复现与逐图数值重绘" },
{ label: "Kimi K3 深读", value: 92, next: "在匹配 CUDA 12.9+ 环境执行 FlashKDA，并接入真实 hidden-state / expert-load traces" },
{ label: "语言模型前史", value: 78, next: "逐图精读 Kneser–Ney、LSTM 与 Bahdanau，并加入真实小语料复现" },
{ label: "Transformer 基础", value: 79, next: "逐图精读多头电路、Pre/Post-LN 与真实 kernel / KV 配置" },
{ label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
@@ -50,7 +50,7 @@ const workstreams = [
<div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
<div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
<div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
<div><dt>UPDATED</dt><dd>2026-07-29 11:54 CST</dd></div>
<div><dt>UPDATED</dt><dd>2026-07-29 12:40 CST</dd></div>
<div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div>
</dl>
</div>
@@ -97,13 +97,14 @@ const workstreams = [
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
<article><span>✓</span><h3>六十七个原创交互视图</h3><p>K3 三轴图与八联实验、语言模型前史、Transformer、表示深度、DeepSeek 四联实验、长上下文、MoE、推理、Agent、多模态，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
<article><span>✓</span><h3>七十一个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，以及语言模型前史、Transformer、表示深度、DeepSeek、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
<article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
<article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
<article><span>✓</span><h3>DeepSeek 技术谱系二轮深读</h3><p>二十四张问题账、十次技术转向、60 个一手/官方节点，以及稀疏容量—MLA 缓存—V3 协同—RL 偏差四联实验。</p></article>
<article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
<article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
<article><span>✓</span><h3>Scaling Laws 深度专题</h3><p>九张账、29 个一手节点、DeepSeek/Kimi 双谱系与曲面—部署—复用—涌现四联实验。</p></article>
<article><span>✓</span><h3>数据工程深度专题</h3><p>十二张账、31 个一手节点、DeepSeek/Kimi 双谱系与流水线—去重—混合—改写四联实验。</p></article>
<article><span>✓</span><h3>长上下文深度专题</h3><p>五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。</p></article>
@@ -209,6 +210,8 @@ const workstreams = [
<div><time>2026-07-29</time><b>K3 二轮按三十二张账重建</b><p>从架构组件摘要升级为覆盖预训练、后训练、环境、系统、评测、案例与附录的完整报告因果链。</p></div>
<div><time>2026-07-29</time><b>K3 原生视觉事实纠错</b><p>MoonViT-V2 从头训练；视觉与文本从训练开始在同一个 NTP objective 中联合优化，不再沿用冻结/解冻式 post-hoc 叙述。</p></div>
<div><time>2026-07-29</time><b>K3 图表与实验永久分级</b><p>Figure 1–16 / Table 1–5 建立视觉契约；报告事实、原论文、确定性推导与教学模型使用 R/P/D/T 四种身份。</p></div>
<div><time>2026-07-29</time><b>K3 开放工件按五种证据身份审计</b><p>真实观测 O、确定性推导 D、本机执行 X、合成探针 S 与未决矛盾 U 分开；作者 benchmark 不冒充本站实测。</p></div>
<div><time>2026-07-29</time><b>A_log 形状冲突保持未决</b><p>checkpoint 的 [128] 与 config / remote code / FlashKDA API 期待的 [96] 并列展示；不宣布权重损坏，也不把 channel-wise 假设写成真实 forward。</p></div>
</div>
</section>
src/pages/roadmap/index.astro
+2 -2
View File
@@ -105,9 +105,9 @@ const stages = [
<p>K3 架构 → 07/06/03 → 05/04 → 11/12 → 08/14/15</p>
<ol>
<li>沿 32 张问题账解释三维信息流</li>
<li>用 8 个实验比较 KDA、MLA、AttnRes 与 LatentMoE</li>
<li>用 8 个报告实验与 4 个工件视图比较机制和真实 shape</li>
<li>分清 2.78T / 104.2B、2.5× 与 1M 的证据口径</li>
<li>读懂九专家 MOPD、AgentENV、混合缓存与评测协议</li>
<li>读懂九专家 MOPD、混合缓存、FlashKDA 与复现边界</li>
</ol>
</article>
<article>