wuyang/llm-atlas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: map DeepSeek Chat sampling robustness

Browse Source
This commit is contained in:
wuyang
2026-07-30 02:01:50 +08:00
parent 18b16e2fdc
commit 580f69675c
21 changed files with 108346 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PROGRESS.md
+14 -4
View File
@@ -1,6 +1,6 @@
# 持续进度 # 持续进度
最后更新：2026-07-29 最后更新：2026-07-30
## 总体状态 ## 总体状态
@@ -14,7 +14,7 @@
| 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 | | 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
| Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 | | Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
| 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces | | 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
| DeepSeek 专题 | 五轮实证进行中 | 99% | 扩大 completion/task 样本与 sampling 复现，再推进 SM90 FlashMLA、FP8/pipeline、干预式 mediation 与 R1-like RL 复现 | | DeepSeek 专题 | 六轮实证进行中 | 99% | 扩大 sampling 的 source/task 覆盖，再推进干预式 mediation、SM90 FlashMLA、FP8/pipeline 与 R1-like RL 复现 |
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 | | 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 | | 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 | | 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
@@ -41,7 +41,7 @@
- [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。 - [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
- [x] 完成可检索、可按专题筛选的论文库页面。 - [x] 完成可检索、可按专题筛选的论文库页面。
- [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。 - [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 十九联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十六个原创交互视图。 - [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 二十联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十七个原创交互视图。
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。 - [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。 - [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。 - [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
@@ -248,11 +248,18 @@
- [x] 全深度新进程复跑的 4 / 4 source objects（去 runtime 后）、1,856 / 1,856 hidden tensor hashes、1,664 / 1,664 ordered-route hashes 与 1,664 / 1,664 route-weight hashes 全部 exact；复现核对抓到并修正 source-level content-token hash 的循环变量缺陷，正式与复跑文件随后全部重跑。 - [x] 全深度新进程复跑的 4 / 4 source objects（去 runtime 后）、1,856 / 1,856 hidden tensor hashes、1,664 / 1,664 ordered-route hashes 与 1,664 / 1,664 route-weight hashes 全部 exact；复现核对抓到并修正 source-level content-token hash 的循环变量缺陷，正式与复跑文件随后全部重跑。
- [x] 第十九个 DeepSeek 交互实验用四页签分开 128→512 completion、Math/Code evaluator、29-stage hidden divergence 与 26-gate route divergence；桌面、390px 移动端、键盘 tab、分域/分边/分指标交互与无横向溢出闸门通过。 - [x] 第十九个 DeepSeek 交互实验用四页签分开 128→512 completion、Math/Code evaluator、29-stage hidden divergence 与 26-gate route divergence；桌面、390px 移动端、键盘 tab、分域/分边/分指标交互与无横向溢出闸门通过。
- [x] DeepSeek completion/full-depth 版本以源提交 `8bb488f`、不可变镜像 `20260729T162625Z-8bb488f` 发布；OCI index digest `sha256:297fef91…b0c02`，复用 NAS `12010→8080`、NPM host 31 / cert 41、门户 `LLM ATLAS / projects / 180`，HTTPS/2、gzip 与十六套生产 Chrome 回归全通过；保留 `20260729T144827Z-96443d4` 回滚。 - [x] DeepSeek completion/full-depth 版本以源提交 `8bb488f`、不可变镜像 `20260729T162625Z-8bb488f` 发布；OCI index digest `sha256:297fef91…b0c02`，复用 NAS `12010→8080`、NPM host 31 / cert 41、门户 `LLM ATLAS / projects / 180`，HTTPS/2、gzip 与十六套生产 Chrome 回归全通过；保留 `20260729T144827Z-96443d4` 回滚。
- [x] DeepSeek Round 06 在看到正式结果前预注册 official sampling 合同：4 条固定 source、8 个 SHA-256 派生 base seed、8 条固定 batch 行、`temperature=.3 / top_p=.95 / top_k=0` 与 512-token 统一上限；明确 batch-seed aligned 不是 common-random-number paired design。
- [x] 16-token smoke 的 32 / 32 prompt hash 与 greedy baseline exact，R0 同进程重放 32 / 32 全合同 exact，R0/R1 的 32 个同格中 18 格分叉；正式 256 条 sampled outputs 全部落盘，251 条 natural EOS、5 条截断、242 个完整 token trajectory hashes。
- [x] sampling 独立 evaluator 把 stopping、task terminal、coverage 与 correctness 分账：单条 GSM8K 的 64 个样本中 62 个 strict exact；单条 HumanEval 的 64 个样本全部 AST 可解析并进入 pinned sandbox，63 个通过 official tests。两条数学失败与一个偶数 prime-check 失败均定位到具体推理/代码错误。
- [x] 新进程只复跑 R0/R1 的 64 格；run seed、prompt hash、完整 generated token IDs、decoded text、EOS、truncation、CPU RNG pre-state 与 CUDA RNG pre-state 八项均 64 / 64 exact，不冒充 256 / 256。
- [x] Round 06 四份冻结 artifact SHA-256 为 formal `46c7edf…345af`、eval `078f486…24c4d8`、rerun `72d050e…91f6a0`、reproduction `4d59a77…f82d15`；compact builder 强制验证三段 hash 链与 64 / 64 复现闸门。
- [x] 第二十个 DeepSeek 交互实验以四页签讲解八 seed 轨迹显微镜、Math/Code 失败账、aligned/nearest 样本集合比较与随机性×复现合同；完整协议、结果审计、运行脚本与 compact 数据均已记录在项目本地。
- [x] Round 06 本地闸门通过：80 个受检文件零诊断，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；DeepSeek 历史全量与 sampling 专项真实 Chrome 回归均通过，桌面/390px 移动端无文档级溢出、无 offender、无运行时异常。
## 正在进行 ## 正在进行
- [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。 - [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。
- [ ] DeepSeek 五轮下一闸门：扩大 GSM8K/HumanEval 与语言 source 样本，加入 sampling robustness 和干预式 mediation；再推进 SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。 - [ ] DeepSeek 六轮下一闸门：扩大 GSM8K/HumanEval 与语言 source 的 sampling 覆盖，加入干预式 mediation；再推进 SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。
- [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。 - [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
- [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。 - [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。 - [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
@@ -426,6 +433,9 @@
| 2026-07-29 | 开放工件证据使用 O / D / X / S / U 五种身份 | 观测、推导、本机执行、合成探针与未决矛盾不互相冒充；FlashKDA 作者 benchmark 不写成本机 benchmark | | 2026-07-29 | 开放工件证据使用 O / D / X / S / U 五种身份 | 观测、推导、本机执行、合成探针与未决矛盾不互相冒充；FlashKDA 作者 benchmark 不写成本机 benchmark |
| 2026-07-29 | `A_log [128]` 与 expected `[96]` 保持未决 | 并列报告 checkpoint、config、remote code 与 kernel API；等待官方 loader / 修订解释，不擅自 reshape | | 2026-07-29 | `A_log [128]` 与 expected `[96]` 保持未决 | 并列报告 checkpoint、config、remote code 与 kernel API；等待官方 loader / 修订解释，不擅自 reshape |
| 2026-07-29 | K3 三轮开放工件里程碑用不可变镜像 `20260729T044605Z-be2b291` 发布 | OCI digest `sha256:99aa953e…d00cdf`；复用 `12010→8080`、NPM host 31 / cert 41、门户 order 180 与公开 Forgejo；十六套生产 Chrome 回归通过，保留 `20260729T040336Z-b669615` 回滚 | | 2026-07-29 | K3 三轮开放工件里程碑用不可变镜像 `20260729T044605Z-be2b291` 发布 | OCI digest `sha256:99aa953e…d00cdf`；复用 `12010→8080`、NPM host 31 / cert 41、门户 order 180 与公开 Forgejo；十六套生产 Chrome 回归通过，保留 `20260729T040336Z-b669615` 回滚 |
| 2026-07-30 | DeepSeek Round 06 在结果前冻结多种子 sampling 协议 | official `.3/.95` sampling、显式 top-k 0、四条 source、八个 SHA-256 seed 与固定八行 batch；batch-seed aligned 不冒充 common random numbers |
| 2026-07-30 | sampled completion 与任务失败分账 | 256 条中 251 natural EOS、242 unique token trajectories；Math 62/64、Code 63/64，具体错误回到推理与官方 tests |
| 2026-07-30 | sampling 的随机性与复现同时过闸 | R0/R1 31/32 同格分叉；新进程复跑 64 格的 seed、prompt、token、text、stop 与 CPU/CUDA RNG pre-state 八项全部 exact |
## 未决问题 ## 未决问题
README.md
+12 -4
View File
@@ -19,7 +19,7 @@
当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读， 当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读，
语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题， 语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
以及 86 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、 以及 87 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。 8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、 第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、
497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图 497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图
@@ -27,8 +27,8 @@
`sm_120a` wheel，在 RTX 5090 上完成 6/6 官方参考 exact-match 和 K3 fixed / varlen 形状计时。详见 `sm_120a` wheel，在 RTX 5090 上完成 6/6 官方参考 exact-match 和 K3 fixed / varlen 形状计时。详见
[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与 [K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。 [checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
DeepSeek 五轮专题以 24 张问题账、10 次技术转向、 DeepSeek 六轮专题以 24 张问题账、10 次技术转向、
19 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4； 20 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、 并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、
MLA/HF eager cache shapes 与 `31/31` exact 独立复跑；进一步用真实 layer-1 权重执行官方 V3 MLA/HF eager cache shapes 与 `31/31` exact 独立复跑；进一步用真实 layer-1 权重执行官方 V3
naive/absorb 路径，实际写入 576 元素 latent cache，并以 FP32 将两种结合顺序的最大误差压到 naive/absorb 路径，实际写入 576 元素 latent cache，并以 FP32 将两种结合顺序的最大误差压到
@@ -72,6 +72,12 @@ checkpoint 又执行 embedding、27 层、final norm 与 26 个 MoE gate 的 pro
全深度 trace，在 1,537 个精确 interior content tokens / condition 上记录 全深度 trace，在 1,537 个精确 interior content tokens / condition 上记录
1,918,176 次 top-6 路由决定。新进程子集复跑达到 `1,856 / 1,856` hidden tensor hashes、 1,918,176 次 top-6 路由决定。新进程子集复跑达到 `1,856 / 1,856` hidden tensor hashes、
`1,664 / 1,664` ordered route hashes 与 `1,664 / 1,664` route-weight hashes exact。 `1,664 / 1,664` ordered route hashes 与 `1,664 / 1,664` route-weight hashes exact。
Round 06 冻结 4 条 source、8 个 SHA-256 派生 seed 与 8 个 condition，启用 checkpoint
随附的 `temperature=.3 / top_p=.95` 并显式 `top_k=0`，生成 256 条 sampled outputs：
251 条 natural EOS，242 个完整 token trajectory hashes；单条 GSM8K 的 strict exact
为 `62 / 64`，单条 HumanEval 的官方 tests pass 为 `63 / 64`，不外推为 benchmark。
全新进程复跑 R0/R1 后，run seed、prompt hash、完整 token IDs、文本、停止状态与
CPU/CUDA RNG pre-state 八项均为 `64 / 64` exact。
详见 详见
[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md) 与 [DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md) 与
[DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)、 [DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)、
@@ -85,7 +91,9 @@ checkpoint 又执行 embedding、27 层、final norm 与 26 个 MoE gate 的 pro
[DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md](./research/DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md) 与 [DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md](./research/DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md) 与
[DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md](./research/DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md)，以及 [DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md](./research/DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md)，以及
[DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_AUDIT.md) 与 [DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_AUDIT.md) 与
[DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md)。 [DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md)、
[DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md](./research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md) 与
[DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md)。
其余专题按进度账本持续扩建。 其余专题按进度账本持续扩建。
## 本地开发 ## 本地开发
experiments/deepseek/README.md
+82
View File
@@ -672,3 +672,85 @@ See `research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_PROTOCOL.md` and
`research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md` for the OOM `research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md` for the OOM
amendment, task evaluators, completion table, depth curves, reproduction amendment, task evaluators, completion table, depth curves, reproduction
audit, artifact hashes, and claim boundaries. audit, artifact hashes, and claim boundaries.
## Preregistered multi-seed Chat sampling
`v2_lite_chat_sampling_probe.py` keeps the same four first-ranked sources,
eight prompt conditions, 512-token cap, BF16 checkpoint, and offload contract.
It enables the pinned official sampling values and explicitly disables top-k:
```text
do_sample=true · temperature=.3 · top_p=.95 · top_k=0
4 sources × 8 SHA-256-derived seeds × 8 conditions = 256 outputs
```
The eight rows of one source remain in a fixed batch. They are batch-seed
aligned, not common-random-number pairs: Transformers samples the whole batch
with one `torch.multinomial` call.
```bash
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B experiments/deepseek/v2_lite_chat_sampling_probe.py \
--artifact-dir /path/to/deepseek-v2-lite-chat \
--reference-routing-json \
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
--greedy-json src/data/deepseek-v2-lite-chat-completion-512.json \
--human-eval /path/to/HumanEval.jsonl.gz \
--gsm8k /path/to/gsm8k/test.jsonl \
--tnews /path/to/tnews/test.json \
--tnews-archive /path/to/tnews_public.zip \
--wikitext /path/to/wikitext-validation.parquet \
--base-seeds \
19683830 1560062173 3978401375 1280933274 \
1467459869 1297359489 2722953988 3330978061 \
--max-new-tokens 512 \
--gpu-memory 28GiB \
--cpu-memory 80GiB \
--output src/data/deepseek-v2-lite-chat-sampling.json
```
The independent evaluator preserves stopping, semantic terminal state,
coverage, and correctness as separate ledgers. Exact duplicate code candidates
reuse a cache keyed by candidate, task tests, and sandbox-harness hashes:
```bash
python -B experiments/deepseek/v2_lite_chat_sampling_evaluator.py \
--sampling-json src/data/deepseek-v2-lite-chat-sampling.json \
--human-eval /path/to/HumanEval.jsonl.gz \
--gsm8k /path/to/gsm8k/test.jsonl \
--sandbox-image \
python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
--output src/data/deepseek-v2-lite-chat-sampling-eval.json
```
A fresh Python process reruns R0/R1. The reproduction comparer checks run
seeds, prompt hashes, complete token IDs, text, stopping state, and CPU/CUDA
RNG pre-state hashes:
```bash
python -B experiments/deepseek/v2_lite_chat_sampling_reproduction.py \
--formal-json src/data/deepseek-v2-lite-chat-sampling.json \
--rerun-json \
src/data/deepseek-v2-lite-chat-sampling-repro-r0r1.json \
--output \
src/data/deepseek-v2-lite-chat-sampling-reproduction.json
```
Formal results are 251/256 natural EOS and 242/256 globally unique complete
token trajectories. Strict GSM8K exact is 62/64; HumanEval official-test pass
is 63/64. These repeat one task per domain and are not benchmark estimates.
The R0/R1 fresh-process rerun reproduces all eight preregistered fields in
64/64 cells.
```text
formal 46c7edfce1409e798d6b0f06e905dd3a9d4ab6911acd8e42e2ec9d06d65345af
eval 078f486e021ffd0e4af8ad942d001a9f2429933b2fb0fb3523b7d1da5b24c4d8
rerun 72d050eace531bb38fb2592e39dc7f1b9c9003306c92c568dcd4acaaa691f6a0
compare 4d59a775943459604ea2d9976bb3be01ff7b3b08d71eb0ca22c376122ef82d15
```
See `research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md` and
`research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md` for seed derivation,
smoke gates, source-condition diversity tables, edge-set comparisons, task
failure cases, exact reproduction scope, primary sources, and non-claims.
experiments/deepseek/v2_lite_chat_sampling_evaluator.py
+560
View File
@@ -0,0 +1,560 @@
#!/usr/bin/env python3
"""Evaluate sampled DeepSeek-V2-Lite-Chat completions safely.
Stopping, task-terminal state, evaluator coverage, and correctness remain
separate ledgers. HumanEval candidates execute only in fresh networkless,
read-only Docker containers. An exact candidate+task+harness cache avoids
re-executing duplicate code while preserving one result row per sampled cell.
"""
from __future__ import annotations
import argparse
import copy
import hashlib
import json
from collections import Counter
from pathlib import Path
from statistics import mean
from typing import Any
import v2_lite_chat_completion_evaluator as completion
COMPARISONS = (
("system_eos", "s0_eos", "s1_eos"),
("system_bos", "s0_bos", "s1_bos"),
("system_x", "s0_x", "s1_x"),
("system_period", "s0_period", "s1_period"),
("bos_at_s0", "s0_eos", "s0_bos"),
("bos_at_s1", "s1_eos", "s1_bos"),
("x_at_s0", "s0_eos", "s0_x"),
("x_at_s1", "s1_eos", "s1_x"),
("period_at_s0", "s0_eos", "s0_period"),
("period_at_s1", "s1_eos", "s1_period"),
)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument("--sampling-json", type=Path, required=True)
parser.add_argument("--human-eval", type=Path, required=True)
parser.add_argument("--gsm8k", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument(
"--sandbox-image",
required=True,
help="Pinned image reference including @sha256 digest.",
)
parser.add_argument("--timeout-seconds", type=float, default=5.0)
parser.add_argument("--skip-code-execution", action="store_true")
return parser.parse_args()
def evaluate_code_cached(
*,
text: str,
hit_eos: bool,
task: dict[str, Any],
image: str,
timeout_seconds: float,
skip_execution: bool,
cache: dict[tuple[str, str, str], dict[str, Any]],
) -> tuple[dict[str, Any], bool]:
extracted = completion.extract_code(
text,
task["prompt"],
task["entry_point"],
)
harness = completion.sandbox_harness(
extracted["candidate"],
task["test"],
task["entry_point"],
)
cache_key = (
extracted["candidate_sha256"],
hashlib.sha256(task["test"].encode()).hexdigest(),
hashlib.sha256(harness.encode()).hexdigest(),
)
cache_hit = cache_key in cache
if cache_hit:
execution = copy.deepcopy(cache[cache_key])
else:
execution = {
"status": "not_run",
"return_code": None,
"runtime_ms": None,
"harness_sha256": cache_key[2],
}
if extracted["python_ast_parse"] and not skip_execution:
execution = completion.execute_code(
extracted["candidate"],
task,
image,
timeout_seconds,
)
cache[cache_key] = copy.deepcopy(execution)
passed = execution["status"] == "passed"
semantic_terminal = bool(
hit_eos
or extracted["closed_code_fence"]
or passed
)
result = {
key: value
for key, value in extracted.items()
if key != "candidate"
} | {
"semantic_terminal": semantic_terminal,
"evaluator_covered": (
extracted["python_ast_parse"]
and execution["status"] != "not_run"
),
"execution": execution,
"fixed_budget_tests_pass": passed,
"strict_complete_tests_pass": (
passed and semantic_terminal
),
"execution_cache_hit": cache_hit,
}
return result, cache_hit
def output_success(row: dict[str, Any]) -> dict[str, int | None]:
evaluation = row["task_evaluation"]
if row["domain"] == "math":
return {
"fixed_budget": int(
evaluation["fixed_budget_numeric_exact"]
),
"strict_complete": int(
evaluation["strict_complete_numeric_exact"]
),
}
if row["domain"] == "code":
return {
"fixed_budget": int(
evaluation["fixed_budget_tests_pass"]
),
"strict_complete": int(
evaluation["strict_complete_tests_pass"]
),
}
return {
"fixed_budget": None,
"strict_complete": None,
}
def summarize_group(rows: list[dict[str, Any]]) -> dict[str, Any]:
math_rows = [
row for row in rows if row["domain"] == "math"
]
code_rows = [
row for row in rows if row["domain"] == "code"
]
answers = Counter(
row["task_evaluation"]["predicted_final"]
for row in math_rows
if row["task_evaluation"]["predicted_final"] is not None
)
max_count = max(answers.values(), default=0)
modes = sorted(
answer
for answer, count in answers.items()
if count == max_count
)
gold = (
math_rows[0]["task_evaluation"]["gold_final"]
if math_rows
else None
)
return {
"outputs": len(rows),
"natural_eos": sum(row["hit_eos"] for row in rows),
"budget_truncated": sum(
row["stopped_at_max_new_tokens"] for row in rows
),
"mean_generated_tokens": (
mean(row["generated_tokens"] for row in rows)
if rows
else None
),
"completion_classes": dict(
Counter(row["completion_class"] for row in rows)
),
"math": {
"outputs": len(math_rows),
"evaluator_covered": sum(
row["task_evaluation"]["evaluator_covered"]
for row in math_rows
),
"fixed_budget_exact": sum(
row["task_evaluation"][
"fixed_budget_numeric_exact"
]
for row in math_rows
),
"strict_complete_exact": sum(
row["task_evaluation"][
"strict_complete_numeric_exact"
]
for row in math_rows
),
"answer_frequencies": dict(answers),
"modal_answers": modes,
"modal_count": max_count,
"absolute_majority_exists": (
max_count > len(math_rows) / 2
if math_rows
else None
),
"unique_absolute_majority": (
modes[0]
if (
math_rows
and len(modes) == 1
and max_count > len(math_rows) / 2
)
else None
),
"gold": gold,
"unique_absolute_majority_matches_gold": (
len(modes) == 1
and max_count > len(math_rows) / 2
and modes[0] == gold
if math_rows
else None
),
},
"code": {
"outputs": len(code_rows),
"ast_parse": sum(
row["task_evaluation"]["python_ast_parse"]
for row in code_rows
),
"executed": sum(
row["task_evaluation"]["execution"]["status"]
!= "not_run"
for row in code_rows
),
"execution_cache_hits": sum(
row["task_evaluation"]["execution_cache_hit"]
for row in code_rows
),
"tests_pass": sum(
row["task_evaluation"][
"fixed_budget_tests_pass"
]
for row in code_rows
),
"strict_complete_tests_pass": sum(
row["task_evaluation"][
"strict_complete_tests_pass"
]
for row in code_rows
),
"execution_statuses": dict(
Counter(
row["task_evaluation"]["execution"]["status"]
for row in code_rows
)
),
},
}
def source_condition_summary(
rows: list[dict[str, Any]],
) -> dict[str, Any]:
result: dict[str, dict[str, Any]] = {}
source_ids = sorted({row["source_id"] for row in rows})
conditions = tuple(
condition
for _, left, right in COMPARISONS
for condition in (left, right)
)
ordered_conditions = tuple(dict.fromkeys(conditions))
for source_id in source_ids:
result[source_id] = {}
for condition in ordered_conditions:
subset = [
row
for row in rows
if row["source_id"] == source_id
and row["condition"] == condition
]
result[source_id][condition] = summarize_group(subset)
return result
def edge_summary(rows: list[dict[str, Any]]) -> dict[str, Any]:
result: dict[str, dict[str, Any]] = {}
source_ids = sorted({row["source_id"] for row in rows})
for source_id in source_ids:
result[source_id] = {}
source_rows = [
row for row in rows if row["source_id"] == source_id
]
for name, left, right in COMPARISONS:
left_rows = [
row
for row in source_rows
if row["condition"] == left
]
right_rows = [
row
for row in source_rows
if row["condition"] == right
]
left_success = [
output_success(row) for row in left_rows
]
right_success = [
output_success(row) for row in right_rows
]
result[source_id][name] = {
"left": left,
"right": right,
"samples_per_side": len(left_rows),
"completion_classes": {
"left": dict(
Counter(
row["completion_class"]
for row in left_rows
)
),
"right": dict(
Counter(
row["completion_class"]
for row in right_rows
)
),
},
"natural_eos_difference_right_minus_left": (
sum(row["hit_eos"] for row in right_rows)
- sum(row["hit_eos"] for row in left_rows)
),
"fixed_budget_success_difference_right_minus_left": (
sum(
row["fixed_budget"]
for row in right_success
if row["fixed_budget"] is not None
)
- sum(
row["fixed_budget"]
for row in left_success
if row["fixed_budget"] is not None
)
if any(
row["fixed_budget"] is not None
for row in [*left_success, *right_success]
)
else None
),
"strict_success_difference_right_minus_left": (
sum(
row["strict_complete"]
for row in right_success
if row["strict_complete"] is not None
)
- sum(
row["strict_complete"]
for row in left_success
if row["strict_complete"] is not None
)
if any(
row["strict_complete"] is not None
for row in [*left_success, *right_success]
)
else None
),
}
return result
def main() -> None:
args = parse_args()
for path in (
args.sampling_json,
args.human_eval,
args.gsm8k,
):
if not path.is_file():
raise FileNotFoundError(path)
if "@sha256:" not in args.sandbox_image:
raise ValueError("--sandbox-image must include an immutable digest")
if args.timeout_seconds <= 0:
raise ValueError("--timeout-seconds must be positive")
sampling = json.loads(
args.sampling_json.read_text(encoding="utf-8")
)
human_eval, gsm8k = completion.load_tasks(
args.human_eval,
args.gsm8k,
)
code_cache: dict[
tuple[str, str, str],
dict[str, Any],
] = {}
rows = []
for source in sampling["sources"]:
for run in source["runs"]:
for output in run["outputs"]:
evaluation = None
cache_hit = False
if source["domain"] == "math":
evaluation = completion.evaluate_math(
output["text"],
output["hit_eos"],
gsm8k[source["id"]]["answer"],
)
elif source["domain"] == "code":
evaluation, cache_hit = evaluate_code_cached(
text=output["text"],
hit_eos=output["hit_eos"],
task=human_eval[source["id"]],
image=args.sandbox_image,
timeout_seconds=args.timeout_seconds,
skip_execution=args.skip_code_execution,
cache=code_cache,
)
rows.append(
{
"source_id": source["id"],
"domain": source["domain"],
"replicate_index": run["replicate_index"],
"replicate_label": run["replicate_label"],
"base_seed": run["base_seed"],
"run_seed": run["run_seed"],
"condition": output["condition"],
"generated_tokens": output[
"generated_tokens"
],
"hit_eos": output["hit_eos"],
"stopped_at_max_new_tokens": output[
"stopped_at_max_new_tokens"
],
"prompt_token_ids_sha256": output[
"prompt_token_ids_sha256"
],
"generated_token_ids_sha256": output[
"generated_token_ids_sha256"
],
"text_sha256": output["text_sha256"],
"task_evaluation": evaluation,
"completion_class": (
completion.completion_class(
output,
evaluation,
)
),
"code_execution_cache_hit": cache_hit,
}
)
by_condition = {
condition: summarize_group(
[
row
for row in rows
if row["condition"] == condition
]
)
for condition in sampling["seed_contract"][
"condition_row_order"
]
}
result = {
"schema_version": 1,
"protocol_id": sampling["protocol_id"],
"input": {
"sampling_path": str(args.sampling_json),
"sampling_sha256": completion.sha256_file(
args.sampling_json
),
"sampling_content_hash": sampling["content_hash"],
"human_eval_sha256": completion.sha256_file(
args.human_eval
),
"gsm8k_sha256": completion.sha256_file(args.gsm8k),
"model_revision": sampling["model"]["revision"],
"base_seeds": sampling["seed_contract"][
"executed_base_seeds"
],
"max_new_tokens": sampling["generation_contract"][
"max_new_tokens"
],
},
"sandbox": {
"image": args.sandbox_image,
"timeout_seconds": args.timeout_seconds,
"code_execution_skipped": args.skip_code_execution,
"network": "none",
"filesystem": "read-only",
"user": "65534:65534",
"capabilities": "ALL dropped",
"no_new_privileges": True,
"memory": "256m",
"memory_swap": "256m",
"pids_limit": 64,
"cpus": 0.5,
"tmpfs": "/tmp:rw,noexec,nosuid,size=16m",
"host_mounts": 0,
"cache_key": (
"candidate SHA-256 + task-test SHA-256 + "
"sandbox-harness SHA-256"
),
"unique_code_cache_entries": len(code_cache),
},
"rows": rows,
"summary": {
**summarize_group(rows),
"by_condition": by_condition,
"by_source_condition": source_condition_summary(rows),
"by_source_edge": edge_summary(rows),
},
"claim_boundary": [
"Four sources and eight seeds are not benchmark estimates.",
"Completion-conditioned metrics are selection-biased diagnostics.",
"A passing HumanEval test is functional evidence, not code-safety evidence.",
"A modal sampled math answer is not standard self-consistency.",
"Counterfactual token sequences are not official-valid chats.",
"Repeated samples from one source are not independent tasks.",
],
}
result["content_hash"] = completion.canonical_hash(rows)
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(
json.dumps(result, ensure_ascii=False, indent=2) + "\n",
encoding="utf-8",
)
payload = args.output.read_bytes()
print(
json.dumps(
{
"output": str(args.output),
"sha256": completion.sha256_bytes(payload),
"bytes": len(payload),
"outputs": len(rows),
"summary": {
key: result["summary"][key]
for key in (
"natural_eos",
"budget_truncated",
"completion_classes",
"math",
"code",
)
},
"unique_code_cache_entries": len(code_cache),
},
ensure_ascii=False,
indent=2,
)
)
if __name__ == "__main__":
main()
experiments/deepseek/v2_lite_chat_sampling_probe.py
+1084
View File
File diff suppressed because it is too large Load Diff
experiments/deepseek/v2_lite_chat_sampling_reproduction.py
+199
View File
@@ -0,0 +1,199 @@
#!/usr/bin/env python3
"""Compare a fresh sampling rerun with matching formal-grid cells."""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
from typing import Any
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument("--formal-json", type=Path, required=True)
parser.add_argument("--rerun-json", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
return parser.parse_args()
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
while chunk := handle.read(16 * 1024 * 1024):
digest.update(chunk)
return digest.hexdigest()
def canonical_hash(value: Any) -> str:
return hashlib.sha256(
json.dumps(
value,
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
).encode()
).hexdigest()
def run_index(
payload: dict[str, Any],
) -> dict[tuple[str, int, str], tuple[dict[str, Any], dict[str, Any]]]:
return {
(
source["id"],
run["base_seed"],
output["condition"],
): (run, output)
for source in payload["sources"]
for run in source["runs"]
for output in run["outputs"]
}
def main() -> None:
args = parse_args()
for path in (args.formal_json, args.rerun_json):
if not path.is_file():
raise FileNotFoundError(path)
formal = json.loads(
args.formal_json.read_text(encoding="utf-8")
)
rerun = json.loads(
args.rerun_json.read_text(encoding="utf-8")
)
if formal["protocol_id"] != rerun["protocol_id"]:
raise RuntimeError("protocol IDs differ")
if formal["model"]["revision"] != rerun["model"]["revision"]:
raise RuntimeError("model revisions differ")
if (
formal["generation_contract"]
!= rerun["generation_contract"]
):
raise RuntimeError("generation contracts differ")
formal_rows = run_index(formal)
rerun_rows = run_index(rerun)
missing = sorted(set(rerun_rows) - set(formal_rows))
if missing:
raise RuntimeError(
f"{len(missing)} rerun cells are absent from formal grid"
)
rows = []
for key in sorted(rerun_rows):
formal_run, formal_output = formal_rows[key]
rerun_run, rerun_output = rerun_rows[key]
checks = {
"run_seed_exact": (
formal_run["run_seed"] == rerun_run["run_seed"]
),
"prompt_hash_exact": (
formal_output["prompt_token_ids_sha256"]
== rerun_output["prompt_token_ids_sha256"]
),
"generated_token_ids_exact": (
formal_output["generated_token_ids"]
== rerun_output["generated_token_ids"]
),
"decoded_text_exact": (
formal_output["text"] == rerun_output["text"]
),
"eos_state_exact": (
formal_output["hit_eos"]
== rerun_output["hit_eos"]
),
"truncation_state_exact": (
formal_output["stopped_at_max_new_tokens"]
== rerun_output["stopped_at_max_new_tokens"]
),
"cpu_rng_pre_state_exact": (
formal_run["rng_state_before"]["cpu_sha256"]
== rerun_run["rng_state_before"]["cpu_sha256"]
),
"cuda_rng_pre_state_exact": (
formal_run["rng_state_before"][
"cuda_combined_sha256"
]
== rerun_run["rng_state_before"][
"cuda_combined_sha256"
]
),
}
rows.append(
{
"source_id": key[0],
"base_seed": key[1],
"condition": key[2],
**checks,
"all_preregistered_fields_exact": all(
checks.values()
),
}
)
check_names = [
key
for key in rows[0]
if key.endswith("_exact")
and key != "all_preregistered_fields_exact"
]
result = {
"schema_version": 1,
"protocol_id": formal["protocol_id"],
"formal": {
"path": str(args.formal_json),
"sha256": sha256_file(args.formal_json),
"content_hash": formal["content_hash"],
"base_seeds": formal["seed_contract"][
"executed_base_seeds"
],
},
"rerun": {
"path": str(args.rerun_json),
"sha256": sha256_file(args.rerun_json),
"content_hash": rerun["content_hash"],
"base_seeds": rerun["seed_contract"][
"executed_base_seeds"
],
},
"rows": rows,
"summary": {
"cells": len(rows),
"all_preregistered_fields_exact": sum(
row["all_preregistered_fields_exact"]
for row in rows
),
"by_field": {
name: sum(row[name] for row in rows)
for name in check_names
},
},
"claim_boundary": [
"Only the rerun seed subset is independently reproduced.",
"Exact replay is scoped to the pinned software and hardware contract.",
"Reproduction does not imply trajectories are seed-invariant.",
],
}
result["content_hash"] = canonical_hash(rows)
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(
json.dumps(result, ensure_ascii=False, indent=2) + "\n",
encoding="utf-8",
)
payload = args.output.read_bytes()
print(
json.dumps(
{
"output": str(args.output),
"sha256": hashlib.sha256(payload).hexdigest(),
"bytes": len(payload),
"summary": result["summary"],
},
ensure_ascii=False,
indent=2,
)
)
if __name__ == "__main__":
main()
package.json
+2
View File
@@ -15,6 +15,7 @@
"build:data:deepseek-role-block": "node scripts/build-deepseek-role-marker-block-compact.mjs", "build:data:deepseek-role-block": "node scripts/build-deepseek-role-marker-block-compact.mjs",
"build:data:deepseek-chat-behavior": "node scripts/build-deepseek-chat-behavior-compact.mjs", "build:data:deepseek-chat-behavior": "node scripts/build-deepseek-chat-behavior-compact.mjs",
"build:data:deepseek-chat-completion-depth": "node scripts/build-deepseek-chat-completion-depth-compact.mjs", "build:data:deepseek-chat-completion-depth": "node scripts/build-deepseek-chat-completion-depth-compact.mjs",
"build:data:deepseek-chat-sampling": "node scripts/build-deepseek-chat-sampling-compact.mjs",
"check:site": "node scripts/check-site.mjs", "check:site": "node scripts/check-site.mjs",
"check:moe-browser": "node scripts/check-moe-browser.mjs", "check:moe-browser": "node scripts/check-moe-browser.mjs",
"check:reasoning-browser": "node scripts/check-reasoning-browser.mjs", "check:reasoning-browser": "node scripts/check-reasoning-browser.mjs",
@@ -31,6 +32,7 @@
"check:evaluation-browser": "node scripts/check-evaluation-browser.mjs", "check:evaluation-browser": "node scripts/check-evaluation-browser.mjs",
"check:representation-browser": "node scripts/check-representation-browser.mjs", "check:representation-browser": "node scripts/check-representation-browser.mjs",
"check:deepseek-browser": "node scripts/check-deepseek-browser.mjs", "check:deepseek-browser": "node scripts/check-deepseek-browser.mjs",
"check:deepseek-sampling-browser": "node scripts/check-deepseek-sampling-browser.mjs",
"check:k3-browser": "node scripts/check-k3-browser.mjs" "check:k3-browser": "node scripts/check-k3-browser.mjs"
}, },
"dependencies": { "dependencies": {
research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md
+450
View File
@@ -0,0 +1,450 @@
# DeepSeek-V2-Lite-Chat 多种子采样稳健性审计
> 执行日期：2026-07-30
>
> 协议：`llm-atlas-deepseek-chat-sampling-v1`
>
> 模型：`deepseek-ai/DeepSeek-V2-Lite-Chat`
>
> revision：`85864749cd611b4353ce1decdb286193298f64c7`
>
> 预注册：`research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md`
## 0. 一句话结论
在固定官方 Chat checkpoint、固定四条 source、固定八格 prompt batch 与官方
`temperature=0.3 / top_p=0.95` 下：
```text
256 sampled outputs
├─ 251 natural EOS
├─ 5 budget truncated（全部来自 English / system off）
├─ 242 个不同的完整 token trajectory hash
├─ Math：62 / 64 strict exact
├─ Code：63 / 64 official tests pass
└─ 新进程 R0/R1：64 / 64 全合同字段 exact
```
这说明在这份执行合同下：
1. sampling 确实产生了跨 seed 轨迹分叉；
2. 同 seed 可以在新进程精确复现；
3. 一条流畅、自然结束、可解析或可执行的回答仍然可能答错；
4. 四条 source 与八个 seed 仍远不足以估计 benchmark 能力或完整生成分布。
---
## 1. 为什么在 greedy 之后还要做这一轮
Round 05 的 512-token 实验固定 `do_sample=false`。它验证的是每一步都取最高概率 token
时的单条轨迹：
```text
同一 source + 同一 checkpoint + 不同输入边界
→ greedy token 轨迹会怎样分叉
```
greedy 不能回答：
- 换 seed 后，同一条件是否只会复述同一条轨迹；
- greedy 轨迹是否会出现在有限的 nucleus samples 里；
- 两个条件的八样本集合是否有 exact 重合；
- 完成与任务结果是否会随 seed 改变。
本轮只补这个证据缺口，不回头把 4 条 source 写成能力评测。
---
## 2. 冻结合同
### 2.1 四条 source
| domain | source ID | 角色 |
|---|---|---|
| English | `wikitext2/raw-validation/0443` | 长 continuation，最容易触及 512 上限 |
| Chinese | `tnews/test/4855` | 中文分类式输入 |
| Code | `HumanEval/31` | `is_prime`，可用官方 tests 执行 |
| Math | `gsm8k/test/1069` | 可抽取数值 gold |
它们是前序固定 source 排序中每域第 1 条，不是看到结果后重选。
### 2.2 八格输入
```text
system off/on × EOS / BOS / x / period
```
固定行顺序：
```text
s0_eos, s1_eos, s0_bos, s1_bos,
s0_x, s1_x, s0_period, s1_period
```
32 / 32 个 prompt token hash 与 Round 05 greedy 运行 exact。只有 EOS 是官方聊天序列；
BOS、`x` 与句点都是单 ID 反事实，不能称为官方有效 chat。
### 2.3 sampling 参数
checkpoint 自带的 `generation_config.json` 固定：
```text
do_sample = true
temperature = 0.3
top_p = 0.95
```
本轮另外显式传入：
```text
top_k = 0
max_new_tokens = 512
use_cache = true
```
`top_k=0` 很重要：否则 Transformers 的通用默认 top-k 可能在不显眼处继续截断候选集合。
### 2.4 八个 seed
八个 base seed 全部由协议字符串做 SHA-256 派生：
| replicate | base seed |
|---:|---:|
| R0 | 19,683,830 |
| R1 | 1,560,062,173 |
| R2 | 3,978,401,375 |
| R3 | 1,280,933,274 |
| R4 | 1,467,459,869 |
| R5 | 1,297,359,489 |
| R6 | 2,722,953,988 |
| R7 | 3,330,978,061 |
每个 source 再由 base seed 与 source ID 派生独立 run seed。每次 batch 前执行
`torch.manual_seed()` 与 `torch.cuda.manual_seed_all()`，并记录 CPU/CUDA RNG 的前后
state hash。
### 2.5 batch-seed aligned，不是 common random numbers
Transformers `4.41.2` 对 batch 调一次 `torch.multinomial`。八行共享 run seed 与调用
时序，但不同 row 消费不同 RNG 子流。因此：
- 可以比较相同 replicate label 的两格；
- 只能称为 batch-seed aligned；
- 不能当作逐行共享同一随机数的 paired causal design；
- 本轮不报告 paired p-value 或置信区间。
---
## 3. 执行闸门
### 3.1 16-token smoke
```text
4 sources × R0/R1 × 8 conditions = 64 short outputs
4 sources × R0 replay × 8 conditions = 32 replay outputs
```
结果：
| gate | 结果 |
|---|---:|
| prompt hash exact | 32 / 32 |
| R0 同进程重放全合同 exact | 32 / 32 |
| R0 vs R1 可比格 | 32 |
| R0 vs R1 分叉 | 18 |
| OOM / NaN / exception | 0 |
16-token smoke 的 64 条输出全部触顶，符合技术检查的预期；它们没有进入正式统计。
### 3.2 正式网格
```text
4 sources × 8 seeds × 8 conditions = 256 outputs
```
结构审计：
| 检查 | 结果 |
|---|---:|
| sources | 4 |
| runs / source | 8 |
| outputs / source | 64 |
| outputs / condition | 32 |
| total outputs | 256 |
| prompt hash exact | 32 / 32 |
| missing / duplicate grid cells | 0 |
生成耗时 `2,251.225s`，CUDA peak allocated `30,868,238,336 bytes`。这是 CPU-offloaded
eager 机制审计延迟，不是服务吞吐。
---
## 4. stopping 与轨迹多样性
### 4.1 总账
| 指标 | 结果 |
|---|---:|
| natural EOS | 251 / 256 |
| budget truncated | 5 / 256 |
| 全局 unique trajectory hashes | 242 / 256 |
| R0/R1 同格 trajectory 不同 | 31 / 32 |
| greedy trajectory 出现在八样本集合 | 9 / 32 source×condition sets |
`242/256 unique` 只说明完整 token ID 序列的 hash 不同，不说明有 242 种语义或思路。
### 4.2 每个 condition
| condition | EOS / 32 | mean tokens | 四个 source-set 的 unique 总数 / 32 | 8/8 unique sets / 4 | greedy included / 4 |
|---|---:|---:|---:|---:|---:|
| `s0_eos` | 29 | 290.1 | 31 | 3 | 1 |
| `s1_eos` | 32 | 241.4 | 32 | 4 | 2 |
| `s0_bos` | 32 | 265.4 | 30 | 3 | 1 |
| `s1_bos` | 32 | 221.2 | 32 | 4 | 1 |
| `s0_x` | 30 | 263.6 | 32 | 4 | 0 |
| `s1_x` | 32 | 146.8 | 30 | 3 | 2 |
| `s0_period` | 32 | 259.5 | 32 | 4 | 0 |
| `s1_period` | 32 | 121.7 | 26 | 3 | 2 |
32 个 source×condition set 中：
```text
28 sets：8 unique trajectories
2 sets：6 unique trajectories
1 set ：7 unique trajectories
1 set ：2 unique trajectories
```
只有 `HumanEval/31 × s1_period` 收缩到 2 条完整轨迹；它不是“低创造力”的证明，只是该
source/condition/参数下八次抽样的 exact-sequence 重复。
### 4.3 五个截断
五格全部属于 `wikitext2/raw-validation/0443`：
| replicate | condition | tokens |
|---|---|---:|
| R0 | `s0_x` | 512 |
| R1 | `s0_eos` | 512 |
| R4 | `s0_eos` | 512 |
| R5 | `s0_x` | 512 |
| R6 | `s0_eos` | 512 |
所以“全局 251/256 EOS”不能简化成每条 source 都同样容易完成。
---
## 5. 十条 edge 的集合比较
每个 source×edge 同时计算：
1. 8 个相同 replicate label 对的 token similarity；
2. 左集合每条到右集合的最高 similarity；
3. 右集合每条到左集合的最高 similarity；
4. 两个方向 16 个最近邻值的平均；
5. 完整 trajectory hash-set 的 intersection / union。
四条 source 的描述性均值：
| edge | aligned similarity | symmetric nearest similarity | exact intersection / union |
|---|---:|---:|---:|
| system · EOS | .366 | .492 | 1 / 62 |
| system · BOS | .314 | .491 | 0 / 62 |
| system · x | .275 | .392 | 0 / 62 |
| system · period | .253 | .355 | 0 / 58 |
| BOS − EOS · S0 | .444 | .589 | 0 / 61 |
| BOS − EOS · S1 | .430 | .589 | 0 / 64 |
| x − EOS · S0 | .332 | .470 | 0 / 63 |
| x − EOS · S1 | .278 | .395 | 0 / 62 |
| period − EOS · S0 | .404 | .546 | 1 / 62 |
| period − EOS · S1 | .261 | .375 | 0 / 58 |
40 个 source-level edge set 中只有两个出现跨侧 exact trajectory 重合：
- `gsm8k/test/1069 × system_eos`；
- `gsm8k/test/1069 × period_at_s0`。
nearest-neighbor 总是高于相同 label 对并不奇怪：它从 8 条右侧样本里主动选择最接近的一条。
这不是无偏分布距离，也不能据此给 boundary effect 排名。
---
## 6. 任务账：完成、可评与正确分开
### 6.1 GSM8K
| 指标 | 结果 |
|---|---:|
| sampled outputs | 64 |
| numeric evaluator covered | 64 |
| strict-complete exact | 62 |
| answer `300` | 62 |
| answer `100` | 2 |
| unique absolute majority | `300` |
| gold | `300` |
两条失败分别位于：
- R4 / `s1_x`；
- R7 / `s1_eos`。
两条都不是抽取器误判。回答明确写出 boxed `100`，错误推理是：
```text
1/4 trucks failed
→ 错写成只有 1/4 trucks delivered
→ 5 trucks × 20 tons = 100
```
正确逻辑应保留 `3/4 × 20 = 15` 辆车，得到 `300`。这说明 natural EOS、明确 final
marker 与可抽取数值都不能保证 reasoning 正确。
这里的多数答案只是 1 条 GSM8K source 上的八条件×八 seed 描述，不是标准
self-consistency benchmark。
### 6.2 HumanEval
| 指标 | 结果 |
|---|---:|
| sampled outputs | 64 |
| Python AST parse | 64 |
| sandbox evaluated | 64 |
| official tests pass | 63 |
| assertion failure | 1 |
| unique candidate execution keys | 24 |
| exact duplicate cache hits | 40 |
唯一失败位于 R5 / `s0_period`。候选代码：
- 正确排除了 `n < 2`；
- 正确特判 `n == 2`；
- 只检查奇数除数；
- **没有先排除大于 2 的偶数**。
因此代码流畅、code fence 闭合、AST 合法、可以执行，却对偶数输入返回错误。这个案例直接
展示了四张账：
```text
NATURAL EOS
→ TASK TERMINAL
→ EVALUATOR COVERED
→ ASSERTION FAILED
```
每个唯一 candidate 的执行环境：
```text
python:3.11-alpine
@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4
network none · read-only filesystem · no host mounts
user 65534:65534 · cap-drop ALL · no-new-privileges
256MiB memory/swap · pids 64 · cpus .5 · timeout 5s
```
cache 只复用完全相同的 candidate hash + task-test hash + harness hash；每格评测行仍保留。
---
## 7. 新进程复现
正式运行结束后重新启动 Python、重新加载 checkpoint，只执行 R0/R1：
```text
4 sources × 2 seeds × 8 conditions = 64 outputs
```
逐格结果：
| field | exact |
|---|---:|
| run seed | 64 / 64 |
| prompt hash | 64 / 64 |
| complete generated token IDs | 64 / 64 |
| decoded text | 64 / 64 |
| EOS state | 64 / 64 |
| truncation state | 64 / 64 |
| CPU RNG pre-state hash | 64 / 64 |
| CUDA RNG pre-state hash | 64 / 64 |
| all preregistered fields | **64 / 64** |
这只能写成“R0/R1 的 64 条轨迹在固定环境中独立复现”。R2–R7 没有新进程复跑，不能写
256 / 256。
---
## 8. artifact hash 链
| artifact | bytes | SHA-256 |
|---|---:|---|
| formal raw | 1,852,892 | `46c7edfce1409e798d6b0f06e905dd3a9d4ab6911acd8e42e2ec9d06d65345af` |
| independent eval | 316,800 | `078f486e021ffd0e4af8ad942d001a9f2429933b2fb0fb3523b7d1da5b24c4d8` |
| R0/R1 rerun raw | 537,669 | `72d050eace531bb38fb2592e39dc7f1b9c9003306c92c568dcd4acaaa691f6a0` |
| reproduction comparison | 29,938 | `4d59a775943459604ea2d9976bb3be01ff7b3b08d71eb0ca22c376122ef82d15` |
| website compact | 371,495 | `c014dc0ed6c27b19a61e7abb9b84a077b414e8472d40cc69d025693c25e1372e` |
compact builder 在写前端 JSON 前强制验证：
1. evaluator 声明的 sampling input SHA 与 formal raw 文件一致；
2. reproduction 声明的 formal/rerun SHA 与两份 raw 一致；
3. reproduction `all_preregistered_fields_exact == cells`。
任一失败都会中止构建。
---
## 9. 依赖与放置
| object | value |
|---|---|
| PyTorch | `2.11.0+cu128` |
| Transformers | `4.41.2` |
| dtype | BF16 |
| local GPU | RTX 5090 · 32,607 MiB |
| CUDA placement | embedding + layers 0–23 |
| CPU offload | layers 24–26 + final norm + LM head |
| max memory | GPU 28 GiB / CPU 80 GiB |
| allocator | `expandable_segments:True` |
| formal generation time | 2,251.225 s |
| peak CUDA allocated | 30,868,238,336 bytes |
官方模型卡写出的 40GB 单 GPU BF16 边界高于本机容量，所以这不是 single-GPU BF16
execution。
---
## 10. 一手来源
- [DeepSeek-V2-Lite-Chat model card](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat)
- [Pinned official generation_config.json](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat/blob/85864749cd611b4353ce1decdb286193298f64c7/generation_config.json)
- [Transformers 4.41.2 generation utils](https://github.com/huggingface/transformers/blob/v4.41.2/src/transformers/generation/utils.py)
- [The Curious Case of Neural Text Degeneration / nucleus sampling](https://arxiv.org/abs/1904.09751)
- [Self-Consistency Improves Chain of Thought Reasoning](https://arxiv.org/abs/2203.11171)
- [HumanEval repository](https://github.com/openai/human-eval)
- [GSM8K paper](https://arxiv.org/abs/2110.14168)
---
## 11. 可以说与不能说
可以说：
- 在固定四条 source 与官方 sampling 参数下，R0/R1 有 31/32 同格轨迹分叉；
- 256 条样本中 251 条自然 EOS，242 条完整 token trajectory hash 不同；
- 这 1 条 GSM8K 的 64 个样本中 62 个 strict exact；
- 这 1 条 HumanEval 的 64 个样本中 63 个通过官方 tests；
- 固定环境下 R0/R1 新进程复跑 64/64 全合同 exact。
不能说：
- 八个 seed 已恢复完整生成分布；
- 242 个 hash 表示 242 种语义；
- 4 条 source 是 benchmark；
- 62/64 与 63/64 可以横比标准 accuracy 或 pass@k；
- majority answer 是标准 self-consistency；
- batch-seed aligned 是 common random numbers；
- nearest-neighbor 是无偏分布距离；
- 某个 condition 更“有创造力”；
- BOS、`x`、句点是官方有效聊天格式；
- output 差异由某层 hidden state 或 router 因果中介；
- fixed-version exact replay 能跨 PyTorch、CUDA 或硬件保证。
research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md
+352
View File
@@ -0,0 +1,352 @@
# DeepSeek-V2-Lite-Chat 多种子采样稳健性协议
> 状态：已执行；正式结果、独立评测与新进程 R0/R1 复跑均通过
>
> 注册日期：2026-07-30
>
> 模型：`deepseek-ai/DeepSeek-V2-Lite-Chat`
>
> revision：`85864749cd611b4353ce1decdb286193298f64c7`
>
> 前序实验：512-token greedy completion 与全 27 层 trace
>
> 结果审计：`research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md`
## 0. 这一轮只补哪一个证据缺口
Round 04–05 已经证明：
```text
固定 greedy 解码下：
同一 source 的 system / 历史边界变化会让输出 token 轨迹分叉；
512-token 预算下有 121 / 128 格自然遇到 EOS；
子集新进程复跑可以逐 token exact。
```
但 greedy 只取每一步概率最大的一个 token。它不能回答：
```text
换一组随机数以后，某个条件是否仍有相似的完成率与任务结果？
greedy 输出是采样分布中的常见轨迹，还是一条很窄的路径？
两个条件的样本集合彼此靠近，还是只有某一对 greedy 输出不同？
```
本轮因此启用 checkpoint 随附的 sampling config，固定多个 seed。它不是新的能力
benchmark，也不尝试用 8 个样本完整估计语言分布。
---
## 1. 冻结不变的对象
继承 Round 05：
- 官方 SFT Chat checkpoint、12 个文件及 SHA-256；
- BF16、官方 eager Python 实现、Transformers `4.41.2`；
- 28 GiB static placement 与 `expandable_segments:True`；
- CUDA resident：embedding + layers 0–23；
- CPU offload residency：layers 24–26 + final norm + `lm_head`；
- 同一 source 的八格进入同一个左填充 batch；
- `system off/on × EOS/BOS/x/period`；
- EOS 是官方序列；BOS、`x`、句点各是单 ID 反事实；
- `PAD=EOS`、padding mask 为 0、`use_cache=true`；
- 公开数据、source ID、source text hash 与 prompt token hash。
本轮只取前序固定次序中每域第 1 条：
| 域 | source |
|---|---|
| English | `wikitext2/raw-validation/0443` |
| Chinese | `tnews/test/4855` |
| Code | `HumanEval/31` |
| Math | `gsm8k/test/1069` |
这是 4 条 source 的机制显微镜，不是 4 个领域的总体估计。
---
## 2. 解码配置来自哪里
固定 revision 的官方 `generation_config.json` 写明：
```json
{
"do_sample": true,
"temperature": 0.3,
"top_p": 0.95,
"bos_token_id": 100000,
"eos_token_id": 100001,
"transformers_version": "4.39.3"
}
```
本轮在已固定的 Transformers `4.41.2` 执行这组参数：
```text
do_sample = true
temperature = 0.3
top_p = 0.95
top_k = 0（显式关闭默认 top-k）
max_new_tokens = 512
use_cache = true
```
Holtzman et al. 2019 的 nucleus sampling 是这里 `top_p` 的方法来源；它动态保留累计概率
达到阈值的候选集合。Wang et al. 2022 的 self-consistency 说明多条推理样本可以聚合答案，
但本轮只在 1 条 GSM8K source 上展示多数答案，绝不把它称为标准 self-consistency
benchmark。
官方文件记录的是参数默认值，不保证不同 Transformers 版本、kernel、batch 顺序与 seed
产生相同随机轨迹；因此依赖栈和 batch 合同必须进入结果。
---
## 3. seed 如何在结果之前冻结
8 个 base seeds 不是手选“看起来分散”的整数。它们由以下字符串逐个做 SHA-256，并把
前 4 bytes 按 big-endian 解释为无符号整数：
```text
llm-atlas-deepseek-chat-sampling-v1/seed/{index}
```
固定结果：
| replicate | base seed |
|---:|---:|
| R0 | 19,683,830 |
| R1 | 1,560,062,173 |
| R2 | 3,978,401,375 |
| R3 | 1,280,933,274 |
| R4 | 1,467,459,869 |
| R5 | 1,297,359,489 |
| R6 | 2,722,953,988 |
| R7 | 3,330,978,061 |
为避免不同 source 重复使用同一 CUDA random stream，实际 run seed 定义为：
```text
SHA256(
"llm-atlas-deepseek-chat-sampling-v1/run\0"
+ decimal(base_seed)
+ "\0"
+ source_id
)[0:8] interpreted as unsigned big-endian
modulo (2^63 - 1)
```
每次 source×replicate batch 前：
```python
torch.manual_seed(run_seed)
torch.cuda.manual_seed_all(run_seed)
```
并记录 CPU / CUDA RNG state 的执行前后 SHA-256。
---
## 4. 很重要：八格共享“批次标签”，不共享同一个随机数
Transformers `4.41.2` 的 `_sample()` 对整个 batch 调用：
```python
torch.multinomial(probs, num_samples=1)
```
它不接受逐行 `Generator`。所以同一个 source×replicate 的八格：
- 在同一时间步、同一 batch 中执行；
- 共享同一个 run seed 与固定行顺序；
- 但每一行消费不同的 RNG 子流；
- 不是 common-random-number paired experiment。
因此“R3 的左格 vs R3 的右格”只能称为 **batch-seed aligned**，不能用普通 paired
bootstrap 或 paired t-test 冒充同随机数因果对照。本轮不报告 edge 的显著性 p 值或
置信区间。
固定 condition 行顺序：
```text
s0_eos, s1_eos, s0_bos, s1_bos,
s0_x, s1_x, s0_period, s1_period
```
改变行顺序会改变随机轨迹，必须判为不同协议。
---
## 5. 网格与执行闸门
### 5.1 技术 smoke
```text
4 sources × 2 base seeds (R0/R1) × 8 conditions × 16 new tokens
= 64 short outputs
另将 R0 在同一进程内重放一次：
4 sources × 1 replay seed (R0) × 8 conditions × 16 new tokens
= 32 replay outputs
```
技术 smoke 只验证：
1. prompt hashes 与 Round 05 的 32 格 exact；
2. 官方 temperature / top-p 实际进入 `generate()`；
3. `top_k=0`，没有隐式默认 top-k；
4. 64 个正式 smoke 输出正常，无 OOM / NaN / runtime exception；
5. R0 在同进程按相同 seed 重跑时 32 / 32 token exact；
6. R0 与 R1 至少有一个同 source、同 condition 的 cell 分叉；若 32 / 32 仍相同，只能继续调查，不能声称 sampling
已生效。
smoke 的 16-token 输出不进入正式统计。
### 5.2 正式网格
```text
4 sources
× 8 base seeds
× 8 conditions
× 512 new-token cap
= 256 sampled outputs
```
source 顺序与 replicate 顺序固定，不根据中间结果提前停止。某一格遇到 EOS 后由
Transformers 填 PAD；不能单独给未完成格追加预算。
### 5.3 独立复跑
正式完成后新启动 Python 进程、重新加载模型：
```text
4 sources × R0/R1 × 8 conditions
= 64 outputs
```
必须逐格比较：
- run seed；
- prompt token hash；
- 完整 generated token IDs；
- decoded text；
- EOS / truncation state；
- CPU / CUDA RNG pre-state hash。
只能写“64 / 64 sampled trajectories independently reproduced”，不能写 256 / 256。
---
## 6. completion 与任务评测仍沿用四张账
每条 sampled output 继续区分：
1. stopping：natural EOS / budget truncated / other stop；
2. task terminal：明确 final marker 或闭合 code fence；
3. evaluator coverage：数值可抽取或代码 AST + 沙箱可执行；
4. correctness：GSM8K gold exact / HumanEval official tests pass。
HumanEval 每个 candidate 进入全新 pinned 容器，沿用 Round 05：
```text
network none · read-only filesystem · user 65534:65534
cap-drop ALL · no-new-privileges · no host mounts
256 MiB memory/swap · pids 64 · cpus .5 · timeout 5s
```
同一代码文本若在多个 seed 重复，评测器仍保存每格结果，但可以按 candidate SHA-256
缓存一次沙箱执行；缓存必须只复用完全相同的 candidate + task + harness。
---
## 7. 预注册统计
### 7.1 每个 source×condition 的 8-sample 集合
- natural EOS / 8；
- budget truncated / 8；
- unique generated-token hashes / 8；
- greedy trajectory 是否出现在 8 个 sampled hashes 中；
- 28 对样本两两 token similarity 的 mean / min / max；
- 生成长度 mean / min / max；
- Math：8 个抽取答案、答案频次、majority final 与 gold；
- Code：AST / executed / official tests pass 频次。
“8/8 unique”表示完整 token 序列不重复，不表示语义有八种。
### 7.2 每条 edge 的两个样本集合
十条 edge 沿用前序定义。每个 source 报告：
- 同 replicate label 的 8 对 token similarity；
- 左集合每条样本到右集合的最高相似度，再与反方向平均：
`symmetric mean nearest-neighbor similarity`；
- 两边完整序列 hash-set intersection / union；
- completion class 计数差；
- Math / Code pass 频次差。
nearest-neighbor similarity 是 8-sample 描述量，不是分布距离的无偏估计。
### 7.3 汇总层
只做：
- 四条 source 合计的分子 / 分母；
- 按域逐 source 展示；
- 按 condition / edge 的描述性均值。
不做：
- 把 256 当作 256 个独立 benchmark tasks；
- 忽略同一 source 的重复采样聚类；
- 用 seed 当 source bootstrap unit；
- 根据 1 条 Math / Code source 给置信区间；
- best-of-8 与标准 pass@k 横比。
---
## 8. reproduction 与随机性要同时成立
采样实验的“可复现”不是要求所有 seed 输出相同，而是：
```text
相同 checkpoint + 输入 + 软件 + 行顺序 + seed
→ 同一条 sampled token trajectory
不同 run seed
→ 允许、并预期至少一部分 trajectory 分叉
```
正式结果必须同时报告：
- same-seed exact reproduction；
- across-seed diversity。
只报告前者会把采样伪装成 greedy；只报告后者又无法判断差异来自随机数还是环境漂移。
---
## 9. 仍然禁止的结论
- 8 seeds 已恢复完整生成分布；
- 某个边界“更有创造力”或“更稳定”；
- BOS / `x` / 句点是官方有效聊天条件；
- 1 条 GSM8K / HumanEval 代表任务能力；
- sampled majority 等于标准 self-consistency；
- unique sequence count 等于语义多样性；
- batch-seed aligned 等于 common-random-number paired；
- greedy 没出现在 8 samples 中就说明 greedy 不可信；
- sampling 输出差异由某层 router 或 hidden state 中介；
- CPU-offloaded eager 延迟代表服务吞吐；
- exact seed replay 可以跨 PyTorch / Transformers / CUDA 版本保证。
---
## 10. 失败与修订规则
- 技术 smoke 前可以修正代码错误，但正式 seed/grid/metrics 不随输出调整；
- OOM 时先降低 static GPU placement，保留八格 batch；任何改动都要用同 seed smoke 做
完整 token exact 跨放置闸门；
- 若必须拆八格 batch，正式协议作废并重新注册，因为 RNG 行消费与数值 batch content 都变了；
- 正式运行在写完整 JSON 前失败，不保留部分 seed 做结果；
- evaluator bug 可以修复并重跑 evaluator，但不能重选 sampled outputs；
- same-seed 复跑不 exact 时，结果只作失败诊断，不发布 sampling robustness 结论。
scripts/build-deepseek-chat-sampling-compact.mjs
+334
View File
@@ -0,0 +1,334 @@
import { createHash } from "node:crypto";
import { readFileSync, statSync, writeFileSync } from "node:fs";
import { resolve } from "node:path";
const root = resolve(import.meta.dirname, "..");
const paths = {
sampling: resolve(
root,
"src/data/deepseek-v2-lite-chat-sampling.json",
),
evaluation: resolve(
root,
"src/data/deepseek-v2-lite-chat-sampling-eval.json",
),
rerun: resolve(
root,
"src/data/deepseek-v2-lite-chat-sampling-repro-r0r1.json",
),
reproduction: resolve(
root,
"src/data/deepseek-v2-lite-chat-sampling-reproduction.json",
),
output: resolve(
root,
"src/data/deepseek-v2-lite-chat-sampling-compact.json",
),
};
const readJson = (path) => JSON.parse(readFileSync(path, "utf8"));
const sha256 = (path) => createHash("sha256")
.update(readFileSync(path))
.digest("hex");
const artifact = (path) => ({
bytes: statSync(path).size,
sha256: sha256(path),
});
const mean = (values) => (
values.length
? values.reduce((total, value) => total + value, 0) / values.length
: null
);
const sampling = readJson(paths.sampling);
const evaluation = readJson(paths.evaluation);
const reproduction = readJson(paths.reproduction);
const samplingArtifact = artifact(paths.sampling);
const evaluationArtifact = artifact(paths.evaluation);
const rerunArtifact = artifact(paths.rerun);
if (evaluation.input.sampling_sha256 !== samplingArtifact.sha256) {
throw new Error("sampling evaluator input hash contract failed");
}
if (reproduction.formal.sha256 !== samplingArtifact.sha256) {
throw new Error("reproduction formal hash contract failed");
}
if (reproduction.rerun.sha256 !== rerunArtifact.sha256) {
throw new Error("reproduction rerun hash contract failed");
}
if (
reproduction.summary.all_preregistered_fields_exact
!== reproduction.summary.cells
) {
throw new Error("sampling reproduction is not exact");
}
const conditions = sampling.seed_contract.condition_row_order;
const edges = Object.keys(
sampling.summary.by_source_edge[sampling.sources[0].id],
);
const edgeLabels = {
system_eos: "System on − off · EOS",
system_bos: "System on − off · BOS",
system_x: "System on − off · x",
system_period: "System on − off · 句点",
bos_at_s0: "BOS − EOS · system off",
bos_at_s1: "BOS − EOS · system on",
x_at_s0: "x − EOS · system off",
x_at_s1: "x − EOS · system on",
period_at_s0: "句点 − EOS · system off",
period_at_s1: "句点 − EOS · system on",
};
const evalKey = (sourceId, baseSeed, condition) => (
`${sourceId}\0${baseSeed}\0${condition}`
);
const evaluationByKey = new Map(
evaluation.rows.map((row) => [
evalKey(row.source_id, row.base_seed, row.condition),
row,
]),
);
const sourceRows = sampling.sources.map((source) => {
const conditionRows = Object.fromEntries(conditions.map((condition) => {
const rawSummary = sampling.summary.by_source_condition[
source.id
][condition];
const evalSummary = evaluation.summary.by_source_condition[
source.id
][condition];
const samples = source.runs.map((run) => {
const output = run.outputs.find(
(candidate) => candidate.condition === condition,
);
const assessed = evaluationByKey.get(
evalKey(source.id, run.base_seed, condition),
);
if (!output || !assessed) {
throw new Error(
`sample/evaluation row missing: ${source.id}/${run.base_seed}/${condition}`,
);
}
return {
replicate: run.replicate_label,
baseSeed: run.base_seed,
runSeed: run.run_seed,
generatedTokens: output.generated_tokens,
hitEos: output.hit_eos,
truncated: output.stopped_at_max_new_tokens,
trajectoryHash: output.generated_token_ids_sha256,
textHash: output.text_sha256,
preview: output.text.replace(/\s+/g, " ").trim().slice(0, 220),
completionClass: assessed.completion_class,
math: assessed.task_evaluation && source.domain === "math"
? {
predicted: assessed.task_evaluation.predicted_final,
gold: assessed.task_evaluation.gold_final,
exact: assessed.task_evaluation.fixed_budget_numeric_exact,
method: assessed.task_evaluation.extraction_method,
}
: null,
code: assessed.task_evaluation && source.domain === "code"
? {
ast: assessed.task_evaluation.python_ast_parse,
status: assessed.task_evaluation.execution.status,
passed: assessed.task_evaluation.fixed_budget_tests_pass,
cacheHit: assessed.task_evaluation.execution_cache_hit,
}
: null,
};
});
return [condition, {
samples: rawSummary.samples,
naturalEos: rawSummary.natural_eos,
truncated: rawSummary.budget_truncated,
uniqueTrajectories: rawSummary.unique_generated_token_hashes,
greedyInSamples: rawSummary.greedy_full_trajectory_in_sample_set,
pairwiseSimilarity: rawSummary.pairwise_token_similarity,
generatedTokens: rawSummary.generated_tokens,
evaluation: evalSummary,
trajectories: samples,
}];
}));
return {
id: source.id,
domain: source.domain,
label: source.label,
sourceCharacters: source.source_characters,
sourceTokens: source.source_tokens,
conditions: conditionRows,
edges: Object.fromEntries(edges.map((edge) => [
edge,
{
...sampling.summary.by_source_edge[source.id][edge],
evaluation: evaluation.summary.by_source_edge[source.id][edge],
},
])),
};
});
const conditionSummary = Object.fromEntries(conditions.map((condition) => {
const sourceConditions = sourceRows.map(
(source) => source.conditions[condition],
);
const assessed = evaluation.summary.by_condition[condition];
return [condition, {
outputs: assessed.outputs,
naturalEos: assessed.natural_eos,
truncated: assessed.budget_truncated,
meanGeneratedTokens: assessed.mean_generated_tokens,
uniqueTrajectoriesAcrossSourceSets: sourceConditions.reduce(
(total, row) => total + row.uniqueTrajectories,
0,
),
fullEightWayDiversitySets: sourceConditions.filter(
(row) => row.uniqueTrajectories === 8,
).length,
greedyIncludedSourceSets: sourceConditions.filter(
(row) => row.greedyInSamples,
).length,
math: assessed.math,
code: assessed.code,
}];
}));
const edgeSummary = Object.fromEntries(edges.map((edge) => {
const rows = sourceRows.map((source) => source.edges[edge]);
return [edge, {
label: edgeLabels[edge],
sources: rows.length,
meanAlignedSimilarity: mean(rows.map(
(row) => row.batch_seed_aligned_similarity.mean,
)),
meanSymmetricNearestSimilarity: mean(rows.map(
(row) => row.symmetric_mean_nearest_neighbor_similarity,
)),
exactHashIntersections: rows.reduce(
(total, row) => total + row.generated_hash_set_intersection,
0,
),
exactHashUnion: rows.reduce(
(total, row) => total + row.generated_hash_set_union,
0,
),
naturalEosDeltaRightMinusLeft: rows.reduce(
(total, row) => (
total + row.natural_eos_count_difference_right_minus_left
),
0,
),
}];
}));
const result = {
schemaVersion: 1,
generatedAt: new Date().toISOString(),
contract: {
protocolId: sampling.protocol_id,
model: sampling.model.repo,
revision: sampling.model.revision,
checkpointIdentity: sampling.model.checkpoint_identity,
conditions,
conditionFactors: sampling.generation_contract.conditions,
edges,
edgeLabels,
sources: sampling.sources.length,
replicates: sampling.seed_contract.executed_base_seeds.length,
baseSeeds: sampling.seed_contract.executed_base_seeds,
decode: {
doSample: sampling.generation_contract.do_sample,
temperature: sampling.generation_contract.temperature,
topP: sampling.generation_contract.top_p,
topK: sampling.generation_contract.top_k,
maxNewTokens: sampling.generation_contract.max_new_tokens,
},
batchSeedAlignedNotCommonRandomNumbers: (
sampling.seed_contract.batch_seed_aligned_not_common_random_numbers
),
counterfactualBoundary: (
sampling.generation_contract.counterfactual_boundary
),
},
headline: {
outputs: sampling.summary.outputs,
naturalEos: sampling.summary.natural_eos,
truncated: sampling.summary.budget_truncated,
uniqueTrajectoryHashes: (
sampling.summary.unique_generated_token_hashes
),
firstTwoSeedComparableCells: (
sampling.summary.first_two_seeds.comparable_cells
),
firstTwoSeedDifferentTrajectories: (
sampling.summary.first_two_seeds.different_trajectories
),
promptHashExact: (
sampling.source_contract.prompt_hash_audit.exact
),
promptHashCells: (
sampling.source_contract.prompt_hash_audit.cells
),
greedyIncludedSourceConditionSets: sourceRows.reduce(
(total, source) => total + conditions.filter(
(condition) => source.conditions[condition].greedyInSamples,
).length,
0,
),
sourceConditionSets: sourceRows.length * conditions.length,
mathExact: evaluation.summary.math.fixed_budget_exact,
mathOutputs: evaluation.summary.math.outputs,
mathMajority: evaluation.summary.math.unique_absolute_majority,
mathGold: evaluation.summary.math.gold,
codePassed: evaluation.summary.code.tests_pass,
codeOutputs: evaluation.summary.code.outputs,
codeUniqueExecutionKeys: (
evaluation.sandbox.unique_code_cache_entries
),
reproducedCells: (
reproduction.summary.all_preregistered_fields_exact
),
reproductionCells: reproduction.summary.cells,
},
conditions: conditionSummary,
edges: edgeSummary,
sources: sourceRows,
reproduction: reproduction.summary,
artifacts: {
sampling: samplingArtifact,
evaluation: evaluationArtifact,
rerun: rerunArtifact,
reproduction: artifact(paths.reproduction),
},
execution: {
generationSeconds: sampling.sources.reduce(
(total, source) => total + source.runs.reduce(
(sourceTotal, run) => sourceTotal + run.generation_seconds,
0,
),
0,
),
peakCudaMemoryAllocatedBytes: (
sampling.execution.peak_cuda_memory_allocated_bytes
),
deviceMap: sampling.execution.device_map,
transformers: sampling.execution.transformers,
torch: sampling.execution.torch,
sandbox: evaluation.sandbox,
},
claimBoundary: [
...sampling.claim_boundary,
...evaluation.claim_boundary,
...reproduction.claim_boundary,
],
};
writeFileSync(
paths.output,
`${JSON.stringify(result, null, 2)}\n`,
);
console.log(JSON.stringify({
output: paths.output,
...artifact(paths.output),
headline: result.headline,
}, null, 2));
scripts/check-deepseek-browser.mjs
+2 -2
View File
@@ -1210,14 +1210,14 @@ console.log(JSON.stringify(report, null, 2));
const numeric = (text) => Number.parseFloat(text.replaceAll(",", "")); const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
const failures = []; const failures = [];
if (!overview.title.includes("为什么转向")) failures.push("专题标题异常"); if (!overview.title.includes("为什么转向")) failures.push("专题标题异常");
if (overview.sections !== 28 || overview.tocLinks !== 28) failures.push("二十七个编号专题加阅读链的目录结构异常"); if (overview.sections !== 29 || overview.tocLinks !== 29) failures.push("二十八个编号专题加阅读链的目录结构异常");
if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常"); if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常");
if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常"); if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常");
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常"); if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
if (overview.artifactTabs !== 13 || overview.artifactPanels !== 13 || overview.artifactLayers !== 27) failures.push("真实权重十三联实验结构异常"); if (overview.artifactTabs !== 13 || overview.artifactPanels !== 13 || overview.artifactLayers !== 27) failures.push("真实权重十三联实验结构异常");
if (overview.behaviorTabs !== 4 || overview.behaviorPanels !== 4 || overview.behaviorSources !== 16 || overview.behaviorEdges !== 10) failures.push("Chat 行为实验结构异常"); if (overview.behaviorTabs !== 4 || overview.behaviorPanels !== 4 || overview.behaviorSources !== 16 || overview.behaviorEdges !== 10) failures.push("Chat 行为实验结构异常");
if (overview.completionDepthTabs !== 4 || overview.completionDepthPanels !== 4 || overview.hiddenStages !== 29 || overview.routerLayers !== 26) failures.push("Chat 完成度与全深度实验结构异常"); if (overview.completionDepthTabs !== 4 || overview.completionDepthPanels !== 4 || overview.hiddenStages !== 29 || overview.routerLayers !== 26) failures.push("Chat 完成度与全深度实验结构异常");
if (overview.heroLabs !== "19 个可操作实验") failures.push("DeepSeek 实验总数账异常"); if (overview.heroLabs !== "20 个可操作实验") failures.push("DeepSeek 实验总数账异常");
if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek"); if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek");
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出"); if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常"); if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常");
scripts/check-deepseek-sampling-browser.mjs
+278
View File
@@ -0,0 +1,278 @@
import { writeFileSync } from "node:fs";
const cdpPort = process.env.CDP_PORT ?? "9230";
const baseUrl = process.env.SITE_URL ?? "http://127.0.0.1:4327";
const pages = await fetch(
`http://127.0.0.1:${cdpPort}/json/list`,
).then((response) => response.json());
const page = pages.find((entry) => entry.type === "page");
if (!page) throw new Error(`CDP ${cdpPort} 没有可用页面`);
const socket = new WebSocket(page.webSocketDebuggerUrl);
await new Promise((resolve, reject) => {
socket.addEventListener("open", resolve, { once: true });
socket.addEventListener("error", reject, { once: true });
});
let nextId = 0;
const pending = new Map();
const exceptions = [];
socket.addEventListener("message", (event) => {
const message = JSON.parse(event.data);
if (message.id && pending.has(message.id)) {
const { resolve, reject } = pending.get(message.id);
pending.delete(message.id);
if (message.error) reject(new Error(message.error.message));
else resolve(message.result);
}
if (message.method === "Runtime.exceptionThrown") {
exceptions.push(
message.params.exceptionDetails.exception?.description
?? message.params.exceptionDetails.text,
);
}
});
const command = (method, params = {}) => new Promise((resolve, reject) => {
const id = ++nextId;
pending.set(id, { resolve, reject });
socket.send(JSON.stringify({ id, method, params }));
});
const pause = (milliseconds) => new Promise(
(resolve) => setTimeout(resolve, milliseconds),
);
const evaluate = async (expression) => {
const result = await command("Runtime.evaluate", {
expression,
returnByValue: true,
awaitPromise: true,
});
if (result.exceptionDetails) {
throw new Error(
result.exceptionDetails.exception?.description
?? result.exceptionDetails.text,
);
}
return result.result.value;
};
const navigate = async (path) => {
await command("Page.navigate", { url: `${baseUrl}${path}` });
for (let attempt = 0; attempt < 80; attempt += 1) {
await pause(100);
if (await evaluate("document.readyState === 'complete'")) return;
}
throw new Error(`${path} 加载超时`);
};
const screenshot = async (path) => {
const result = await command("Page.captureScreenshot", {
format: "png",
captureBeyondViewport: false,
});
writeFileSync(path, Buffer.from(result.data, "base64"));
};
const assert = (condition, message) => {
if (!condition) throw new Error(message);
};
await command("Page.enable");
await command("Runtime.enable");
await command("Emulation.setDeviceMetricsOverride", {
width: 1440,
height: 1100,
deviceScaleFactor: 1,
mobile: false,
});
await navigate("/deepseek/");
const overview = await evaluate(`(() => {
const root = document.querySelector("[data-sampling-lab]");
document.documentElement.style.scrollBehavior = "auto";
window.scrollTo(0, root.getBoundingClientRect().top + window.scrollY);
return {
tabs: root.querySelectorAll("[data-sp-tab]").length,
panels: root.querySelectorAll("[data-sp-panel]").length,
seedBars: root.querySelectorAll("[data-sp-seeds] button").length,
conditions: root.querySelectorAll(".condition-overview > div").length,
edges: root.querySelectorAll(".edge-overview > div").length,
heroLabs: [...document.querySelectorAll(".page-facts > div")]
.find((node) => node.querySelector("dt")?.textContent.trim() === "LABS")
?.querySelector("dd")?.textContent.trim(),
activePanel: root.querySelector("[data-sp-panel]:not([hidden])")
?.dataset.spPanel,
unique: root.querySelector("[data-sp-unique]").textContent.trim(),
eos: root.querySelector("[data-sp-eos]").textContent.trim(),
greedy: root.querySelector("[data-sp-greedy]").textContent.trim(),
documentOverflow: (
document.documentElement.scrollWidth
- document.documentElement.clientWidth
),
};
})()`);
await pause(300);
await screenshot("/tmp/llm-atlas-sampling-desktop.png");
const trajectorySwitch = await evaluate(`(() => {
const root = document.querySelector("[data-sampling-lab]");
const source = root.querySelector("[data-sp-source]");
const condition = root.querySelector("[data-sp-condition]");
source.value = "HumanEval/31";
source.dispatchEvent(new Event("change", { bubbles: true }));
condition.value = "s1_period";
condition.dispatchEvent(new Event("change", { bubbles: true }));
const buttons = [...root.querySelectorAll("[data-sp-seeds] button")];
buttons.at(-1).click();
return {
unique: root.querySelector("[data-sp-unique]").textContent.trim(),
eos: root.querySelector("[data-sp-eos]").textContent.trim(),
greedy: root.querySelector("[data-sp-greedy]").textContent.trim(),
seedBars: buttons.length,
repeats: buttons.filter((button) => button.classList.contains("repeat")).length,
selected: buttons.filter((button) => button.classList.contains("selected")).length,
focus: root.querySelector("[data-sp-focus-seed]").textContent.trim(),
hash: root.querySelector("[data-sp-focus-hash]").textContent.trim(),
};
})()`);
const taskSwitch = await evaluate(`(() => {
const root = document.querySelector("[data-sampling-lab]");
root.querySelector('[data-sp-tab="tasks"]').click();
const condition = root.querySelector("[data-sp-task-condition]");
const read = () => ({
panel: root.querySelector("[data-sp-panel]:not([hidden])").dataset.spPanel,
math: root.querySelector("[data-sp-math-total]").textContent.trim(),
code: root.querySelector("[data-sp-code-total]").textContent.trim(),
mathCells: root.querySelectorAll("[data-sp-math-cells] article").length,
codeCells: root.querySelectorAll("[data-sp-code-cells] article").length,
});
condition.value = "s1_x";
condition.dispatchEvent(new Event("change", { bubbles: true }));
const systemX = read();
condition.value = "s0_period";
condition.dispatchEvent(new Event("change", { bubbles: true }));
const period = read();
return { systemX, period };
})()`);
const edgeSwitch = await evaluate(`(() => {
const root = document.querySelector("[data-sampling-lab]");
root.querySelector('[data-sp-tab="edges"]').click();
const source = root.querySelector("[data-sp-edge-source]");
const edge = root.querySelector("[data-sp-edge]");
source.value = "gsm8k/test/1069";
source.dispatchEvent(new Event("change", { bubbles: true }));
edge.value = "system_eos";
edge.dispatchEvent(new Event("change", { bubbles: true }));
return {
panel: root.querySelector("[data-sp-panel]:not([hidden])").dataset.spPanel,
aligned: root.querySelector("[data-sp-edge-aligned]").textContent.trim(),
nearest: root.querySelector("[data-sp-edge-nearest]").textContent.trim(),
overlap: root.querySelector("[data-sp-edge-overlap]").textContent.trim(),
left: root.querySelectorAll("[data-sp-edge-left-set] article").length,
right: root.querySelectorAll("[data-sp-edge-right-set] article").length,
highlighted: root.querySelectorAll(".two-sets article.overlap").length,
};
})()`);
const keyboard = await evaluate(`(() => {
const root = document.querySelector("[data-sampling-lab]");
const first = root.querySelector('[data-sp-tab="trajectories"]');
first.click();
first.focus();
first.dispatchEvent(new KeyboardEvent("keydown", {
key: "ArrowRight",
bubbles: true,
}));
const selected = root.querySelector(
'[data-sp-tab][aria-selected="true"]',
).dataset.spTab;
root.querySelector('[data-sp-tab="reproduction"]').click();
return {
keyboardSelected: selected,
visible: root.querySelector("[data-sp-panel]:not([hidden])")
.dataset.spPanel,
reproFields: root.querySelectorAll(".repro-fields article").length,
};
})()`);
await command("Emulation.setDeviceMetricsOverride", {
width: 390,
height: 844,
deviceScaleFactor: 1,
mobile: true,
});
await navigate("/deepseek/#sampling");
const mobile = await evaluate(`(() => {
const root = document.querySelector("[data-sampling-lab]");
document.documentElement.style.scrollBehavior = "auto";
window.scrollTo(0, root.getBoundingClientRect().top + window.scrollY);
return {
tabs: root.querySelectorAll("[data-sp-tab]").length,
seedBars: root.querySelectorAll("[data-sp-seeds] button").length,
documentOverflow: (
document.documentElement.scrollWidth
- document.documentElement.clientWidth
),
rootOverflow: root.scrollWidth - root.clientWidth,
};
})()`);
await pause(300);
await screenshot("/tmp/llm-atlas-sampling-mobile.png");
assert(overview.tabs === 4, `sampling tabs=${overview.tabs}`);
assert(overview.panels === 4, `sampling panels=${overview.panels}`);
assert(overview.seedBars === 8, `initial seed bars=${overview.seedBars}`);
assert(overview.conditions === 8, `condition rows=${overview.conditions}`);
assert(overview.edges === 10, `edge rows=${overview.edges}`);
assert(overview.heroLabs?.startsWith("20"), `hero labs=${overview.heroLabs}`);
assert(overview.activePanel === "trajectories", `active=${overview.activePanel}`);
assert(overview.unique === "8 / 8", `initial unique=${overview.unique}`);
assert(overview.eos === "5 / 8", `initial eos=${overview.eos}`);
assert(overview.greedy === "NO", `initial greedy=${overview.greedy}`);
assert(overview.documentOverflow === 0, `desktop overflow=${overview.documentOverflow}`);
assert(trajectorySwitch.unique === "2 / 8", `switched unique=${trajectorySwitch.unique}`);
assert(trajectorySwitch.eos === "8 / 8", `switched eos=${trajectorySwitch.eos}`);
assert(trajectorySwitch.greedy === "YES", `switched greedy=${trajectorySwitch.greedy}`);
assert(trajectorySwitch.seedBars === 8, `switched seeds=${trajectorySwitch.seedBars}`);
assert(trajectorySwitch.repeats >= 2, `repeat markers=${trajectorySwitch.repeats}`);
assert(trajectorySwitch.selected === 1, `selected seeds=${trajectorySwitch.selected}`);
assert(trajectorySwitch.focus.startsWith("R7"), `focus=${trajectorySwitch.focus}`);
assert(trajectorySwitch.hash.includes("sha256"), `focus hash=${trajectorySwitch.hash}`);
assert(taskSwitch.systemX.panel === "tasks", `task panel=${taskSwitch.systemX.panel}`);
assert(taskSwitch.systemX.math === "7 / 8 PASS", `system x math=${taskSwitch.systemX.math}`);
assert(taskSwitch.systemX.code === "8 / 8 PASS", `system x code=${taskSwitch.systemX.code}`);
assert(taskSwitch.period.code === "7 / 8 PASS", `period code=${taskSwitch.period.code}`);
assert(taskSwitch.period.mathCells === 8, `math cells=${taskSwitch.period.mathCells}`);
assert(taskSwitch.period.codeCells === 8, `code cells=${taskSwitch.period.codeCells}`);
assert(edgeSwitch.panel === "edges", `edge panel=${edgeSwitch.panel}`);
assert(edgeSwitch.aligned === "77.3%", `edge aligned=${edgeSwitch.aligned}`);
assert(edgeSwitch.nearest === "89.4%", `edge nearest=${edgeSwitch.nearest}`);
assert(edgeSwitch.overlap === "1 / 14", `edge overlap=${edgeSwitch.overlap}`);
assert(edgeSwitch.left === 8 && edgeSwitch.right === 8, `edge sets=${edgeSwitch.left}/${edgeSwitch.right}`);
assert(edgeSwitch.highlighted >= 2, `edge highlights=${edgeSwitch.highlighted}`);
assert(keyboard.keyboardSelected === "tasks", `keyboard selected=${keyboard.keyboardSelected}`);
assert(keyboard.visible === "reproduction", `repro visible=${keyboard.visible}`);
assert(keyboard.reproFields === 8, `repro fields=${keyboard.reproFields}`);
assert(mobile.tabs === 4 && mobile.seedBars === 8, `mobile controls=${mobile.tabs}/${mobile.seedBars}`);
assert(mobile.documentOverflow === 0, `mobile document overflow=${mobile.documentOverflow}`);
assert(mobile.rootOverflow === 0, `mobile root overflow=${mobile.rootOverflow}`);
assert(exceptions.length === 0, `runtime exceptions: ${exceptions.join(" | ")}`);
console.log(JSON.stringify({
overview,
trajectorySwitch,
taskSwitch,
edgeSwitch,
keyboard,
mobile,
exceptions,
screenshots: [
"/tmp/llm-atlas-sampling-desktop.png",
"/tmp/llm-atlas-sampling-mobile.png",
],
}, null, 2));
socket.close();
src/components/DeepSeekSamplingLab.astro
+998
View File
@@ -0,0 +1,998 @@
---
import rawLab from "@/data/deepseek-v2-lite-chat-sampling-compact.json";
const lab = rawLab as any;
const json = JSON.stringify(lab).replaceAll("<", "\\u003c");
const conditions = lab.contract.conditions as string[];
const edges = lab.contract.edges as string[];
const conditionLabels: Record<string, string> = {
s0_eos: "S0 · EOS",
s1_eos: "S1 · EOS",
s0_bos: "S0 · BOS",
s1_bos: "S1 · BOS",
s0_x: "S0 · x",
s1_x: "S1 · x",
s0_period: "S0 · 句点",
s1_period: "S1 · 句点",
};
const sourceLabels: Record<string, string> = {
"wikitext2/raw-validation/0443": "English · WikiText",
"tnews/test/4855": "中文 · TNEWS",
"HumanEval/31": "Code · HumanEval/31",
"gsm8k/test/1069": "Math · GSM8K/1069",
};
---
<figure class="sampling-lab" data-sampling-lab>
<header class="sp-head">
<div>
<p>ROUND 06 / PREREGISTERED SAMPLING</p>
<h3>greedy 只是一条路：固定八个 seed，打开有限的生成轨迹集合</h3>
</div>
<p>
同一官方 SFT Chat checkpoint、同四条 source、同八格 prompt batch。
只把解码切到官方 <code>temperature=.3 · top_p=.95</code>，显式关闭 top-k；
生成、任务评测与新进程复跑仍分三层保存。
</p>
</header>
<div class="sp-ledger">
<article class="pass"><span>SAMPLED OUTPUTS</span><b>256</b><p>4 sources × 8 seeds × 8 cells</p></article>
<article class="pass"><span>NATURAL EOS</span><b>251 / 256</b><p>5 格在 512-token 触顶</p></article>
<article><span>UNIQUE TRAJECTORIES</span><b>242 / 256</b><p>完整 token hash；不是语义类别</p></article>
<article><span>MATH · STRICT</span><b>62 / 64</b><p>同一题的重复采样</p></article>
<article><span>CODE · TESTS</span><b>63 / 64</b><p>24 个唯一执行候选</p></article>
<article class="pass"><span>FRESH PROCESS</span><b>64 / 64</b><p>八项预注册字段 exact</p></article>
</div>
<div class="sp-tabs" role="tablist" aria-label="选择采样稳健性实验视图">
<button type="button" role="tab" data-sp-tab="trajectories" aria-selected="true">
<span>01</span><b>八个 seed 生成了什么</b><small>trajectory microscope</small>
</button>
<button type="button" role="tab" data-sp-tab="tasks" aria-selected="false" tabindex="-1">
<span>02</span><b>完成为什么仍会答错</b><small>math + code ledgers</small>
</button>
<button type="button" role="tab" data-sp-tab="edges" aria-selected="false" tabindex="-1">
<span>03</span><b>两个样本集合怎样比</b><small>aligned + nearest set</small>
</button>
<button type="button" role="tab" data-sp-tab="reproduction" aria-selected="false" tabindex="-1">
<span>04</span><b>随机但仍可复现</b><small>seed contract + rerun</small>
</button>
</div>
<section class="sp-panel" data-sp-panel="trajectories">
<div class="sp-panel-lead">
<div><span>I / TRAJECTORY MICROSCOPE</span><h4>同一格不是一个答案，而是八条有限样本</h4></div>
<p>
点任意 seed 查看长度、停止状态、hash 与短预览。unique 只比较完整 token IDs；
pairwise similarity 用 token 编辑距离，不把同义改写冒充 exact。
</p>
</div>
<div class="sp-controls">
<label>
<span>SOURCE</span>
<select data-sp-source aria-label="选择采样 source">
{lab.sources.map((source: any) => (
<option value={source.id}>{sourceLabels[source.id]}</option>
))}
</select>
</label>
<label>
<span>CONDITION</span>
<select data-sp-condition aria-label="选择采样 condition">
{conditions.map((condition) => (
<option value={condition}>{conditionLabels[condition]}</option>
))}
</select>
</label>
<div class="sampling-config">
<span>DECODE CONTRACT</span>
<b>T .3 · P .95 · K 0 · CAP 512</b>
</div>
</div>
<div class="trajectory-summary">
<article><span>UNIQUE / 8</span><b data-sp-unique>—</b><p>完整 trajectory hash</p></article>
<article><span>NATURAL EOS / 8</span><b data-sp-eos>—</b><p>触顶与自然结束分开</p></article>
<article><span>GREEDY IN SET</span><b data-sp-greedy>—</b><p>八样本是否抽到 mode 轨迹</p></article>
<article><span>PAIRWISE SIM</span><b data-sp-pairwise>—</b><p>28 对 mean · min–max</p></article>
</div>
<div class="seed-stage">
<header>
<div><span>8 PREREGISTERED SEEDS</span><b data-sp-stage-title>—</b></div>
<p><i class="eos"></i>EOS <i class="truncated"></i>512 截断 <i class="repeat"></i>重复完整轨迹</p>
</header>
<div class="seed-bars" data-sp-seeds></div>
<article class="seed-focus" data-sp-focus>
<div>
<span data-sp-focus-seed>选择一个 seed</span>
<b data-sp-focus-meta>—</b>
</div>
<p data-sp-focus-preview>—</p>
<code data-sp-focus-hash>—</code>
</article>
</div>
<div class="condition-overview" aria-label="八个 condition 的采样摘要">
<header><b>CONDITION</b><b>EOS</b><b>MEAN TOKENS</b><b>UNIQUE</b><b>GREEDY SETS</b></header>
{conditions.map((condition) => {
const row = lab.conditions[condition];
return (
<div>
<span>{conditionLabels[condition]}</span>
<b class={row.naturalEos === 32 ? "good" : "warn"}>{row.naturalEos} / 32</b>
<b>{row.meanGeneratedTokens.toFixed(1)}</b>
<b>{row.uniqueTrajectoriesAcrossSourceSets} / 32</b>
<b>{row.greedyIncludedSourceSets} / 4</b>
</div>
);
})}
</div>
<aside class="sp-warning">
<b>9 / 32 个八样本集合包含 greedy 完整轨迹</b>
<p>
这不等于 greedy “不可信”。greedy 每步选 mode；nucleus sampling 从截断后的分布抽样。
只抽八次没有遇到 mode 路径很正常。
</p>
</aside>
</section>
<section class="sp-panel" data-sp-panel="tasks" hidden>
<div class="sp-panel-lead">
<div><span>II / TASK LEDGERS</span><h4>自然结束、能解析、能运行，仍然不保证正确</h4></div>
<p>
这里各只有一条 task source。64 是 8 conditions × 8 seeds，不是 64 道题；
gold 与 tests 在生成冻结后才进入独立 evaluator。
</p>
</div>
<div class="task-condition-control">
<label><span>查看一个 condition 的 8 个 seed</span>
<select data-sp-task-condition aria-label="选择任务 condition">
{conditions.map((condition) => (
<option value={condition}>{conditionLabels[condition]}</option>
))}
</select>
</label>
<div><span>全局任务账</span><b>Math 62 / 64 · Code 63 / 64</b></div>
</div>
<div class="sample-task-grid">
<article>
<header><div><span>GSM8K / 1069</span><b data-sp-math-total>—</b></div><em>gold = 300</em></header>
<div class="seed-task-cells" data-sp-math-cells></div>
</article>
<article>
<header><div><span>HUMANEVAL / 31</span><b data-sp-code-total>—</b></div><em>official tests</em></header>
<div class="seed-task-cells" data-sp-code-cells></div>
</article>
</div>
<div class="failure-cases">
<article>
<span>MATH FAILURE · 2 / 64</span>
<h5>“坏掉 1/4”被误写成“只剩 1/4”</h5>
<div class="reasoning-bug">
<b>20 trucks × 1/4 = 5</b><i>错误语义跳跃</i><b>5 × 20 = 100</b>
</div>
<p>
两条都自然 EOS，并明确输出 <code>{"\\boxed{100}"}</code>；抽取器没有错，
是推理把仍可工作的 <code>3/4</code> 丢掉了。
</p>
</article>
<article>
<span>CODE FAILURE · 1 / 64</span>
<h5>AST 合法、可以执行，却漏掉所有大于 2 的偶数</h5>
<pre><code>if n &lt; 2: return False
if n == 2: return True
for i in range(3, sqrt(n), 2): ...
# 缺少 if n % 2 == 0: return False</code></pre>
<p>official test 触发 assertion failure；流畅说明与闭合 code fence 都没有替它兜底。</p>
</article>
</div>
<div class="four-ledgers">
<article class="done"><span>01</span><b>STOPPING</b><p>EOS / budget cap</p></article>
<i>→</i>
<article class="done"><span>02</span><b>TASK TERMINAL</b><p>final marker / fence</p></article>
<i>→</i>
<article class="done"><span>03</span><b>COVERAGE</b><p>number / AST + sandbox</p></article>
<i>→</i>
<article><span>04</span><b>CORRECTNESS</b><p>gold / official tests</p></article>
</div>
</section>
<section class="sp-panel" data-sp-panel="edges" hidden>
<div class="sp-panel-lead">
<div><span>III / SET-TO-SET COMPARISON</span><h4>不要拿一对随机回答，冒充两个条件的分布</h4></div>
<p>
aligned 看同 replicate label 的八对；nearest 让每条样本去另一侧寻找最接近轨迹，
再双向平均。两者都只是八样本描述量。
</p>
</div>
<div class="sp-controls edge-controls">
<label><span>SOURCE</span>
<select data-sp-edge-source aria-label="选择 edge source">
{lab.sources.map((source: any) => (
<option value={source.id}>{sourceLabels[source.id]}</option>
))}
</select>
</label>
<label><span>EDGE</span>
<select data-sp-edge aria-label="选择采样 edge">
{edges.map((edge) => (
<option value={edge}>{lab.contract.edgeLabels[edge]}</option>
))}
</select>
</label>
<div class="sampling-config"><span>PAIRING STATUS</span><b>BATCH-SEED ALIGNED ≠ CRN</b></div>
</div>
<div class="set-metrics">
<article><span>ALIGNED MEAN</span><b data-sp-edge-aligned>—</b><i><u data-sp-edge-aligned-bar></u></i></article>
<article><span>SYMMETRIC NEAREST</span><b data-sp-edge-nearest>—</b><i><u data-sp-edge-nearest-bar></u></i></article>
<article><span>EXACT HASH ∩ / ∪</span><b data-sp-edge-overlap>—</b><p>完整 token trajectory</p></article>
</div>
<div class="two-sets">
<article>
<header><span>LEFT SAMPLE SET</span><b data-sp-edge-left-label>—</b></header>
<div data-sp-edge-left-set></div>
</article>
<div class="set-bridge"><b>8 ↔ 8</b><span>双向找最近邻</span><i></i></div>
<article>
<header><span>RIGHT SAMPLE SET</span><b data-sp-edge-right-label>—</b></header>
<div data-sp-edge-right-set></div>
</article>
</div>
<div class="edge-overview">
<header><b>EDGE · FOUR-SOURCE MEAN</b><b>ALIGNED</b><b>NEAREST</b><b>EXACT ∩ / ∪</b></header>
{edges.map((edge) => {
const row = lab.edges[edge];
return (
<div>
<span>{row.label}</span>
<b>{(row.meanAlignedSimilarity * 100).toFixed(1)}%</b>
<b>{(row.meanSymmetricNearestSimilarity * 100).toFixed(1)}%</b>
<b>{row.exactHashIntersections} / {row.exactHashUnion}</b>
<i style={`--edge-nearest:${row.meanSymmetricNearestSimilarity * 100}%`}></i>
</div>
);
})}
</div>
<aside class="sp-warning dark">
<b>为什么 nearest 总比 aligned 高？</b>
<p>
nearest 主动从另一侧八条里挑最像的一条；aligned 没有这个选择自由。
因此差值不是“条件效应”，nearest 也不是无偏分布距离。
</p>
</aside>
</section>
<section class="sp-panel" data-sp-panel="reproduction" hidden>
<div class="sp-panel-lead">
<div><span>IV / RANDOMNESS × REPRODUCTION</span><h4>“会变化”和“可复现”必须同时成立</h4></div>
<p>
不同 seed 应该分叉；相同 checkpoint、输入、行顺序、软件与 seed 则应该回到同一条
token trajectory。两者不是矛盾，而是采样实验的两道独立闸门。
</p>
</div>
<div class="randomness-equation">
<article><span>DIFFERENT RUN SEED</span><b>R0 ≠ R1</b><p>31 / 32 同格 trajectory 分叉</p></article>
<i>+</i>
<article><span>SAME FULL CONTRACT</span><b>R0 = R0′</b><p>新进程完整 token IDs exact</p></article>
<i>=</i>
<article class="result"><span>REPRODUCIBLE SAMPLING</span><b>64 / 64</b><p>随机轨迹不是环境漂移</p></article>
</div>
<div class="seed-derivation">
<header><span>SEED CONTRACT</span><b>结果之前冻结，不手选“好看 seed”</b></header>
<div>
<article><span>01 / BASE</span><b>SHA256(protocol / seed / index)</b><p>取前 4 bytes</p></article>
<i>→</i>
<article><span>02 / SOURCE</span><b>SHA256(protocol · base · source ID)</b><p>取前 8 bytes mod 2⁶³−1</p></article>
<i>→</i>
<article><span>03 / RNG</span><b>torch + CUDA manual seed</b><p>记录 pre/post state hash</p></article>
<i>→</i>
<article><span>04 / BATCH</span><b>固定八行顺序</b><p>顺序改变即新协议</p></article>
</div>
</div>
<div class="repro-fields">
{[
["RUN SEED", "64 / 64"],
["PROMPT HASH", "64 / 64"],
["TOKEN IDS", "64 / 64"],
["DECODED TEXT", "64 / 64"],
["EOS STATE", "64 / 64"],
["TRUNCATION", "64 / 64"],
["CPU RNG PRE", "64 / 64"],
["CUDA RNG PRE", "64 / 64"],
].map(([label, value]) => (
<article><span>{label}</span><b>{value}</b><i>EXACT</i></article>
))}
</div>
<div class="environment-lock">
<article><span>CHECKPOINT</span><b>85864749…f64c7</b><p>12 个模型文件逐 SHA-256</p></article>
<article><span>SOFTWARE</span><b>Torch 2.11 · TF 4.41.2</b><p>generation utils 也记录 hash</p></article>
<article><span>PLACEMENT</span><b>CUDA L0–23 · CPU L24–26</b><p>final norm + LM head offload</p></article>
<article><span>SCOPE</span><b>R0 / R1 only</b><p>不能扩大写成 256 / 256</p></article>
</div>
<aside class="sp-warning">
<b>exact replay 不是跨版本承诺</b>
<p>
更换 PyTorch、Transformers、CUDA kernel、硬件或 batch 行顺序都可能改变轨迹。
这里证明的是固定执行合同内的可复现性。
</p>
</aside>
</section>
<figcaption>
<b>证据边界</b>
<span>
4 sources × 8 seeds 不是 benchmark，也没有恢复完整生成分布。BOS / x / 句点格不是
官方有效聊天格式；unique hash 不是语义多样性；batch-seed aligned 不是逐行共享随机数。
</span>
<code>FORMAL 46c7edf…45af · EVAL 078f486…c4d8 · RERUN 72d050e…f6a0</code>
</figcaption>
<script is:inline type="application/json" data-sp-data set:html={json}></script>
</figure>
<script>
document.querySelectorAll<HTMLElement>("[data-sampling-lab]").forEach((root) => {
const payload = root.querySelector<HTMLScriptElement>("[data-sp-data]");
if (!payload) return;
const data = JSON.parse(payload.textContent ?? "{}");
const one = <T extends Element>(selector: string) => (
root.querySelector<T>(selector)
);
const set = (selector: string, value: string) => {
const node = one<HTMLElement>(selector);
if (node) node.textContent = value;
};
const pct = (value: number, digits = 1) => (
`${(value * 100).toFixed(digits)}%`
);
const labels: Record<string, string> = {
s0_eos: "S0 · EOS",
s1_eos: "S1 · EOS",
s0_bos: "S0 · BOS",
s1_bos: "S1 · BOS",
s0_x: "S0 · x",
s1_x: "S1 · x",
s0_period: "S0 · 句点",
s1_period: "S1 · 句点",
};
const sourceById = new Map(
data.sources.map((source: any) => [source.id, source]),
);
const tabs = [
...root.querySelectorAll<HTMLButtonElement>("[data-sp-tab]"),
];
const panels = [
...root.querySelectorAll<HTMLElement>("[data-sp-panel]"),
];
tabs.forEach((button, index) => {
button.addEventListener("click", () => {
const target = button.dataset.spTab;
tabs.forEach((candidate) => {
const active = candidate === button;
candidate.setAttribute("aria-selected", String(active));
candidate.tabIndex = active ? 0 : -1;
});
panels.forEach((panel) => {
panel.hidden = panel.dataset.spPanel !== target;
});
});
button.addEventListener("keydown", (event) => {
if (!["ArrowLeft", "ArrowRight"].includes(event.key)) return;
event.preventDefault();
const delta = event.key === "ArrowRight" ? 1 : -1;
tabs[(index + delta + tabs.length) % tabs.length].click();
tabs[(index + delta + tabs.length) % tabs.length].focus();
});
});
const sourceSelect = one<HTMLSelectElement>("[data-sp-source]");
const conditionSelect = one<HTMLSelectElement>("[data-sp-condition]");
const seedContainer = one<HTMLElement>("[data-sp-seeds]");
const renderFocus = (sample: any, repeated: boolean) => {
set(
"[data-sp-focus-seed]",
`${sample.replicate} · base ${sample.baseSeed.toLocaleString()}`,
);
set(
"[data-sp-focus-meta]",
`${sample.generatedTokens} TOKENS · ${sample.hitEos ? "NATURAL EOS" : "BUDGET TRUNCATED"}${repeated ? " · REPEATED HASH" : ""}`,
);
set(
"[data-sp-focus-preview]",
sample.preview || "（空文本）",
);
set(
"[data-sp-focus-hash]",
`trajectory sha256 · ${sample.trajectoryHash}`,
);
};
const renderTrajectories = () => {
const source = sourceById.get(sourceSelect?.value) as any;
const condition = conditionSelect?.value ?? "s0_eos";
if (!source) return;
const row = source.conditions[condition];
set("[data-sp-unique]", `${row.uniqueTrajectories} / 8`);
set("[data-sp-eos]", `${row.naturalEos} / 8`);
set("[data-sp-greedy]", row.greedyInSamples ? "YES" : "NO");
set(
"[data-sp-pairwise]",
`${pct(row.pairwiseSimilarity.mean)} · ${pct(row.pairwiseSimilarity.min)}–${pct(row.pairwiseSimilarity.max)}`,
);
set(
"[data-sp-stage-title]",
`${source.id} · ${labels[condition]}`,
);
const counts = new Map<string, number>();
row.trajectories.forEach((sample: any) => {
counts.set(
sample.trajectoryHash,
(counts.get(sample.trajectoryHash) ?? 0) + 1,
);
});
if (!seedContainer) return;
const buttons = row.trajectories.map((sample: any, index: number) => {
const button = document.createElement("button");
button.type = "button";
button.className = sample.hitEos ? "eos" : "truncated";
const repeated = (counts.get(sample.trajectoryHash) ?? 0) > 1;
if (repeated) button.classList.add("repeat");
button.setAttribute(
"aria-label",
`${sample.replicate}, ${sample.generatedTokens} tokens`,
);
const label = document.createElement("span");
label.textContent = sample.replicate;
const chart = document.createElement("i");
chart.style.setProperty(
"--height",
`${Math.max(4, sample.generatedTokens / 512 * 100)}%`,
);
const meta = document.createElement("b");
meta.textContent = String(sample.generatedTokens);
const hash = document.createElement("code");
hash.textContent = sample.trajectoryHash.slice(0, 7);
button.append(label, chart, meta, hash);
button.addEventListener("click", () => {
buttons.forEach((candidate: HTMLButtonElement) => (
candidate.classList.toggle("selected", candidate === button)
));
renderFocus(sample, repeated);
});
if (index === 0) button.classList.add("selected");
return button;
});
seedContainer.replaceChildren(...buttons);
renderFocus(
row.trajectories[0],
(counts.get(row.trajectories[0].trajectoryHash) ?? 0) > 1,
);
};
sourceSelect?.addEventListener("change", renderTrajectories);
conditionSelect?.addEventListener("change", renderTrajectories);
const taskCondition = one<HTMLSelectElement>("[data-sp-task-condition]");
const renderTaskCells = (
domain: "math" | "code",
containerSelector: string,
totalSelector: string,
) => {
const source = data.sources.find(
(candidate: any) => candidate.domain === domain,
);
const condition = taskCondition?.value ?? "s0_eos";
const rows = source.conditions[condition].trajectories;
const container = one<HTMLElement>(containerSelector);
if (!container) return;
let passing = 0;
const cells = rows.map((sample: any) => {
const article = document.createElement("article");
const passed = domain === "math"
? sample.math?.exact
: sample.code?.passed;
passing += Number(Boolean(passed));
article.className = passed ? "pass" : "fail";
const seed = document.createElement("span");
seed.textContent = sample.replicate;
const result = document.createElement("b");
result.textContent = domain === "math"
? `${sample.math?.predicted ?? "—"}${passed ? " ✓" : " ✕"}`
: `${(sample.code?.status ?? "not run").replaceAll("_", " ")}${passed ? " ✓" : " ✕"}`;
const meta = document.createElement("em");
meta.textContent = `${sample.generatedTokens} TOKENS · ${sample.hitEos ? "EOS" : "CAP"}`;
article.append(seed, result, meta);
return article;
});
container.replaceChildren(...cells);
set(totalSelector, `${passing} / ${rows.length} PASS`);
};
const renderTasks = () => {
renderTaskCells(
"math",
"[data-sp-math-cells]",
"[data-sp-math-total]",
);
renderTaskCells(
"code",
"[data-sp-code-cells]",
"[data-sp-code-total]",
);
};
taskCondition?.addEventListener("change", renderTasks);
const edgeSource = one<HTMLSelectElement>("[data-sp-edge-source]");
const edgeSelect = one<HTMLSelectElement>("[data-sp-edge]");
const renderSet = (
samples: any[],
otherHashes: Set<string>,
selector: string,
) => {
const container = one<HTMLElement>(selector);
if (!container) return;
const cells = samples.map((sample) => {
const article = document.createElement("article");
article.className = otherHashes.has(sample.trajectoryHash)
? "overlap"
: "";
const seed = document.createElement("span");
seed.textContent = sample.replicate;
const hash = document.createElement("b");
hash.textContent = sample.trajectoryHash.slice(0, 8);
const meta = document.createElement("em");
meta.textContent = `${sample.generatedTokens}T · ${sample.hitEos ? "EOS" : "CAP"}`;
article.append(seed, hash, meta);
return article;
});
container.replaceChildren(...cells);
};
const renderEdge = () => {
const source = sourceById.get(edgeSource?.value) as any;
const edgeName = edgeSelect?.value ?? "system_eos";
if (!source) return;
const row = source.edges[edgeName];
const left = source.conditions[row.left];
const right = source.conditions[row.right];
set(
"[data-sp-edge-aligned]",
pct(row.batch_seed_aligned_similarity.mean),
);
set(
"[data-sp-edge-nearest]",
pct(row.symmetric_mean_nearest_neighbor_similarity),
);
set(
"[data-sp-edge-overlap]",
`${row.generated_hash_set_intersection} / ${row.generated_hash_set_union}`,
);
const alignedBar = one<HTMLElement>("[data-sp-edge-aligned-bar]");
const nearestBar = one<HTMLElement>("[data-sp-edge-nearest-bar]");
if (alignedBar) alignedBar.style.width = pct(
row.batch_seed_aligned_similarity.mean,
);
if (nearestBar) nearestBar.style.width = pct(
row.symmetric_mean_nearest_neighbor_similarity,
);
set("[data-sp-edge-left-label]", labels[row.left]);
set("[data-sp-edge-right-label]", labels[row.right]);
renderSet(
left.trajectories,
new Set(right.trajectories.map(
(sample: any) => sample.trajectoryHash,
)),
"[data-sp-edge-left-set]",
);
renderSet(
right.trajectories,
new Set(left.trajectories.map(
(sample: any) => sample.trajectoryHash,
)),
"[data-sp-edge-right-set]",
);
};
edgeSource?.addEventListener("change", renderEdge);
edgeSelect?.addEventListener("change", renderEdge);
renderTrajectories();
renderTasks();
renderEdge();
});
</script>
<style>
.sampling-lab {
--ink: #182c33;
--muted: #62706e;
--line: rgba(25, 43, 50, .14);
--paper: #f7f3e9;
--teal: #2f7d70;
--orange: #b35935;
margin: 2.2rem 0 0;
overflow: hidden;
color: var(--ink);
border: 1px solid var(--line);
background: var(--paper);
box-shadow: 0 28px 70px rgba(22, 35, 43, .1);
}
.sp-head {
display: grid;
grid-template-columns: 1.08fr .92fr;
gap: 2.4rem;
align-items: end;
padding: 2rem;
color: #fff;
background:
radial-gradient(circle at 78% 18%, rgba(212, 124, 73, .3), transparent 27%),
radial-gradient(circle at 18% 90%, rgba(79, 165, 147, .23), transparent 30%),
linear-gradient(137deg, #112a33, #28444a 64%, #3f4b49);
}
.sp-head p { margin: 0; color: rgba(255,255,255,.74); font-size: .76rem; line-height: 1.72; }
.sp-head > div > p { color: #8ed1c5; font: 760 .6rem/1.2 var(--font-mono); letter-spacing: .11em; }
.sp-head h3 { max-width: 680px; margin: .72rem 0 0; color: #fff; font: 760 clamp(1.55rem,3vw,2.45rem)/1.14 var(--font-display); }
.sp-head code { padding: .08rem .25rem; color: #f3d3bd; background: rgba(0,0,0,.16); font: .64rem/1.2 var(--font-mono); }
.sp-ledger { display: grid; grid-template-columns: repeat(6,1fr); border-bottom: 1px solid var(--line); }
.sp-ledger article { min-width: 0; padding: 1rem; border-right: 1px solid var(--line); background: #ebe7dc; }
.sp-ledger article:last-child { border-right: 0; }
.sp-ledger article.pass { background: rgba(48,126,111,.12); }
.sp-ledger span,
.trajectory-summary span,
.set-metrics span,
.environment-lock span { display: block; color: #61706d; font: 730 .5rem/1.25 var(--font-mono); letter-spacing: .07em; }
.sp-ledger b { display: block; margin-top: .45rem; font: 780 .82rem/1.2 var(--font-mono); }
.sp-ledger p { margin: .34rem 0 0; color: #69736f; font-size: .56rem; line-height: 1.45; }
.sp-tabs { display: grid; grid-template-columns: repeat(4,1fr); border-bottom: 1px solid var(--line); }
.sp-tabs button { display: grid; grid-template-columns: auto 1fr; grid-template-rows: auto auto; column-gap: .72rem; min-width: 0; padding: .92rem 1rem; color: #263a40; border: 0; border-right: 1px solid var(--line); background: #fbf8f1; text-align: left; cursor: pointer; }
.sp-tabs button:last-child { border-right: 0; }
.sp-tabs button[aria-selected="true"] { color: #fff; background: var(--orange); }
.sp-tabs span { grid-row: 1 / 3; opacity: .66; font: 720 .54rem/1.2 var(--font-mono); }
.sp-tabs b { font: 730 .73rem/1.25 var(--font-display); }
.sp-tabs small { opacity: .68; font: .51rem/1.3 var(--font-mono); }
.sp-panel { padding: 1.55rem; }
.sp-panel[hidden] { display: none; }
.sp-panel-lead { display: grid; grid-template-columns: 1fr 1fr; gap: 2rem; align-items: end; margin-bottom: 1.2rem; }
.sp-panel-lead span { color: var(--orange); font: 750 .55rem/1.2 var(--font-mono); letter-spacing: .09em; }
.sp-panel-lead h4 { margin: .35rem 0 0; font: 750 1.24rem/1.2 var(--font-display); }
.sp-panel-lead p { margin: 0; color: var(--muted); font-size: .68rem; line-height: 1.68; }
.sp-controls { display: grid; grid-template-columns: 1fr 1fr .82fr; gap: 1px; border: 1px solid var(--line); background: var(--line); }
.sp-controls > * { min-width: 0; padding: .76rem; background: #ece8de; }
.sp-controls span,
.task-condition-control span { display: block; margin-bottom: .42rem; color: #65716f; font: 720 .5rem/1.2 var(--font-mono); letter-spacing: .06em; }
.sp-controls select,
.task-condition-control select { width: 100%; min-width: 0; padding: .58rem; color: #24383e; border: 1px solid rgba(30,38,43,.2); background: #fffdf8; font: 670 .6rem/1.3 var(--font-mono); }
.sampling-config { display: flex; flex-direction: column; justify-content: center; }
.sampling-config b { color: #315d58; font: 720 .62rem/1.45 var(--font-mono); }
.trajectory-summary { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
.trajectory-summary article { padding: .8rem; background: #f0ece2; }
.trajectory-summary b { display: block; margin-top: .38rem; color: #2b6d62; font: 780 .75rem/1.2 var(--font-mono); }
.trajectory-summary p { margin: .35rem 0 0; color: #6c7571; font-size: .54rem; }
.seed-stage { margin-top: 1rem; border: 1px solid var(--line); background: #eee9df; }
.seed-stage > header { display: flex; justify-content: space-between; gap: 1rem; align-items: center; padding: .75rem .85rem; color: #e8f0ed; background: #29434a; }
.seed-stage header span { display: block; color: #91c9c0; font: 700 .47rem/1.2 var(--font-mono); }
.seed-stage header b { display: block; margin-top: .25rem; font: 690 .62rem/1.2 var(--font-mono); }
.seed-stage header p { margin: 0; font: .5rem/1.4 var(--font-mono); }
.seed-stage header p i { display: inline-block; width: .55rem; height: .55rem; margin: 0 .25rem 0 .6rem; vertical-align: -.08rem; background: #3f9b89; }
.seed-stage header p i:first-child { margin-left: 0; }
.seed-stage header p i.truncated { background: #b45a37; }
.seed-stage header p i.repeat { background: #d4a44d; }
.seed-bars { display: grid; grid-template-columns: repeat(8,1fr); gap: 1px; height: 13rem; background: var(--line); }
.seed-bars button { position: relative; display: grid; grid-template-rows: auto 1fr auto auto; gap: .35rem; min-width: 0; padding: .62rem .48rem; color: #31464b; border: 0; background: #f7f3ea; cursor: pointer; }
.seed-bars button.selected { background: #fffdf8; box-shadow: inset 0 0 0 2px var(--orange); }
.seed-bars button > span { font: 760 .56rem/1 var(--font-mono); }
.seed-bars button > i { position: relative; align-self: end; width: 100%; height: var(--height); max-height: 8.2rem; background: linear-gradient(to top, #2f7d70, #76b6aa); }
.seed-bars button.truncated > i { background: linear-gradient(to top, #a94f31, #d68d63); }
.seed-bars button.repeat > i::after { position: absolute; inset: .25rem; border: 1px dashed #fff6d9; content: ""; }
.seed-bars button > b { font: 740 .6rem/1 var(--font-mono); }
.seed-bars button > code { overflow: hidden; color: #77807c; font: .43rem/1 var(--font-mono); text-overflow: ellipsis; }
.seed-focus { display: grid; grid-template-columns: .34fr 1fr; gap: 1rem; padding: .9rem; border-top: 1px solid var(--line); background: #fbf8f1; }
.seed-focus span { color: var(--orange); font: 750 .53rem/1.2 var(--font-mono); }
.seed-focus b { display: block; margin-top: .4rem; color: #345b57; font: 690 .55rem/1.45 var(--font-mono); }
.seed-focus p { margin: 0; color: #53615f; font-size: .62rem; line-height: 1.55; }
.seed-focus code { grid-column: 1 / -1; overflow-wrap: anywhere; color: #737d79; font: .46rem/1.4 var(--font-mono); }
.condition-overview,
.edge-overview { margin-top: 1rem; border: 1px solid var(--line); }
.condition-overview > header,
.condition-overview > div { display: grid; grid-template-columns: 1.25fr repeat(4,1fr); gap: .65rem; align-items: center; padding: .62rem .8rem; border-bottom: 1px solid rgba(30,38,43,.1); }
.condition-overview > header { color: #e5edeb; background: #29434a; font: 680 .46rem/1.2 var(--font-mono); }
.condition-overview > div:last-child { border-bottom: 0; }
.condition-overview > div:nth-child(odd) { background: #ede9df; }
.condition-overview span { font: 680 .59rem/1.2 var(--font-mono); }
.condition-overview b { color: #5d6865; font: 670 .57rem/1.2 var(--font-mono); }
.condition-overview b.good { color: #277563; }
.condition-overview b.warn { color: #a95131; }
.sp-warning { display: grid; grid-template-columns: .35fr 1fr; gap: 1rem; margin-top: 1rem; padding: .85rem; color: #5c4a3b; border: 1px solid rgba(177,91,50,.2); background: rgba(183,98,53,.09); }
.sp-warning b { font: 730 .63rem/1.4 var(--font-display); }
.sp-warning p { margin: 0; font-size: .61rem; line-height: 1.6; }
.sp-warning.dark { color: #dfe9e6; border: 0; background: #29434a; }
.task-condition-control { display: grid; grid-template-columns: 1fr 1fr; gap: 1px; border: 1px solid var(--line); background: var(--line); }
.task-condition-control > * { min-width: 0; padding: .78rem; background: #ede9df; }
.task-condition-control > div { display: flex; flex-direction: column; justify-content: center; }
.task-condition-control > div b { color: #326a61; font: 740 .69rem/1.2 var(--font-mono); }
.sample-task-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
.sample-task-grid > article { min-width: 0; padding: .95rem; background: #efebe1; }
.sample-task-grid > article > header { display: flex; justify-content: space-between; align-items: end; gap: .8rem; }
.sample-task-grid header span { display: block; color: #6a7571; font: 680 .49rem/1.2 var(--font-mono); }
.sample-task-grid header b { display: block; margin-top: .28rem; color: #326d62; font: 750 .72rem/1.2 var(--font-mono); }
.sample-task-grid header em { color: #8c5a43; font: 680 .52rem/1.2 var(--font-mono); }
.seed-task-cells { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: .72rem; background: var(--line); }
.seed-task-cells article { min-width: 0; padding: .62rem; background: #fffaf2; box-shadow: inset 0 .22rem #ac5434; }
.seed-task-cells article.pass { box-shadow: inset 0 .22rem #358274; }
.seed-task-cells span,
.seed-task-cells b,
.seed-task-cells em { display: block; overflow-wrap: anywhere; }
.seed-task-cells span { color: #727b77; font: .46rem/1.2 var(--font-mono); }
.seed-task-cells b { margin-top: .32rem; font: 680 .55rem/1.25 var(--font-mono); }
.seed-task-cells em { margin-top: .3rem; color: #77817d; font: .41rem/1.35 var(--font-mono); }
.failure-cases { display: grid; grid-template-columns: 1fr 1fr; gap: 1px; margin-top: 1rem; border: 1px solid rgba(174,82,47,.19); background: rgba(174,82,47,.19); }
.failure-cases > article { min-width: 0; padding: 1rem; background: #f4e9df; }
.failure-cases > article > span { color: #a04d31; font: 740 .49rem/1.2 var(--font-mono); letter-spacing: .08em; }
.failure-cases h5 { margin: .45rem 0 .75rem; font: 730 .86rem/1.3 var(--font-display); }
.failure-cases p { margin: .7rem 0 0; color: #665f58; font-size: .6rem; line-height: 1.55; }
.failure-cases p code { font: .52rem/1.2 var(--font-mono); }
.reasoning-bug { display: grid; grid-template-columns: 1fr auto 1fr; gap: .5rem; align-items: center; }
.reasoning-bug b { padding: .62rem; color: #6a3d2e; background: rgba(174,82,47,.1); font: 680 .57rem/1.35 var(--font-mono); }
.reasoning-bug i { color: #a74f31; font: 680 .45rem/1.2 var(--font-mono); }
.failure-cases pre { overflow: auto; margin: 0; padding: .7rem; color: #dfe9e6; background: #293f45; font: .52rem/1.55 var(--font-mono); }
.four-ledgers { display: grid; grid-template-columns: 1fr auto 1fr auto 1fr auto 1fr; gap: .45rem; align-items: center; margin-top: 1rem; }
.four-ledgers article { padding: .75rem; border: 1px solid var(--line); background: #eeeae0; }
.four-ledgers article.done { background: rgba(48,126,111,.11); }
.four-ledgers article > span { color: var(--orange); font: 730 .47rem/1 var(--font-mono); }
.four-ledgers article > b { display: block; margin-top: .35rem; font: 720 .58rem/1.2 var(--font-mono); }
.four-ledgers article > p { margin: .3rem 0 0; color: #6a7470; font-size: .52rem; }
.four-ledgers > i { color: var(--orange); font: 750 .7rem/1 var(--font-mono); }
.set-metrics { display: grid; grid-template-columns: repeat(3,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
.set-metrics article { padding: .85rem; background: #eeeae0; }
.set-metrics b { display: block; margin-top: .38rem; color: #2e6d62; font: 760 .76rem/1.2 var(--font-mono); }
.set-metrics article > i { display: block; height: .38rem; margin-top: .6rem; background: rgba(47,125,112,.16); }
.set-metrics article > i > u { display: block; width: 0; height: 100%; background: var(--teal); text-decoration: none; transition: width .25s ease; }
.set-metrics p { margin: .45rem 0 0; color: #6c7672; font-size: .53rem; }
.two-sets { display: grid; grid-template-columns: 1fr 5.5rem 1fr; gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
.two-sets > article { min-width: 0; padding: .85rem; background: #efebe1; }
.two-sets > article > header { display: flex; justify-content: space-between; gap: .7rem; }
.two-sets header span { color: #717b77; font: .47rem/1.2 var(--font-mono); }
.two-sets header b { color: #3a645e; font: 690 .54rem/1.2 var(--font-mono); }
.two-sets article > div { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: .68rem; background: var(--line); }
.two-sets article > div article { min-width: 0; padding: .55rem; background: #fffaf2; }
.two-sets article > div article.overlap { background: #d9eee7; box-shadow: inset 0 0 0 1px var(--teal); }
.two-sets article > div span,
.two-sets article > div b,
.two-sets article > div em { display: block; overflow: hidden; text-overflow: ellipsis; }
.two-sets article > div span { color: #77817d; font: .43rem/1.2 var(--font-mono); }
.two-sets article > div b { margin-top: .3rem; font: 680 .49rem/1.2 var(--font-mono); }
.two-sets article > div em { margin-top: .25rem; color: #7b817e; font: .39rem/1.2 var(--font-mono); }
.set-bridge { display: grid; place-content: center; gap: .3rem; color: #e6eeeb; background: #29434a; text-align: center; }
.set-bridge b { font: 760 .72rem/1 var(--font-mono); }
.set-bridge span { font: .42rem/1.3 var(--font-mono); }
.set-bridge i { display: block; height: 1px; background: #8dc8bd; }
.edge-overview > header,
.edge-overview > div { position: relative; display: grid; grid-template-columns: 1.6fr repeat(3,1fr); gap: .65rem; align-items: center; padding: .62rem .8rem; overflow: hidden; border-bottom: 1px solid rgba(30,38,43,.1); }
.edge-overview > header { color: #e5edeb; background: #29434a; font: 680 .46rem/1.2 var(--font-mono); }
.edge-overview > div:last-child { border-bottom: 0; }
.edge-overview > div:nth-child(odd) { background: #ede9df; }
.edge-overview span { z-index: 1; font: 650 .56rem/1.2 var(--font-mono); }
.edge-overview b { z-index: 1; color: #5d6965; font: 670 .55rem/1.2 var(--font-mono); }
.edge-overview > div > i { position: absolute; right: 0; bottom: 0; width: var(--edge-nearest); height: .16rem; background: var(--teal); opacity: .7; }
.randomness-equation { display: grid; grid-template-columns: 1fr auto 1fr auto 1fr; gap: .65rem; align-items: center; }
.randomness-equation article { padding: 1rem; border: 1px solid var(--line); background: #eeeae0; }
.randomness-equation article.result { color: #e9f1ef; border: 0; background: #2d776c; }
.randomness-equation span { display: block; opacity: .72; font: .48rem/1.2 var(--font-mono); }
.randomness-equation b { display: block; margin-top: .45rem; font: 780 .95rem/1.2 var(--font-mono); }
.randomness-equation p { margin: .4rem 0 0; opacity: .75; font-size: .55rem; }
.randomness-equation > i { color: var(--orange); font: 780 1rem/1 var(--font-display); }
.seed-derivation { margin-top: 1rem; border: 1px solid var(--line); }
.seed-derivation > header { display: flex; justify-content: space-between; gap: 1rem; padding: .72rem .85rem; color: #e6eeeb; background: #29434a; }
.seed-derivation > header span { color: #8fc9bd; font: .47rem/1.2 var(--font-mono); }
.seed-derivation > header b { font: 670 .56rem/1.2 var(--font-mono); }
.seed-derivation > div { display: grid; grid-template-columns: 1fr auto 1fr auto 1fr auto 1fr; gap: .45rem; align-items: center; padding: .9rem; background: #eeeae0; }
.seed-derivation article { min-width: 0; padding: .65rem; border: 1px solid var(--line); background: #faf7ef; }
.seed-derivation article span { color: var(--orange); font: .44rem/1.2 var(--font-mono); }
.seed-derivation article b { display: block; margin-top: .35rem; overflow-wrap: anywhere; font: 650 .49rem/1.35 var(--font-mono); }
.seed-derivation article p { margin: .32rem 0 0; color: #727b77; font-size: .47rem; }
.seed-derivation > div > i { color: var(--orange); font: 720 .65rem/1 var(--font-mono); }
.repro-fields { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
.repro-fields article { padding: .72rem; background: rgba(48,126,111,.1); }
.repro-fields span { color: #5a716c; font: .46rem/1.2 var(--font-mono); }
.repro-fields b { display: block; margin-top: .35rem; color: #296a5f; font: 740 .62rem/1.2 var(--font-mono); }
.repro-fields i { display: block; margin-top: .28rem; color: #528078; font: .4rem/1.2 var(--font-mono); }
.environment-lock { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
.environment-lock article { min-width: 0; padding: .78rem; background: #eeeae0; }
.environment-lock b { display: block; margin-top: .38rem; overflow-wrap: anywhere; font: 690 .57rem/1.3 var(--font-mono); }
.environment-lock p { margin: .35rem 0 0; color: #6c7571; font-size: .51rem; line-height: 1.45; }
.sampling-lab > figcaption { display: grid; grid-template-columns: auto 1fr auto; gap: 1rem; align-items: center; padding: .9rem 1.1rem; color: #dfe8e6; background: #203a42; }
.sampling-lab > figcaption b { color: #8dc8bd; font: 730 .53rem/1.2 var(--font-mono); }
.sampling-lab > figcaption span { font-size: .54rem; line-height: 1.5; }
.sampling-lab > figcaption code { color: #d9af95; font: .42rem/1.4 var(--font-mono); }
@media (max-width: 900px) {
.sp-head,
.sp-panel-lead { grid-template-columns: 1fr; }
.sp-ledger { grid-template-columns: repeat(3,1fr); }
.sp-ledger article:nth-child(3) { border-right: 0; }
.sp-tabs { grid-template-columns: 1fr 1fr; }
.sp-controls,
.sample-task-grid,
.failure-cases { grid-template-columns: 1fr; }
.two-sets { grid-template-columns: 1fr; }
.set-bridge { min-height: 4rem; }
.sampling-lab > figcaption { grid-template-columns: 1fr; }
}
@media (max-width: 640px) {
.sp-panel { padding: 1rem; }
.sp-head { padding: 1.35rem; }
.sp-ledger,
.trajectory-summary,
.set-metrics,
.repro-fields,
.environment-lock { grid-template-columns: 1fr 1fr; }
.sp-tabs { grid-template-columns: 1fr; }
.seed-bars { grid-template-columns: repeat(4,1fr); height: 22rem; }
.seed-focus { grid-template-columns: 1fr; }
.condition-overview > header,
.condition-overview > div {
grid-template-columns: 1.2fr repeat(4,.72fr);
gap: .25rem;
min-width: 0;
padding: .5rem .35rem;
}
.condition-overview > header b,
.condition-overview > div b,
.condition-overview > div span { font-size: .42rem; }
.seed-task-cells { grid-template-columns: 1fr 1fr; }
.four-ledgers,
.randomness-equation,
.seed-derivation > div { grid-template-columns: 1fr; }
.four-ledgers > i,
.randomness-equation > i,
.seed-derivation > div > i { transform: rotate(90deg); text-align: center; }
.edge-overview > header,
.edge-overview > div {
grid-template-columns: 1.45fr repeat(3,.65fr);
gap: .25rem;
min-width: 0;
padding: .5rem .35rem;
}
.edge-overview > header b,
.edge-overview > div b,
.edge-overview > div span { font-size: .4rem; }
}
</style>
<style is:global>
[data-sampling-lab] [data-sp-seeds] button {
position: relative;
display: grid;
grid-template-rows: auto minmax(4rem, 1fr) auto auto;
gap: .35rem;
min-width: 0;
padding: .62rem .48rem;
overflow: hidden;
color: #31464b;
border: 0;
background: #f7f3ea;
cursor: pointer;
}
[data-sampling-lab] [data-sp-seeds] button.selected {
background: #fffdf8;
box-shadow: inset 0 0 0 2px #b35935;
}
[data-sampling-lab] [data-sp-seeds] button > span {
font: 760 .56rem/1 var(--font-mono);
}
[data-sampling-lab] [data-sp-seeds] button > i {
position: relative;
display: block;
align-self: end;
width: 100%;
height: var(--height);
max-height: 8.2rem;
background: linear-gradient(to top, #2f7d70, #76b6aa);
}
[data-sampling-lab] [data-sp-seeds] button.truncated > i {
background: linear-gradient(to top, #a94f31, #d68d63);
}
[data-sampling-lab] [data-sp-seeds] button.repeat > i::after {
position: absolute;
inset: .25rem;
border: 1px dashed #fff6d9;
content: "";
}
[data-sampling-lab] [data-sp-seeds] button > b {
display: block;
font: 740 .6rem/1 var(--font-mono);
}
[data-sampling-lab] [data-sp-seeds] button > code {
display: block;
overflow: hidden;
color: #77807c;
font: .43rem/1 var(--font-mono);
text-overflow: ellipsis;
}
[data-sampling-lab] [data-sp-math-cells] article,
[data-sampling-lab] [data-sp-code-cells] article {
min-width: 0;
padding: .62rem;
background: #fffaf2;
box-shadow: inset 0 .22rem #ac5434;
}
[data-sampling-lab] [data-sp-math-cells] article.pass,
[data-sampling-lab] [data-sp-code-cells] article.pass {
box-shadow: inset 0 .22rem #358274;
}
[data-sampling-lab] [data-sp-math-cells] article > span,
[data-sampling-lab] [data-sp-math-cells] article > b,
[data-sampling-lab] [data-sp-math-cells] article > em,
[data-sampling-lab] [data-sp-code-cells] article > span,
[data-sampling-lab] [data-sp-code-cells] article > b,
[data-sampling-lab] [data-sp-code-cells] article > em {
display: block;
overflow-wrap: anywhere;
}
[data-sampling-lab] [data-sp-math-cells] article > span,
[data-sampling-lab] [data-sp-code-cells] article > span {
color: #727b77;
font: .46rem/1.2 var(--font-mono);
}
[data-sampling-lab] [data-sp-math-cells] article > b,
[data-sampling-lab] [data-sp-code-cells] article > b {
margin-top: .32rem;
font: 680 .55rem/1.25 var(--font-mono);
}
[data-sampling-lab] [data-sp-math-cells] article > em,
[data-sampling-lab] [data-sp-code-cells] article > em {
margin-top: .3rem;
color: #77817d;
font: .41rem/1.35 var(--font-mono);
}
[data-sampling-lab] [data-sp-edge-left-set] article,
[data-sampling-lab] [data-sp-edge-right-set] article {
min-width: 0;
padding: .55rem;
background: #fffaf2;
}
[data-sampling-lab] [data-sp-edge-left-set] article.overlap,
[data-sampling-lab] [data-sp-edge-right-set] article.overlap {
background: #d9eee7;
box-shadow: inset 0 0 0 1px #2f7d70;
}
[data-sampling-lab] [data-sp-edge-left-set] article > span,
[data-sampling-lab] [data-sp-edge-left-set] article > b,
[data-sampling-lab] [data-sp-edge-left-set] article > em,
[data-sampling-lab] [data-sp-edge-right-set] article > span,
[data-sampling-lab] [data-sp-edge-right-set] article > b,
[data-sampling-lab] [data-sp-edge-right-set] article > em {
display: block;
overflow: hidden;
text-overflow: ellipsis;
}
[data-sampling-lab] [data-sp-edge-left-set] article > span,
[data-sampling-lab] [data-sp-edge-right-set] article > span {
color: #77817d;
font: .43rem/1.2 var(--font-mono);
}
[data-sampling-lab] [data-sp-edge-left-set] article > b,
[data-sampling-lab] [data-sp-edge-right-set] article > b {
margin-top: .3rem;
font: 680 .49rem/1.2 var(--font-mono);
}
[data-sampling-lab] [data-sp-edge-left-set] article > em,
[data-sampling-lab] [data-sp-edge-right-set] article > em {
margin-top: .25rem;
color: #7b817e;
font: .39rem/1.2 var(--font-mono);
}
</style>
src/data/deepseek-v2-lite-chat-sampling-compact.json
+8389
View File
File diff suppressed because it is too large Load Diff
src/data/deepseek-v2-lite-chat-sampling-eval.json
+8442
View File
File diff suppressed because it is too large Load Diff
src/data/deepseek-v2-lite-chat-sampling-repro-r0r1.json
+19030
View File
File diff suppressed because it is too large Load Diff
src/data/deepseek-v2-lite-chat-sampling-reproduction.json
+946
View File
@@ -0,0 +1,946 @@
{
"schema_version": 1,
"protocol_id": "llm-atlas-deepseek-chat-sampling-v1",
"formal": {
"path": "/tmp/deepseek-v2-lite-chat-sampling-formal.json",
"sha256": "46c7edfce1409e798d6b0f06e905dd3a9d4ab6911acd8e42e2ec9d06d65345af",
"content_hash": "7ed89b59e852136892e56da434b212e80a1f26f771e63bd6f92eeb8617a18ad5",
"base_seeds": [
19683830,
1560062173,
3978401375,
1280933274,
1467459869,
1297359489,
2722953988,
3330978061
]
},
"rerun": {
"path": "/tmp/deepseek-v2-lite-chat-sampling-repro-r0r1.json",
"sha256": "72d050eace531bb38fb2592e39dc7f1b9c9003306c92c568dcd4acaaa691f6a0",
"content_hash": "f8a5fb017ab0e517b1df2899e388c47d07b4bd910912639df542703d1d9509e0",
"base_seeds": [
19683830,
1560062173
]
},
"rows": [
{
"source_id": "HumanEval/31",
"base_seed": 19683830,
"condition": "s0_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 19683830,
"condition": "s0_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 19683830,
"condition": "s0_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 19683830,
"condition": "s0_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 19683830,
"condition": "s1_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 19683830,
"condition": "s1_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 19683830,
"condition": "s1_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 19683830,
"condition": "s1_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 1560062173,
"condition": "s0_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 1560062173,
"condition": "s0_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 1560062173,
"condition": "s0_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 1560062173,
"condition": "s0_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 1560062173,
"condition": "s1_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 1560062173,
"condition": "s1_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 1560062173,
"condition": "s1_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "HumanEval/31",
"base_seed": 1560062173,
"condition": "s1_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 19683830,
"condition": "s0_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 19683830,
"condition": "s0_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 19683830,
"condition": "s0_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 19683830,
"condition": "s0_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 19683830,
"condition": "s1_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 19683830,
"condition": "s1_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 19683830,
"condition": "s1_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 19683830,
"condition": "s1_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 1560062173,
"condition": "s0_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 1560062173,
"condition": "s0_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 1560062173,
"condition": "s0_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 1560062173,
"condition": "s0_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 1560062173,
"condition": "s1_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 1560062173,
"condition": "s1_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 1560062173,
"condition": "s1_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "gsm8k/test/1069",
"base_seed": 1560062173,
"condition": "s1_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 19683830,
"condition": "s0_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 19683830,
"condition": "s0_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 19683830,
"condition": "s0_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 19683830,
"condition": "s0_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 19683830,
"condition": "s1_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 19683830,
"condition": "s1_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 19683830,
"condition": "s1_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 19683830,
"condition": "s1_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 1560062173,
"condition": "s0_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 1560062173,
"condition": "s0_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 1560062173,
"condition": "s0_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 1560062173,
"condition": "s0_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 1560062173,
"condition": "s1_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 1560062173,
"condition": "s1_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 1560062173,
"condition": "s1_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "tnews/test/4855",
"base_seed": 1560062173,
"condition": "s1_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 19683830,
"condition": "s0_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 19683830,
"condition": "s0_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 19683830,
"condition": "s0_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 19683830,
"condition": "s0_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 19683830,
"condition": "s1_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 19683830,
"condition": "s1_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 19683830,
"condition": "s1_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 19683830,
"condition": "s1_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 1560062173,
"condition": "s0_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 1560062173,
"condition": "s0_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 1560062173,
"condition": "s0_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 1560062173,
"condition": "s0_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 1560062173,
"condition": "s1_bos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 1560062173,
"condition": "s1_eos",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 1560062173,
"condition": "s1_period",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
},
{
"source_id": "wikitext2/raw-validation/0443",
"base_seed": 1560062173,
"condition": "s1_x",
"run_seed_exact": true,
"prompt_hash_exact": true,
"generated_token_ids_exact": true,
"decoded_text_exact": true,
"eos_state_exact": true,
"truncation_state_exact": true,
"cpu_rng_pre_state_exact": true,
"cuda_rng_pre_state_exact": true,
"all_preregistered_fields_exact": true
}
],
"summary": {
"cells": 64,
"all_preregistered_fields_exact": 64,
"by_field": {
"run_seed_exact": 64,
"prompt_hash_exact": 64,
"generated_token_ids_exact": 64,
"decoded_text_exact": 64,
"eos_state_exact": 64,
"truncation_state_exact": 64,
"cpu_rng_pre_state_exact": 64,
"cuda_rng_pre_state_exact": 64
}
},
"claim_boundary": [
"Only the rerun seed subset is independently reproduced.",
"Exact replay is scoped to the pinned software and hardware contract.",
"Reproduction does not imply trajectories are seed-invariant."
],
"content_hash": "d02d0a23efdea333fb0b04fc9a889c7f81eb755f8ef4a9c66f14226a230d94a3"
}
src/data/deepseek-v2-lite-chat-sampling.json
+67139
View File
File diff suppressed because it is too large Load Diff
src/pages/deepseek/index.astro
+23 -8
View File
@@ -5,6 +5,7 @@ import DeepSeekLab from "@/components/DeepSeekLab.astro";
import DeepSeekArtifactLab from "@/components/DeepSeekArtifactLab.astro"; import DeepSeekArtifactLab from "@/components/DeepSeekArtifactLab.astro";
import DeepSeekBehaviorLab from "@/components/DeepSeekBehaviorLab.astro"; import DeepSeekBehaviorLab from "@/components/DeepSeekBehaviorLab.astro";
import DeepSeekCompletionDepthLab from "@/components/DeepSeekCompletionDepthLab.astro"; import DeepSeekCompletionDepthLab from "@/components/DeepSeekCompletionDepthLab.astro";
import DeepSeekSamplingLab from "@/components/DeepSeekSamplingLab.astro";
import { deepseekBranches, deepseekLedgers, deepseekPaperChain, deepseekWaves } from "@/data/deepseek"; import { deepseekBranches, deepseekLedgers, deepseekPaperChain, deepseekWaves } from "@/data/deepseek";
const toc = [ const toc = [
@@ -33,21 +34,22 @@ const toc = [
["22", "artifact", "真实权重执行"], ["22", "artifact", "真实权重执行"],
["23", "behavior", "Chat：最终生成行为"], ["23", "behavior", "Chat：最终生成行为"],
["24", "completion-depth", "Chat：完成度与全深度"], ["24", "completion-depth", "Chat：完成度与全深度"],
["25", "branches", "别漏掉旁支"], ["25", "sampling", "Chat：多种子采样稳健性"],
["26", "audit", "事实、推导与教学模型"], ["26", "branches", "别漏掉旁支"],
["27", "audit", "事实、推导与教学模型"],
["↳", "papers", "六十节点阅读链"], ["↳", "papers", "六十节点阅读链"],
]; ];
--- ---
<BaseLayout <BaseLayout
title="DeepSeek 技术谱系与真实权重深读：从 Dense、MoE、MLA 到 R1 与 V4" title="DeepSeek 技术谱系与真实权重深读：从 Dense、MoE、MLA 到 R1 与 V4"
description="用二十四张问题账、十次技术转向、十九个交互实验、真实 V2-Lite Base / Chat 权重、512-token 完成度评测、29 阶段隐藏状态与 26 层 MoE 路由追踪、吸收式缓存 trace 和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。" description="用二十四张问题账、十次技术转向、二十个交互实验、真实 V2-Lite Base / Chat 权重、512-token 完成度评测、29 阶段隐藏状态、26 层 MoE 路由追踪与 256 条多种子采样，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
section="deepseek" section="deepseek"
> >
<header class="page-hero deepseek-hero"> <header class="page-hero deepseek-hero">
<div class="page-hero-inner"> <div class="page-hero-inner">
<div> <div>
<p class="eyebrow"><span>SPOTLIGHT / DEEPSEEK · ROUND 05</span> COMPLETION × FULL DEPTH × REAL WEIGHTS</p> <p class="eyebrow"><span>SPOTLIGHT / DEEPSEEK · ROUND 06</span> SAMPLING × COMPLETION × FULL DEPTH</p>
<h1>不要背模型名<br />要看懂每次为什么转向</h1> <h1>不要背模型名<br />要看懂每次为什么转向</h1>
<p class="lead"> <p class="lead">
这不是七篇报告的摘要，而是一套可追问、可计算、可反驳的技术谱系： 这不是七篇报告的摘要，而是一套可追问、可计算、可反驳的技术谱系：
@@ -59,9 +61,9 @@ const toc = [
<div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div> <div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
<div><dt>LEDGERS</dt><dd>24 张问题账</dd></div> <div><dt>LEDGERS</dt><dd>24 张问题账</dd></div>
<div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div> <div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div>
<div><dt>LABS</dt><dd>19 个可操作实验</dd></div> <div><dt>LABS</dt><dd>20 个可操作实验</dd></div>
<div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div> <div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div>
<div><dt>STATUS</dt><dd>五轮 · 全 27 层执行</dd></div> <div><dt>STATUS</dt><dd>六轮 · 256 条采样</dd></div>
</dl> </dl>
</div> </div>
</header> </header>
@@ -810,8 +812,21 @@ const toc = [
<DeepSeekCompletionDepthLab /> <DeepSeekCompletionDepthLab />
</section> </section>
<section class="article-section" id="sampling">
<p class="eyebrow"><span>25</span> GREEDY IS NOT A DISTRIBUTION</p>
<h2>把单条最大概率路径打开：八个预注册 seed 下，轨迹集合是否仍然分叉</h2>
<p class="lede">
上一轮的 512-token 结果仍是 deterministic greedy：它只能看到每一步概率最大的
一条路。这一轮冻结四条 source、八格 prompt batch 与八个 SHA-256 派生 seed，
启用 checkpoint 随附的 <code>temperature=.3 / top_p=.95</code>，生成 256 条
sampled outputs。停止、任务正确、集合相似度与新进程复现继续分账；
batch-seed aligned 也明确不冒充逐行共享随机数的 paired causal design。
</p>
<DeepSeekSamplingLab />
</section>
<section class="article-section" id="branches"> <section class="article-section" id="branches">
<p class="eyebrow"><span>25</span> THE MAIN LINE IS NOT THE WHOLE TREE</p> <p class="eyebrow"><span>26</span> THE MAIN LINE IS NOT THE WHOLE TREE</p>
<h2>如果只读 V2 → V3 → R1 → V4，会漏掉五条反过来影响主线的旁支</h2> <h2>如果只读 V2 → V3 → R1 → V4，会漏掉五条反过来影响主线的旁支</h2>
<div class="branch-grid"> <div class="branch-grid">
{deepseekBranches.map(([name, line, text, url]) => ( {deepseekBranches.map(([name, line, text, url]) => (
@@ -831,7 +846,7 @@ const toc = [
</section> </section>
<section class="article-section" id="audit"> <section class="article-section" id="audit">
<p class="eyebrow"><span>26</span> EVIDENCE AUDIT</p> <p class="eyebrow"><span>27</span> EVIDENCE AUDIT</p>
<h2>同一张页面里有三种知识，它们的语气必须不同</h2> <h2>同一张页面里有三种知识，它们的语气必须不同</h2>
<div class="audit-grid"> <div class="audit-grid">
<article class="reported"> <article class="reported">
src/pages/index.astro
+5 -5
View File
@@ -145,18 +145,18 @@ const paths = [
</a> </a>
<a class="release-card deepseek-release" href="/deepseek/"> <a class="release-card deepseek-release" href="/deepseek/">
<div> <div>
<p class="eyebrow"><span>NEW / DEEPSEEK ROUND 05</span> COMPLETION · TASK TESTS · FULL DEPTH</p> <p class="eyebrow"><span>NEW / DEEPSEEK ROUND 06</span> MULTI-SEED SAMPLING · EXACT REPLAY</p>
<h2>从 Dense 到百万上下文：每次创新都在偿还上一代最贵的一张账</h2> <h2>从 Dense 到百万上下文：每次创新都在偿还上一代最贵的一张账</h2>
<p> <p>
用二十四张问题账和十次技术转向走完 Dense→V4，再把官方 V2-Lite-Chat 的 在 512-token greedy 与 29-stage / 26-gate 全深度 trace 之后，再按预注册的
128 个输出统一延长到 512-token：121 个自然 EOS、Math / Code 真实 evaluator； 8 个 seed 生成 256 条官方 nucleus samples：251 条自然 EOS、242 条不同完整
同时沿 29 个隐藏阶段与 26 个 MoE gate 追踪 1,918,176 次目标路由决定。 token 轨迹；Math 62 / 64、Code 63 / 64，并在新进程复现 R0/R1 的 64 / 64 格。
</p> </p>
</div> </div>
<dl> <dl>
<div><dt>LINEAGE</dt><dd>1991 → 2026 · 10 次转向</dd></div> <div><dt>LINEAGE</dt><dd>1991 → 2026 · 10 次转向</dd></div>
<div><dt>NODES</dt><dd>60 个一手 / 官方节点</dd></div> <div><dt>NODES</dt><dd>60 个一手 / 官方节点</dd></div>
<div><dt>LAB</dt><dd>19 · Base / Chat / full depth</dd></div> <div><dt>LAB</dt><dd>20 · Base / Chat / sampling</dd></div>
</dl> </dl>
<span class="release-arrow" aria-hidden="true">进入 DeepSeek 完整技术谱系 →</span> <span class="release-arrow" aria-hidden="true">进入 DeepSeek 完整技术谱系 →</span>
</a> </a>
src/pages/progress/index.astro
+5 -5
View File
@@ -15,7 +15,7 @@ const workstreams = [
{ label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" }, { label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
{ label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" }, { label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" },
{ label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM，加入真实去重与 mixture traces" }, { label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM，加入真实去重与 mixture traces" },
{ label: "DeepSeek 专题", value: 99, next: "扩大 completion/task 样本、sampling robustness 与干预式 mediation，再推进 SM90 FlashMLA、FP8/pipeline 与 R1-like RL" }, { label: "DeepSeek 专题", value: 99, next: "扩大 sampling 的 source/task 覆盖并推进干预式 mediation、SM90 FlashMLA、FP8/pipeline 与 R1-like RL" },
{ label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" }, { label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" }, { label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
{ label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" }, { label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
@@ -50,7 +50,7 @@ const workstreams = [
<div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div> <div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
<div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div> <div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
<div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div> <div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
<div><dt>UPDATED</dt><dd>2026-07-29 19:55 CST</dd></div> <div><dt>UPDATED</dt><dd>2026-07-30 01:50 CST</dd></div>
<div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div> <div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div>
</dl> </dl>
</div> </div>
@@ -97,12 +97,12 @@ const workstreams = [
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article> <article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article> <article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article> <article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
<article><span>✓</span><h3>八十六个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验、十三联 Base 工件实验、Chat 行为与 completion/full-depth 两轮实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article> <article><span>✓</span><h3>八十七个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验、十三联 Base 工件实验、Chat 行为、completion/full-depth 与 multi-seed sampling 三轮实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
<article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article> <article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article> <article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article> <article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
<article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article> <article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
<article><span>✓</span><h3>DeepSeek 五轮真实权重里程碑</h3><p>统一 512-token 预算让自然 EOS 从 31 / 128 增至 121 / 128；GSM8K strict exact 23 / 32、HumanEval 官方 tests pass 24 / 32，并保留四任务/域的样本边界。相同 Chat checkpoint 再执行 29-stage hidden 与 26-gate route trace；1,856 个 hidden hashes、1,664 个 route hashes 和 1,664 个 weight hashes 在新进程子集复跑中全部 exact。</p></article> <article><span>✓</span><h3>DeepSeek 六轮真实权重里程碑</h3><p>在 512-token greedy 与 29-stage / 26-gate 全深度 trace 之后，预注册 8 个 SHA-256 seed，生成 256 条 official nucleus samples：251 条 natural EOS、242 个 unique trajectory hashes；GSM8K 62 / 64 strict exact、HumanEval 63 / 64 tests pass，新进程 R0/R1 的八项合同字段 64 / 64 exact。</p></article>
<article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article> <article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
<article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article> <article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
<article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article> <article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article>
@@ -134,7 +134,7 @@ const workstreams = [
<div class="queue-table"> <div class="queue-table">
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div> <div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
<div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div> <div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div>
<div><span>P0</span><strong>DeepSeek 五轮后续</strong><p>扩大任务与语言 source → sampling robustness → 干预式 mediation → SM90 FlashMLA / FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div> <div><span>P0</span><strong>DeepSeek 六轮后续</strong><p>扩大任务与语言 source 的 sampling 覆盖 → 干预式 mediation → SM90 FlashMLA / FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
<div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div> <div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div>
<div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div> <div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div>
<div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div> <div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div>