Compare commits
5 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 wuyang
|
975ed3dce2 | ||
|
wuyang
|
f041c15e81 | ||
|
wuyang
|
65a91ae178 | ||
|
wuyang
|
a09e6b14ce | ||
|
wuyang
|
6181af90c8 |
+13
-3
@@ -14,7 +14,7 @@
|
||||
| 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
|
||||
| Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
|
||||
| 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
|
||||
| DeepSeek 专题 | 七轮实证进行中 | 99% | 扩大到可做 task-level bootstrap，加入 per-row RNG，再推进干预式 mediation、SM90 FlashMLA、FP8/pipeline 与 R1-like RL |
|
||||
| DeepSeek 专题 | 八轮实证进行中 | 99% | 推进干预式 mediation、SM90 FlashMLA、FP8/pipeline 与 R1-like RL |
|
||||
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
|
||||
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
|
||||
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
|
||||
@@ -41,7 +41,7 @@
|
||||
- [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||
- [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 二十一联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十八个原创交互视图。
|
||||
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 二十二联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十九个原创交互视图。
|
||||
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
||||
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
||||
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
||||
@@ -264,11 +264,17 @@
|
||||
- [x] 第二十一个 DeepSeek 交互实验以四页签讲解 source→seed 层级、Math/Code 4×4 task matrix、source-level 方向与均值反例、64 格八字段复现及五段 hash chain；协议、完整审计、探针、独立 evaluator、analysis 与 compact builder 全部落盘。
|
||||
- [x] Round 07 本地闸门通过：83 个 Astro 文件零诊断，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；Python 探针/评测/分析编译通过，compact builder 重建 SHA-256 不变。跨来源专项与 DeepSeek 全量真实 Chrome 回归通过，4×4 矩阵、English 反例、键盘页签、桌面/390px 移动端无溢出及零运行时异常均有自动断言。
|
||||
- [x] DeepSeek Round 07 以源提交 `9211333`、不可变镜像 `20260729T195952Z-9211333` 发布；OCI index digest `sha256:6c92883b…5584e2`，复用 NAS `12010→8080`、NPM host 31 / cert 41 与门户 `LLM ATLAS / projects / 180`。容器 healthy、0 次重启，NAS / VPS Tailscale / HTTPS / gzip / immutable assets、跨来源专项与 DeepSeek 全量生产 Chrome 回归通过；保留 `20260729T180228Z-580f696` 回滚。
|
||||
- [x] DeepSeek Round 08 在任何正式输出前冻结 HumanEval / GSM8K 各 32 条任务、四条件、T0 主分析与 T1–T3 诊断，并把共同随机数实现为 SHA-256 派生的显式 uniform tape；四个条件对同一 `u_t` 各自计算 float32 CDF，不再把 generator seed 重置冒充 CRN。
|
||||
- [x] 64 条短 smoke 的 prompt 64/64 exact、同进程重放 32/32 exact、CPU/CUDA RNG 16/16 不变；正式执行 64 tasks × T0 × 4 conditions 与 8 tasks × T1–T3 × 4 conditions，共 352 条输出，343 条 natural EOS、9 条截断、320 个完整 trajectory hashes。
|
||||
- [x] 独立 evaluator 对 256 条 T0 输出逐题评分：Code 四格 pass 为 `16/32 · 16/32 · 15/32 · 12/32`，Math 为 `19/32 · 19/32 · 17/32 · 16/32`；不把 59/128 与 71/128 写成 benchmark accuracy。
|
||||
- [x] 两域分别做 10,000 次选定任务 paired bootstrap，八个正确性 contrast 区间都跨零；长度 system-at-period 在 Code 为 `−130.875 [−178.688,−83.218]`，Math 为 `+25.125 [7.874,44.531]`，揭示任务域方向相反。
|
||||
- [x] 共同随机数对照的 256 / 256 个 contrast 全部共享相同 uniform tape；新进程重放 64 条结果，uniform hashes、token IDs、文本、停止状态与 RNG 等十二字段全部 64 / 64 exact。审计同时公开运行器在生成进程结束前已加载 gold、但 gold 没有进入 prompt/tokenizer/logits/warpers/CDF/tape/selection 的流程偏差。
|
||||
- [x] 第二十二个 DeepSeek 交互实验用五页签讲解显式 CRN sampler、选定任务 bootstrap、32 题浏览器、4×4 tape 诊断与十二字段复现/失败/偏差账；正式协议、manifest、runner、独立 evaluator、分析、复现与完整/compact 数据均已进入开源树。
|
||||
|
||||
## 正在进行
|
||||
|
||||
- [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。
|
||||
- [ ] DeepSeek 七轮下一闸门：扩大到可做 task-level bootstrap 的预注册 source 抽样框，加入 per-row RNG/common-random-number 对照与 failure taxonomy；再推进干预式 mediation、SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||
- [ ] DeepSeek 八轮下一闸门：推进干预式 mediation、SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||
- [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
|
||||
- [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
|
||||
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
|
||||
@@ -451,6 +457,10 @@
|
||||
| 2026-07-30 | 域均值必须与 source 方向同时展示 | English 句点 contrast 的均值 −8.5，但 3/4 source 为正；Code task interaction 的 +1 与 −1 在域均值 0 中抵消 |
|
||||
| 2026-07-30 | Round 07 随机分叉与复现同时过闸 | R0/R1 63/64 同格 trajectory 分叉；新进程 R0 的八项预注册字段 64/64 exact，五段 artifact hash chain 闭合 |
|
||||
| 2026-07-30 | DeepSeek Round 07 跨来源采样里程碑发布 | 源提交 `9211333`、镜像 `20260729T195952Z-9211333`、OCI `sha256:6c92883b…5584e2`；生产专项/全量 Chrome、HTTPS 与门户通过，保留 Round 06 回滚点 |
|
||||
| 2026-07-30 | Round 08 用显式 uniform tape 实现真正的共同随机数 | 64 tasks × T0 × 4 conditions，加 8 tasks × 3 诊断 tapes × 4 conditions；352 条正式输出的四条件 CDF 共享逐步 `u_t` |
|
||||
| 2026-07-30 | bootstrap 只描述冻结任务集的敏感性 | HumanEval / GSM8K 分开做 10,000 次 task-paired resamples；正确性带全跨零，不报告总体/seed CI 或 p-value |
|
||||
| 2026-07-30 | 输出长度揭示强任务域交互 | system-at-period 在 Code 为负、Math 为正，两个选定任务带都不跨零；不从长度外推能力 |
|
||||
| 2026-07-30 | Round 08 十二字段重放过闸 | 64/64 exact；uniform hash、完整 token IDs、文本、stop 与 RNG 一并进入复现合同，评分 gold 提前加载的流程偏差公开保留 |
|
||||
|
||||
## 未决问题
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读，
|
||||
语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
|
||||
以及 88 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||
以及 89 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||
8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
|
||||
第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、
|
||||
497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图
|
||||
@@ -27,8 +27,8 @@
|
||||
`sm_120a` wheel，在 RTX 5090 上完成 6/6 官方参考 exact-match 和 K3 fixed / varlen 形状计时。详见
|
||||
[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
|
||||
[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
|
||||
DeepSeek 七轮专题以 24 张问题账、10 次技术转向、
|
||||
21 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||
DeepSeek 八轮专题以 24 张问题账、10 次技术转向、
|
||||
22 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||
并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、
|
||||
MLA/HF eager cache shapes 与 `31/31` exact 独立复跑；进一步用真实 layer-1 权重执行官方 V3
|
||||
naive/absorb 路径，实际写入 576 元素 latent cache，并以 FP32 将两种结合顺序的最大误差压到
|
||||
@@ -85,6 +85,14 @@ Round 07 保持 256 条输出预算不变，改为 16 条预先冻结的 source
|
||||
`16/16 · 8/16 · 12/16 · 16/16`。句点在 11 / 16 条 source 上缩短平均长度，但
|
||||
English 的首条 source 为 `−121`、其余三条为正，证明单题均值会与多数 source
|
||||
方向相反；新进程 R0 的八项合同字段仍为 `64 / 64` exact。
|
||||
Round 08 再把覆盖扩大到 HumanEval / GSM8K 各 32 条冻结任务，并以 SHA-256 显式
|
||||
uniform tape 代替“重置同一 seed”的伪共同随机数：T0 全任务四条件 256 条、额外三条
|
||||
随机带诊断 96 条，共 352 条正式输出，343 条 natural EOS、320 个完整 trajectory hashes。
|
||||
两个域分别做 10,000 次选定任务 paired bootstrap；八个正确性区间全部跨零，但
|
||||
system-at-period 的输出长度在 Code 为 `−130.9 [−178.7,−83.2]`、在 Math 为
|
||||
`+25.1 [7.9,44.5]`，说明干预方向依赖任务域。全新进程对 64 条输出的 uniform hash、
|
||||
token IDs、文本与停止等十二字段逐一重放，`64 / 64` exact；同时公开评分金标准被运行器
|
||||
提前加载、但没有进入 prompt / logits / CDF / tape / selection 的流程偏差。
|
||||
详见
|
||||
[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md) 与
|
||||
[DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)、
|
||||
@@ -102,7 +110,9 @@ English 的首条 source 为 `−121`、其余三条为正，证明单题均值
|
||||
[DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md](./research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md) 与
|
||||
[DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md)，以及
|
||||
[DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_PROTOCOL.md](./research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_PROTOCOL.md) 与
|
||||
[DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md)。
|
||||
[DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md)，以及
|
||||
[DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md](./research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md) 与
|
||||
[DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_CRN_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_CRN_AUDIT.md)。
|
||||
其余专题按进度账本持续扩建。
|
||||
|
||||
## 本地开发
|
||||
|
||||
@@ -853,3 +853,135 @@ See `research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_PROTOCOL.md` and
|
||||
`research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md` for the
|
||||
preregistered source frame, task matrices, source-direction contrasts,
|
||||
failure identities, hash chain, exact replay scope, and non-claims.
|
||||
|
||||
## Selected-task bootstrap with explicit common random numbers
|
||||
|
||||
`v2_lite_chat_task_bootstrap_crn_probe.py` expands the task frame to all 32
|
||||
frozen HumanEval and 32 frozen GSM8K sources. The primary analysis runs one
|
||||
preregistered tape T0 on every task. Four tasks per domain additionally run
|
||||
T1-T3 as a crossed tape-sensitivity diagnostic:
|
||||
|
||||
```text
|
||||
64 tasks × T0 × 4 conditions = 256 outputs
|
||||
8 diagnostic tasks × 3 additional tapes × 4 conditions = 96 outputs
|
||||
formal total = 352 outputs
|
||||
```
|
||||
|
||||
Unlike the preceding batch-seed-aligned studies, Round 08 constructs a true
|
||||
step-index common-random-number tape. The same SHA-256-derived `u_t` is
|
||||
mapped through each condition's official `.3/.95` temperature/top-p
|
||||
distribution using a float32 token-ID-ordered inverse CDF. It does not call
|
||||
`transformers.generate` or `torch.multinomial`.
|
||||
|
||||
The frozen manifest and all 256 Chat prompt hashes must be built before any
|
||||
model outputs:
|
||||
|
||||
```bash
|
||||
node scripts/build-deepseek-chat-task-bootstrap-manifest.mjs \
|
||||
--reference \
|
||||
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||
--output \
|
||||
research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json
|
||||
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps \
|
||||
python -B scripts/freeze-deepseek-chat-task-bootstrap-prompts.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite-chat \
|
||||
--manifest \
|
||||
research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json \
|
||||
--reference-routing-json \
|
||||
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet
|
||||
```
|
||||
|
||||
The formal runner writes one hash-checked journal per source and supports
|
||||
`--resume`:
|
||||
|
||||
```bash
|
||||
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps:experiments/deepseek \
|
||||
python -B \
|
||||
experiments/deepseek/v2_lite_chat_task_bootstrap_crn_probe.py \
|
||||
--execution-mode formal \
|
||||
--artifact-dir /path/to/deepseek-v2-lite-chat \
|
||||
--manifest \
|
||||
research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json \
|
||||
--reference-routing-json \
|
||||
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--journal-dir /path/to/formal-journal \
|
||||
--resume \
|
||||
--output \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json
|
||||
```
|
||||
|
||||
Evaluation keeps T0 primary rows separate from T1-T3 diagnostics:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=experiments/deepseek \
|
||||
python -B \
|
||||
experiments/deepseek/v2_lite_chat_task_bootstrap_crn_evaluator.py \
|
||||
--sampling-json \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--sandbox-image \
|
||||
python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
|
||||
--output \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json
|
||||
```
|
||||
|
||||
The analysis resamples the 32 selected tasks 10,000 times within each domain
|
||||
while preserving condition pairing. These are selected-task resampling
|
||||
bands under T0, not benchmark-population or generation-seed confidence
|
||||
intervals:
|
||||
|
||||
```bash
|
||||
python -B \
|
||||
experiments/deepseek/v2_lite_chat_task_bootstrap_crn_analysis.py \
|
||||
--sampling-json \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json \
|
||||
--evaluation-json \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json \
|
||||
--reproduction-json \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-reproduction.json \
|
||||
--output \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-analysis.json
|
||||
```
|
||||
|
||||
The formal grid contains 343/352 natural EOS outputs and 320 unique complete
|
||||
trajectories. T0 correctness totals are Code 59/128 and Math 71/128 across
|
||||
32 tasks × four prompt conditions; they are not standard benchmark scores.
|
||||
All correctness contrast bands cross zero. The clearest length interaction
|
||||
is system-on minus system-off under the period boundary: Code is -130.875
|
||||
tokens with band [-178.688, -83.218], while Math is +25.125 with band
|
||||
[7.874, 44.531]. The preregistered 64-cell cross-process replay is exact in
|
||||
all twelve audited fields.
|
||||
|
||||
```text
|
||||
formal ea0607f2b197fac3f794655c1538ce1f9a1cb072d637eb31d35573682e311809
|
||||
eval 82b2fc5d1f854a7e0cd7aba7c70ff74d733d24221522a4f92b962478c76177ab
|
||||
replay 6519947e2fa4327c1ba2cc506b0861f172a6bdbd4f2d6787447edaf8fbcac508
|
||||
compare 63ed39e5dcdbc2a30e516172f3657dfa453ff3b23d5bd5c49c734939242a70f5
|
||||
analysis 9ab17561ced930a668c082141f7c6e013cbda70e42b09de63d41f1b82c01a6ae
|
||||
compact 082e3c56373e5f91d51d88b1430d14ad3b68b0f200e80901171c2fb62a1e57bf
|
||||
```
|
||||
|
||||
Run the complete offline hash and invariant audit with:
|
||||
|
||||
```bash
|
||||
node scripts/check-deepseek-chat-task-bootstrap-crn-data.mjs
|
||||
```
|
||||
|
||||
See `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md` and
|
||||
`research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_CRN_AUDIT.md` for the
|
||||
pre-output corrections, explicit sampler, task bootstrap, transition tables,
|
||||
multi-tape matrices, exact replay, reported gold-loading process deviation,
|
||||
and claim boundaries.
|
||||
|
||||
@@ -0,0 +1,620 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build task-paired bootstrap and multi-tape diagnostics for Round 08."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from statistics import mean, median
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-deepseek-chat-task-bootstrap-crn-v1"
|
||||
CONDITIONS = (
|
||||
"s0_eos",
|
||||
"s1_eos",
|
||||
"s0_period",
|
||||
"s1_period",
|
||||
)
|
||||
DOMAINS = ("code", "math")
|
||||
BOOTSTRAP_SEED = 1364512825
|
||||
BOOTSTRAP_RESAMPLES = 10000
|
||||
CONTRASTS = {
|
||||
"period_at_s0": ("s0_eos", "s0_period"),
|
||||
"period_at_s1": ("s1_eos", "s1_period"),
|
||||
"system_at_eos": ("s0_eos", "s1_eos"),
|
||||
"system_at_period": ("s0_period", "s1_period"),
|
||||
}
|
||||
METRICS = (
|
||||
"fixed_budget_success",
|
||||
"strict_complete_success",
|
||||
"natural_eos",
|
||||
"generated_tokens",
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--sampling-json", type=Path, required=True)
|
||||
parser.add_argument("--evaluation-json", type=Path, required=True)
|
||||
parser.add_argument("--reproduction-json", type=Path)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while chunk := handle.read(16 * 1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_hash(value: Any) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def metric_value(row: dict[str, Any], metric: str) -> float:
|
||||
evaluation = row["task_evaluation"]
|
||||
if metric == "fixed_budget_success":
|
||||
key = (
|
||||
"fixed_budget_tests_pass"
|
||||
if row["domain"] == "code"
|
||||
else "fixed_budget_numeric_exact"
|
||||
)
|
||||
return float(evaluation[key])
|
||||
if metric == "strict_complete_success":
|
||||
key = (
|
||||
"strict_complete_tests_pass"
|
||||
if row["domain"] == "code"
|
||||
else "strict_complete_numeric_exact"
|
||||
)
|
||||
return float(evaluation[key])
|
||||
if metric == "natural_eos":
|
||||
return float(row["hit_eos"])
|
||||
if metric == "generated_tokens":
|
||||
return float(row["generated_tokens"])
|
||||
raise KeyError(metric)
|
||||
|
||||
|
||||
def summarize(values: list[float]) -> dict[str, Any]:
|
||||
return {
|
||||
"count": len(values),
|
||||
"mean": mean(values) if values else None,
|
||||
"median": median(values) if values else None,
|
||||
"min": min(values) if values else None,
|
||||
"max": max(values) if values else None,
|
||||
}
|
||||
|
||||
|
||||
def percentile_band(values: np.ndarray) -> dict[str, float]:
|
||||
quantiles = np.percentile(values, [2.5, 50.0, 97.5])
|
||||
return {
|
||||
"p2_5": float(quantiles[0]),
|
||||
"p50": float(quantiles[1]),
|
||||
"p97_5": float(quantiles[2]),
|
||||
}
|
||||
|
||||
|
||||
def direction_counts(values: list[float]) -> dict[str, int]:
|
||||
epsilon = 1e-12
|
||||
return {
|
||||
"positive": sum(value > epsilon for value in values),
|
||||
"zero": sum(abs(value) <= epsilon for value in values),
|
||||
"negative": sum(value < -epsilon for value in values),
|
||||
}
|
||||
|
||||
|
||||
def success_transition(
|
||||
left: list[float],
|
||||
right: list[float],
|
||||
) -> dict[str, int]:
|
||||
pairs = [(int(a), int(b)) for a, b in zip(left, right, strict=True)]
|
||||
return {
|
||||
"fail_to_fail": sum(a == 0 and b == 0 for a, b in pairs),
|
||||
"fail_to_pass": sum(a == 0 and b == 1 for a, b in pairs),
|
||||
"pass_to_fail": sum(a == 1 and b == 0 for a, b in pairs),
|
||||
"pass_to_pass": sum(a == 1 and b == 1 for a, b in pairs),
|
||||
}
|
||||
|
||||
|
||||
def common_prefix(left: list[int], right: list[int]) -> int:
|
||||
count = 0
|
||||
for left_id, right_id in zip(left, right):
|
||||
if left_id != right_id:
|
||||
break
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
def tape_uint64(tape: str, source_id: str, step: int) -> int:
|
||||
payload = (
|
||||
f"{PROTOCOL_ID}\0uniform\0"
|
||||
f"{tape}\0{source_id}\0{step}"
|
||||
).encode()
|
||||
return int.from_bytes(hashlib.sha256(payload).digest()[:8], "big")
|
||||
|
||||
|
||||
def build_indices(
|
||||
sampling: dict[str, Any],
|
||||
evaluation: dict[str, Any],
|
||||
) -> tuple[
|
||||
dict[tuple[str, str, str], dict[str, Any]],
|
||||
dict[tuple[str, str, str], dict[str, Any]],
|
||||
]:
|
||||
sample_index = {
|
||||
(
|
||||
source["id"],
|
||||
run["tape_label"],
|
||||
output["condition"],
|
||||
): output
|
||||
for source in sampling["sources"]
|
||||
for run in source["runs"]
|
||||
for output in run["outputs"]
|
||||
}
|
||||
eval_index = {
|
||||
(
|
||||
row["source_id"],
|
||||
row["tape_label"],
|
||||
row["condition"],
|
||||
): row
|
||||
for row in evaluation["rows"]
|
||||
}
|
||||
if set(sample_index) != set(eval_index):
|
||||
raise RuntimeError("sampling/evaluation cell keys differ")
|
||||
return sample_index, eval_index
|
||||
|
||||
|
||||
def main_analysis(
|
||||
*,
|
||||
sampling: dict[str, Any],
|
||||
sample_index: dict[tuple[str, str, str], dict[str, Any]],
|
||||
eval_index: dict[tuple[str, str, str], dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
result = {}
|
||||
for domain in DOMAINS:
|
||||
sources = [
|
||||
source
|
||||
for source in sampling["sources"]
|
||||
if source["domain"] == domain
|
||||
]
|
||||
sources.sort(key=lambda row: row["within_domain_index"])
|
||||
if len(sources) != 32:
|
||||
raise RuntimeError(f"{domain}: main frame must have 32 tasks")
|
||||
source_ids = [source["id"] for source in sources]
|
||||
rng = np.random.default_rng(BOOTSTRAP_SEED)
|
||||
sampled_indices = rng.integers(
|
||||
0,
|
||||
len(source_ids),
|
||||
size=(BOOTSTRAP_RESAMPLES, len(source_ids)),
|
||||
endpoint=False,
|
||||
)
|
||||
conditions = {}
|
||||
for condition in CONDITIONS:
|
||||
rows = [
|
||||
eval_index[(source_id, "T0", condition)]
|
||||
for source_id in source_ids
|
||||
]
|
||||
conditions[condition] = {
|
||||
metric: summarize(
|
||||
[metric_value(row, metric) for row in rows]
|
||||
)
|
||||
for metric in METRICS
|
||||
} | {
|
||||
"task_outcomes": dict(
|
||||
sorted(
|
||||
Counter(row["task_outcome"] for row in rows).items()
|
||||
)
|
||||
)
|
||||
}
|
||||
contrasts = {}
|
||||
for name, (left_condition, right_condition) in CONTRASTS.items():
|
||||
metrics = {}
|
||||
for metric in METRICS:
|
||||
left = [
|
||||
metric_value(
|
||||
eval_index[(source_id, "T0", left_condition)],
|
||||
metric,
|
||||
)
|
||||
for source_id in source_ids
|
||||
]
|
||||
right = [
|
||||
metric_value(
|
||||
eval_index[(source_id, "T0", right_condition)],
|
||||
metric,
|
||||
)
|
||||
for source_id in source_ids
|
||||
]
|
||||
differences = np.asarray(right) - np.asarray(left)
|
||||
bootstrap_means = differences[
|
||||
sampled_indices
|
||||
].mean(axis=1)
|
||||
metric_result = {
|
||||
"right_minus_left_point": float(differences.mean()),
|
||||
"selected_task_resampling_band": percentile_band(
|
||||
bootstrap_means
|
||||
),
|
||||
"task_differences": summarize(
|
||||
differences.tolist()
|
||||
),
|
||||
"direction_counts": direction_counts(
|
||||
differences.tolist()
|
||||
),
|
||||
"by_source": [
|
||||
{
|
||||
"source_id": source_id,
|
||||
"domain_index": index,
|
||||
"left": float(left[index]),
|
||||
"right": float(right[index]),
|
||||
"right_minus_left": float(
|
||||
differences[index]
|
||||
),
|
||||
}
|
||||
for index, source_id in enumerate(source_ids)
|
||||
],
|
||||
}
|
||||
if metric in {
|
||||
"fixed_budget_success",
|
||||
"strict_complete_success",
|
||||
}:
|
||||
metric_result["transition"] = success_transition(
|
||||
left,
|
||||
right,
|
||||
)
|
||||
metrics[metric] = metric_result
|
||||
|
||||
trajectory_rows = []
|
||||
crn_exact = 0
|
||||
for source_id in source_ids:
|
||||
left_output = sample_index[
|
||||
(source_id, "T0", left_condition)
|
||||
]
|
||||
right_output = sample_index[
|
||||
(source_id, "T0", right_condition)
|
||||
]
|
||||
left_ids = left_output["generated_token_ids"]
|
||||
right_ids = right_output["generated_token_ids"]
|
||||
shared_steps = min(len(left_ids), len(right_ids))
|
||||
expected_left_uniforms = [
|
||||
tape_uint64("T0", source_id, step)
|
||||
for step in range(
|
||||
left_output["uniform_steps_consumed"]
|
||||
)
|
||||
]
|
||||
expected_right_uniforms = [
|
||||
tape_uint64("T0", source_id, step)
|
||||
for step in range(
|
||||
right_output["uniform_steps_consumed"]
|
||||
)
|
||||
]
|
||||
left_uniform_exact = (
|
||||
canonical_hash(expected_left_uniforms)
|
||||
== left_output["uniform_uint64_prefix_sha256"]
|
||||
)
|
||||
right_uniform_exact = (
|
||||
canonical_hash(expected_right_uniforms)
|
||||
== right_output["uniform_uint64_prefix_sha256"]
|
||||
)
|
||||
crn_exact += left_uniform_exact and right_uniform_exact
|
||||
trajectory_rows.append(
|
||||
{
|
||||
"source_id": source_id,
|
||||
"shared_active_steps": shared_steps,
|
||||
"common_prefix_tokens": common_prefix(
|
||||
left_ids,
|
||||
right_ids,
|
||||
),
|
||||
"token_ids_exact": left_ids == right_ids,
|
||||
"shared_uniform_prefix_exact": (
|
||||
left_uniform_exact and right_uniform_exact
|
||||
),
|
||||
}
|
||||
)
|
||||
contrasts[name] = {
|
||||
"left": left_condition,
|
||||
"right": right_condition,
|
||||
"metrics": metrics,
|
||||
"trajectory": {
|
||||
"sources": len(trajectory_rows),
|
||||
"shared_uniform_prefix_exact": crn_exact,
|
||||
"exact_trajectories": sum(
|
||||
row["token_ids_exact"]
|
||||
for row in trajectory_rows
|
||||
),
|
||||
"common_prefix_tokens": summarize(
|
||||
[
|
||||
float(row["common_prefix_tokens"])
|
||||
for row in trajectory_rows
|
||||
]
|
||||
),
|
||||
"rows": trajectory_rows,
|
||||
},
|
||||
}
|
||||
result[domain] = {
|
||||
"tasks": len(source_ids),
|
||||
"source_ids": source_ids,
|
||||
"tape": "T0",
|
||||
"conditions": conditions,
|
||||
"contrasts": contrasts,
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def tape_diagnostic(
|
||||
*,
|
||||
sampling: dict[str, Any],
|
||||
eval_index: dict[tuple[str, str, str], dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
result = {}
|
||||
for domain in DOMAINS:
|
||||
sources = [
|
||||
source
|
||||
for source in sampling["sources"]
|
||||
if source["domain"] == domain and len(source["runs"]) == 4
|
||||
]
|
||||
sources.sort(key=lambda row: row["within_domain_index"])
|
||||
if [row["within_domain_index"] for row in sources] != [
|
||||
0,
|
||||
8,
|
||||
16,
|
||||
24,
|
||||
]:
|
||||
raise RuntimeError(
|
||||
f"{domain}: diagnostic task indices differ"
|
||||
)
|
||||
source_ids = [row["id"] for row in sources]
|
||||
tapes = ["T0", "T1", "T2", "T3"]
|
||||
by_tape_condition = {
|
||||
tape: {
|
||||
condition: {
|
||||
metric: summarize(
|
||||
[
|
||||
metric_value(
|
||||
eval_index[
|
||||
(source_id, tape, condition)
|
||||
],
|
||||
metric,
|
||||
)
|
||||
for source_id in source_ids
|
||||
]
|
||||
)
|
||||
for metric in METRICS
|
||||
}
|
||||
for condition in CONDITIONS
|
||||
}
|
||||
for tape in tapes
|
||||
}
|
||||
contrasts = {}
|
||||
for name, (left_condition, right_condition) in CONTRASTS.items():
|
||||
metric_results = {}
|
||||
for metric in METRICS:
|
||||
matrix = np.asarray(
|
||||
[
|
||||
[
|
||||
metric_value(
|
||||
eval_index[
|
||||
(
|
||||
source_id,
|
||||
tape,
|
||||
right_condition,
|
||||
)
|
||||
],
|
||||
metric,
|
||||
)
|
||||
- metric_value(
|
||||
eval_index[
|
||||
(
|
||||
source_id,
|
||||
tape,
|
||||
left_condition,
|
||||
)
|
||||
],
|
||||
metric,
|
||||
)
|
||||
for tape in tapes
|
||||
]
|
||||
for source_id in source_ids
|
||||
],
|
||||
dtype=float,
|
||||
)
|
||||
tape_means = matrix.mean(axis=0)
|
||||
task_means = matrix.mean(axis=1)
|
||||
task_ranges_within_tape = (
|
||||
matrix.max(axis=0) - matrix.min(axis=0)
|
||||
)
|
||||
tape_ranges_within_task = (
|
||||
matrix.max(axis=1) - matrix.min(axis=1)
|
||||
)
|
||||
metric_results[metric] = {
|
||||
"matrix_task_by_tape": matrix.tolist(),
|
||||
"tape_means": {
|
||||
tape: float(tape_means[index])
|
||||
for index, tape in enumerate(tapes)
|
||||
},
|
||||
"task_means": {
|
||||
source_id: float(task_means[index])
|
||||
for index, source_id in enumerate(source_ids)
|
||||
},
|
||||
"direction_by_tape": {
|
||||
tape: direction_counts(
|
||||
matrix[:, index].tolist()
|
||||
)
|
||||
for index, tape in enumerate(tapes)
|
||||
},
|
||||
"task_range_within_tape": summarize(
|
||||
task_ranges_within_tape.tolist()
|
||||
),
|
||||
"tape_range_within_task": summarize(
|
||||
tape_ranges_within_task.tolist()
|
||||
),
|
||||
"grand_mean_descriptive": float(matrix.mean()),
|
||||
}
|
||||
contrasts[name] = {
|
||||
"left": left_condition,
|
||||
"right": right_condition,
|
||||
"metrics": metric_results,
|
||||
}
|
||||
result[domain] = {
|
||||
"tasks": len(source_ids),
|
||||
"source_ids": source_ids,
|
||||
"tapes": tapes,
|
||||
"by_tape_condition": by_tape_condition,
|
||||
"contrasts": contrasts,
|
||||
"independence_warning": (
|
||||
"The 4 tasks x 4 tapes are crossed repeated measures, "
|
||||
"not 16 independent tasks."
|
||||
),
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
for path in (args.sampling_json, args.evaluation_json):
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
sampling = json.loads(
|
||||
args.sampling_json.read_text(encoding="utf-8")
|
||||
)
|
||||
evaluation = json.loads(
|
||||
args.evaluation_json.read_text(encoding="utf-8")
|
||||
)
|
||||
if (
|
||||
sampling["protocol_id"] != PROTOCOL_ID
|
||||
or evaluation["protocol_id"] != PROTOCOL_ID
|
||||
):
|
||||
raise RuntimeError("protocol ID differs")
|
||||
if sampling["summary"]["outputs"] != 352:
|
||||
raise RuntimeError("sampling formal output count differs")
|
||||
if len(evaluation["rows"]) != 352:
|
||||
raise RuntimeError("evaluation row count differs")
|
||||
sample_index, eval_index = build_indices(sampling, evaluation)
|
||||
reproduction = None
|
||||
if args.reproduction_json is not None:
|
||||
if not args.reproduction_json.is_file():
|
||||
raise FileNotFoundError(args.reproduction_json)
|
||||
reproduction_payload = json.loads(
|
||||
args.reproduction_json.read_text(encoding="utf-8")
|
||||
)
|
||||
if reproduction_payload["protocol_id"] != PROTOCOL_ID:
|
||||
raise RuntimeError("reproduction protocol ID differs")
|
||||
reproduction = {
|
||||
"path": str(args.reproduction_json),
|
||||
"sha256": sha256_file(args.reproduction_json),
|
||||
"summary": reproduction_payload["summary"],
|
||||
}
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"inputs": {
|
||||
"sampling": {
|
||||
"path": str(args.sampling_json),
|
||||
"sha256": sha256_file(args.sampling_json),
|
||||
"content_hash": sampling["content_hash"],
|
||||
},
|
||||
"evaluation": {
|
||||
"path": str(args.evaluation_json),
|
||||
"sha256": sha256_file(args.evaluation_json),
|
||||
"content_hash": evaluation["content_hash"],
|
||||
},
|
||||
"reproduction": reproduction,
|
||||
},
|
||||
"bootstrap_contract": {
|
||||
"resamples": BOOTSTRAP_RESAMPLES,
|
||||
"seed": BOOTSTRAP_SEED,
|
||||
"rng": "numpy.random.default_rng reset per domain",
|
||||
"unit": "selected task",
|
||||
"paired_conditions": True,
|
||||
"tape": "T0",
|
||||
"interval_label": (
|
||||
"selected-task resampling band for the fixed "
|
||||
"32-task frame and T0"
|
||||
),
|
||||
"not": [
|
||||
"benchmark-population confidence interval",
|
||||
"model-ability confidence interval",
|
||||
"generation-seed uncertainty interval",
|
||||
"causal-effect confidence interval",
|
||||
],
|
||||
},
|
||||
"main_t0_selected_task_analysis": main_analysis(
|
||||
sampling=sampling,
|
||||
sample_index=sample_index,
|
||||
eval_index=eval_index,
|
||||
),
|
||||
"multi_tape_diagnostic": tape_diagnostic(
|
||||
sampling=sampling,
|
||||
eval_index=eval_index,
|
||||
),
|
||||
"claim_boundary": [
|
||||
"HumanEval and GSM8K are analyzed separately.",
|
||||
(
|
||||
"T0 selected-task bands describe only this frozen "
|
||||
"32-task frame."
|
||||
),
|
||||
(
|
||||
"T1-T3 are sensitivity diagnostics and are not pooled "
|
||||
"into the T0 primary success rates."
|
||||
),
|
||||
(
|
||||
"Common random numbers align probability quantiles; "
|
||||
"they do not force identical sampled tokens."
|
||||
),
|
||||
"Period prompts are counterfactual, not official-valid chats.",
|
||||
],
|
||||
}
|
||||
result["content_hash"] = canonical_hash(
|
||||
{
|
||||
"bootstrap_contract": result["bootstrap_contract"],
|
||||
"main": result["main_t0_selected_task_analysis"],
|
||||
"diagnostic": result["multi_tape_diagnostic"],
|
||||
}
|
||||
)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
payload = args.output.read_bytes()
|
||||
compact = {
|
||||
domain: {
|
||||
name: {
|
||||
metric: result[
|
||||
"main_t0_selected_task_analysis"
|
||||
][domain]["contrasts"][name]["metrics"][metric][
|
||||
"right_minus_left_point"
|
||||
]
|
||||
for metric in METRICS
|
||||
}
|
||||
for name in CONTRASTS
|
||||
}
|
||||
for domain in DOMAINS
|
||||
}
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||
"bytes": len(payload),
|
||||
"main_contrast_points": compact,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,364 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Evaluate Round 08 while keeping T0 and multi-tape diagnostics separate."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from statistics import mean
|
||||
from typing import Any
|
||||
|
||||
import v2_lite_chat_completion_evaluator as completion
|
||||
import v2_lite_chat_sampling_evaluator as shared
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-deepseek-chat-task-bootstrap-crn-v1"
|
||||
CONDITIONS = (
|
||||
"s0_eos",
|
||||
"s1_eos",
|
||||
"s0_period",
|
||||
"s1_period",
|
||||
)
|
||||
DOMAINS = ("code", "math")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--sampling-json", type=Path, required=True)
|
||||
parser.add_argument("--human-eval", type=Path, required=True)
|
||||
parser.add_argument("--gsm8k", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--sandbox-image",
|
||||
required=True,
|
||||
help="Pinned image reference including @sha256 digest.",
|
||||
)
|
||||
parser.add_argument("--timeout-seconds", type=float, default=5.0)
|
||||
parser.add_argument("--skip-code-execution", action="store_true")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def task_outcome(
|
||||
domain: str,
|
||||
evaluation: dict[str, Any],
|
||||
) -> str:
|
||||
if domain == "math":
|
||||
if evaluation["fixed_budget_numeric_exact"]:
|
||||
return "passed"
|
||||
if evaluation["predicted_final"] is None:
|
||||
return "no_numeric_answer"
|
||||
return "wrong_numeric_answer"
|
||||
if not evaluation["contains_entry_point_definition"]:
|
||||
return "extract_failed"
|
||||
if not evaluation["python_ast_parse"]:
|
||||
return "syntax_error"
|
||||
status = evaluation["execution"]["status"]
|
||||
return {
|
||||
"passed": "passed",
|
||||
"timeout": "timeout",
|
||||
"runtime_error": "runtime_error",
|
||||
"assertion_failed": "assertion_failed",
|
||||
"not_run": "not_run",
|
||||
}.get(status, f"execution_{status}")
|
||||
|
||||
|
||||
def success(row: dict[str, Any], *, strict: bool = False) -> int:
|
||||
evaluation = row["task_evaluation"]
|
||||
if row["domain"] == "code":
|
||||
key = (
|
||||
"strict_complete_tests_pass"
|
||||
if strict
|
||||
else "fixed_budget_tests_pass"
|
||||
)
|
||||
else:
|
||||
key = (
|
||||
"strict_complete_numeric_exact"
|
||||
if strict
|
||||
else "fixed_budget_numeric_exact"
|
||||
)
|
||||
return int(evaluation[key])
|
||||
|
||||
|
||||
def compact_summary(rows: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
result = {
|
||||
"outputs": len(rows),
|
||||
"sources": len({row["source_id"] for row in rows}),
|
||||
"tapes": sorted({row["tape_label"] for row in rows}),
|
||||
"natural_eos": sum(row["hit_eos"] for row in rows),
|
||||
"budget_truncated": sum(
|
||||
row["stopped_at_max_new_tokens"] for row in rows
|
||||
),
|
||||
"mean_generated_tokens": (
|
||||
mean(row["generated_tokens"] for row in rows)
|
||||
if rows
|
||||
else None
|
||||
),
|
||||
"fixed_budget_success": sum(success(row) for row in rows),
|
||||
"strict_complete_success": sum(
|
||||
success(row, strict=True) for row in rows
|
||||
),
|
||||
"task_outcomes": dict(
|
||||
sorted(Counter(row["task_outcome"] for row in rows).items())
|
||||
),
|
||||
"completion_classes": dict(
|
||||
sorted(
|
||||
Counter(
|
||||
row["completion_class"] for row in rows
|
||||
).items()
|
||||
)
|
||||
),
|
||||
}
|
||||
result["by_domain"] = {
|
||||
domain: compact_summary_no_recursion(
|
||||
[row for row in rows if row["domain"] == domain]
|
||||
)
|
||||
for domain in DOMAINS
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def compact_summary_no_recursion(
|
||||
rows: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"outputs": len(rows),
|
||||
"sources": len({row["source_id"] for row in rows}),
|
||||
"tapes": sorted({row["tape_label"] for row in rows}),
|
||||
"natural_eos": sum(row["hit_eos"] for row in rows),
|
||||
"budget_truncated": sum(
|
||||
row["stopped_at_max_new_tokens"] for row in rows
|
||||
),
|
||||
"mean_generated_tokens": (
|
||||
mean(row["generated_tokens"] for row in rows)
|
||||
if rows
|
||||
else None
|
||||
),
|
||||
"fixed_budget_success": sum(success(row) for row in rows),
|
||||
"strict_complete_success": sum(
|
||||
success(row, strict=True) for row in rows
|
||||
),
|
||||
"task_outcomes": dict(
|
||||
sorted(Counter(row["task_outcome"] for row in rows).items())
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
for path in (
|
||||
args.sampling_json,
|
||||
args.human_eval,
|
||||
args.gsm8k,
|
||||
):
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
if "@sha256:" not in args.sandbox_image:
|
||||
raise ValueError("--sandbox-image must include immutable digest")
|
||||
if args.timeout_seconds <= 0:
|
||||
raise ValueError("--timeout-seconds must be positive")
|
||||
sampling = json.loads(
|
||||
args.sampling_json.read_text(encoding="utf-8")
|
||||
)
|
||||
if sampling["protocol_id"] != PROTOCOL_ID:
|
||||
raise RuntimeError("sampling protocol ID differs")
|
||||
if tuple(
|
||||
sampling["seed_contract"]["condition_row_order"]
|
||||
) != CONDITIONS:
|
||||
raise RuntimeError("sampling condition order differs")
|
||||
if sampling["summary"]["outputs"] != 352:
|
||||
raise RuntimeError("formal grid must contain 352 outputs")
|
||||
|
||||
human_eval, gsm8k = completion.load_tasks(
|
||||
args.human_eval,
|
||||
args.gsm8k,
|
||||
)
|
||||
code_cache: dict[
|
||||
tuple[str, str, str],
|
||||
dict[str, Any],
|
||||
] = {}
|
||||
rows = []
|
||||
for source in sampling["sources"]:
|
||||
for run in source["runs"]:
|
||||
for output in run["outputs"]:
|
||||
if source["domain"] == "math":
|
||||
evaluation = completion.evaluate_math(
|
||||
output["text"],
|
||||
output["hit_eos"],
|
||||
gsm8k[source["id"]]["answer"],
|
||||
)
|
||||
cache_hit = False
|
||||
else:
|
||||
evaluation, cache_hit = (
|
||||
shared.evaluate_code_cached(
|
||||
text=output["text"],
|
||||
hit_eos=output["hit_eos"],
|
||||
task=human_eval[source["id"]],
|
||||
image=args.sandbox_image,
|
||||
timeout_seconds=args.timeout_seconds,
|
||||
skip_execution=args.skip_code_execution,
|
||||
cache=code_cache,
|
||||
)
|
||||
)
|
||||
row = {
|
||||
"source_id": source["id"],
|
||||
"domain": source["domain"],
|
||||
"domain_index": source["within_domain_index"],
|
||||
"replicate_index": run["replicate_index"],
|
||||
"replicate_label": run["replicate_label"],
|
||||
"tape_label": run["tape_label"],
|
||||
"base_seed": run["base_seed"],
|
||||
"run_seed": run["run_seed"],
|
||||
"condition": output["condition"],
|
||||
"generated_tokens": output["generated_tokens"],
|
||||
"hit_eos": output["hit_eos"],
|
||||
"stopped_at_max_new_tokens": output[
|
||||
"stopped_at_max_new_tokens"
|
||||
],
|
||||
"prompt_token_ids_sha256": output[
|
||||
"prompt_token_ids_sha256"
|
||||
],
|
||||
"generated_token_ids_sha256": output[
|
||||
"generated_token_ids_sha256"
|
||||
],
|
||||
"text_sha256": output["text_sha256"],
|
||||
"uniform_uint64_prefix_sha256": output[
|
||||
"uniform_uint64_prefix_sha256"
|
||||
],
|
||||
"task_evaluation": evaluation,
|
||||
"completion_class": completion.completion_class(
|
||||
output,
|
||||
evaluation,
|
||||
),
|
||||
"code_execution_cache_hit": cache_hit,
|
||||
}
|
||||
row["task_outcome"] = task_outcome(
|
||||
source["domain"],
|
||||
evaluation,
|
||||
)
|
||||
rows.append(row)
|
||||
|
||||
main_rows = [
|
||||
row for row in rows if row["tape_label"] == "T0"
|
||||
]
|
||||
diagnostic_sources = {
|
||||
source["id"]
|
||||
for source in sampling["sources"]
|
||||
if len(source["runs"]) == 4
|
||||
}
|
||||
diagnostic_rows = [
|
||||
row
|
||||
for row in rows
|
||||
if row["source_id"] in diagnostic_sources
|
||||
]
|
||||
additional_rows = [
|
||||
row for row in rows if row["tape_label"] != "T0"
|
||||
]
|
||||
if (
|
||||
len(main_rows) != 256
|
||||
or len(diagnostic_rows) != 128
|
||||
or len(additional_rows) != 96
|
||||
):
|
||||
raise RuntimeError(
|
||||
"T0 / diagnostic grid counts differ from protocol"
|
||||
)
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"input": {
|
||||
"sampling_path": str(args.sampling_json),
|
||||
"sampling_sha256": completion.sha256_file(
|
||||
args.sampling_json
|
||||
),
|
||||
"sampling_content_hash": sampling["content_hash"],
|
||||
"human_eval_sha256": completion.sha256_file(
|
||||
args.human_eval
|
||||
),
|
||||
"gsm8k_sha256": completion.sha256_file(args.gsm8k),
|
||||
"model_revision": sampling["model"]["revision"],
|
||||
"max_new_tokens": sampling["generation_contract"][
|
||||
"max_new_tokens"
|
||||
],
|
||||
},
|
||||
"sandbox": {
|
||||
"image": args.sandbox_image,
|
||||
"timeout_seconds": args.timeout_seconds,
|
||||
"code_execution_skipped": args.skip_code_execution,
|
||||
"network": "none",
|
||||
"filesystem": "read-only",
|
||||
"user": "65534:65534",
|
||||
"capabilities": "ALL dropped",
|
||||
"no_new_privileges": True,
|
||||
"memory": "256m",
|
||||
"memory_swap": "256m",
|
||||
"pids_limit": 64,
|
||||
"cpus": 0.5,
|
||||
"tmpfs": "/tmp:rw,noexec,nosuid,size=16m",
|
||||
"host_mounts": 0,
|
||||
"unique_code_cache_entries": len(code_cache),
|
||||
},
|
||||
"rows": rows,
|
||||
"summary": {
|
||||
"formal_all_outputs": compact_summary(rows),
|
||||
"main_t0": compact_summary(main_rows),
|
||||
"diagnostic_all_four_tapes": compact_summary(
|
||||
diagnostic_rows
|
||||
),
|
||||
"diagnostic_additional_t1_t3": compact_summary(
|
||||
additional_rows
|
||||
),
|
||||
"main_t0_by_domain_condition": {
|
||||
domain: {
|
||||
condition: compact_summary_no_recursion(
|
||||
[
|
||||
row
|
||||
for row in main_rows
|
||||
if row["domain"] == domain
|
||||
and row["condition"] == condition
|
||||
]
|
||||
)
|
||||
for condition in CONDITIONS
|
||||
}
|
||||
for domain in DOMAINS
|
||||
},
|
||||
},
|
||||
"claim_boundary": [
|
||||
(
|
||||
"Primary summaries use T0 only; T1-T3 are isolated "
|
||||
"multi-tape sensitivity diagnostics."
|
||||
),
|
||||
"HumanEval and GSM8K are never pooled into one ability rate.",
|
||||
(
|
||||
"The frozen 32 tasks per domain are a selected task "
|
||||
"frame, not a full benchmark sample."
|
||||
),
|
||||
"Passing HumanEval tests is functional, not safety, evidence.",
|
||||
"Counterfactual period prompts are not official-valid chats.",
|
||||
],
|
||||
}
|
||||
result["content_hash"] = completion.canonical_hash(rows)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
payload = args.output.read_bytes()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"sha256": completion.sha256_bytes(payload),
|
||||
"bytes": len(payload),
|
||||
"rows": len(rows),
|
||||
"main_t0": result["summary"]["main_t0"],
|
||||
"unique_code_cache_entries": len(code_cache),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,224 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare the preregistered 64-cell Round 08 replay with the formal grid."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-deepseek-chat-task-bootstrap-crn-v1"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--formal-json", type=Path, required=True)
|
||||
parser.add_argument("--rerun-json", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while chunk := handle.read(16 * 1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_hash(value: Any) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def run_index(
|
||||
payload: dict[str, Any],
|
||||
) -> dict[
|
||||
tuple[str, str, str],
|
||||
tuple[dict[str, Any], dict[str, Any]],
|
||||
]:
|
||||
return {
|
||||
(
|
||||
source["id"],
|
||||
run["tape_label"],
|
||||
output["condition"],
|
||||
): (run, output)
|
||||
for source in payload["sources"]
|
||||
for run in source["runs"]
|
||||
for output in run["outputs"]
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
for path in (args.formal_json, args.rerun_json):
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
formal = json.loads(
|
||||
args.formal_json.read_text(encoding="utf-8")
|
||||
)
|
||||
rerun = json.loads(
|
||||
args.rerun_json.read_text(encoding="utf-8")
|
||||
)
|
||||
if (
|
||||
formal["protocol_id"] != PROTOCOL_ID
|
||||
or rerun["protocol_id"] != PROTOCOL_ID
|
||||
):
|
||||
raise RuntimeError("protocol ID differs")
|
||||
if formal["model"]["revision"] != rerun["model"]["revision"]:
|
||||
raise RuntimeError("model revision differs")
|
||||
if formal["generation_contract"] != rerun["generation_contract"]:
|
||||
raise RuntimeError("generation contract differs")
|
||||
if rerun["summary"]["outputs"] != 64:
|
||||
raise RuntimeError("replay grid must contain 64 outputs")
|
||||
formal_rows = run_index(formal)
|
||||
rerun_rows = run_index(rerun)
|
||||
if any(key[1] != "T0" for key in rerun_rows):
|
||||
raise RuntimeError("replay grid must use T0 only")
|
||||
missing = sorted(set(rerun_rows) - set(formal_rows))
|
||||
if missing:
|
||||
raise RuntimeError(
|
||||
f"{len(missing)} replay cells are absent from formal grid"
|
||||
)
|
||||
|
||||
rows = []
|
||||
for key in sorted(rerun_rows):
|
||||
formal_run, formal_output = formal_rows[key]
|
||||
rerun_run, rerun_output = rerun_rows[key]
|
||||
checks = {
|
||||
"run_seed_exact": (
|
||||
formal_run["run_seed"] == rerun_run["run_seed"]
|
||||
),
|
||||
"prompt_hash_exact": (
|
||||
formal_output["prompt_token_ids_sha256"]
|
||||
== rerun_output["prompt_token_ids_sha256"]
|
||||
),
|
||||
"uniform_run_hash_exact": (
|
||||
formal_run["uniform_uint64_sha256"]
|
||||
== rerun_run["uniform_uint64_sha256"]
|
||||
),
|
||||
"uniform_output_prefix_hash_exact": (
|
||||
formal_output["uniform_uint64_prefix_sha256"]
|
||||
== rerun_output["uniform_uint64_prefix_sha256"]
|
||||
),
|
||||
"uniform_steps_exact": (
|
||||
formal_output["uniform_steps_consumed"]
|
||||
== rerun_output["uniform_steps_consumed"]
|
||||
),
|
||||
"generated_token_ids_exact": (
|
||||
formal_output["generated_token_ids"]
|
||||
== rerun_output["generated_token_ids"]
|
||||
),
|
||||
"decoded_text_exact": (
|
||||
formal_output["text"] == rerun_output["text"]
|
||||
),
|
||||
"eos_state_exact": (
|
||||
formal_output["hit_eos"]
|
||||
== rerun_output["hit_eos"]
|
||||
),
|
||||
"truncation_state_exact": (
|
||||
formal_output["stopped_at_max_new_tokens"]
|
||||
== rerun_output["stopped_at_max_new_tokens"]
|
||||
),
|
||||
"cpu_rng_pre_state_exact": (
|
||||
formal_run["rng_state_before"]["cpu_sha256"]
|
||||
== rerun_run["rng_state_before"]["cpu_sha256"]
|
||||
),
|
||||
"cuda_rng_pre_state_exact": (
|
||||
formal_run["rng_state_before"][
|
||||
"cuda_combined_sha256"
|
||||
]
|
||||
== rerun_run["rng_state_before"][
|
||||
"cuda_combined_sha256"
|
||||
]
|
||||
),
|
||||
"torch_rng_unchanged_exact": (
|
||||
formal_run["torch_rng_unchanged"]
|
||||
and rerun_run["torch_rng_unchanged"]
|
||||
),
|
||||
}
|
||||
rows.append(
|
||||
{
|
||||
"source_id": key[0],
|
||||
"tape_label": key[1],
|
||||
"condition": key[2],
|
||||
**checks,
|
||||
"all_preregistered_fields_exact": all(
|
||||
checks.values()
|
||||
),
|
||||
}
|
||||
)
|
||||
check_names = [
|
||||
key
|
||||
for key in rows[0]
|
||||
if key.endswith("_exact")
|
||||
and key != "all_preregistered_fields_exact"
|
||||
]
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"formal": {
|
||||
"path": str(args.formal_json),
|
||||
"sha256": sha256_file(args.formal_json),
|
||||
"content_hash": formal["content_hash"],
|
||||
},
|
||||
"rerun": {
|
||||
"path": str(args.rerun_json),
|
||||
"sha256": sha256_file(args.rerun_json),
|
||||
"content_hash": rerun["content_hash"],
|
||||
},
|
||||
"rows": rows,
|
||||
"summary": {
|
||||
"cells": len(rows),
|
||||
"all_preregistered_fields_exact": sum(
|
||||
row["all_preregistered_fields_exact"]
|
||||
for row in rows
|
||||
),
|
||||
"by_field": {
|
||||
name: sum(row[name] for row in rows)
|
||||
for name in check_names
|
||||
},
|
||||
},
|
||||
"claim_boundary": [
|
||||
"Only the preregistered 16-source T0 subset is replayed.",
|
||||
(
|
||||
"Exact replay is scoped to the pinned checkpoint, "
|
||||
"software, sampler, precision, and hardware contract."
|
||||
),
|
||||
"Reproduction does not imply tape-invariant trajectories.",
|
||||
],
|
||||
}
|
||||
if result["summary"]["cells"] != 64:
|
||||
raise RuntimeError("reproduction comparison must contain 64 cells")
|
||||
result["content_hash"] = canonical_hash(rows)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
payload = args.output.read_bytes()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||
"bytes": len(payload),
|
||||
"summary": result["summary"],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -17,6 +17,8 @@
|
||||
"build:data:deepseek-chat-completion-depth": "node scripts/build-deepseek-chat-completion-depth-compact.mjs",
|
||||
"build:data:deepseek-chat-sampling": "node scripts/build-deepseek-chat-sampling-compact.mjs",
|
||||
"build:data:deepseek-chat-cross-source-sampling": "node scripts/build-deepseek-chat-cross-source-sampling-compact.mjs",
|
||||
"build:data:deepseek-chat-task-bootstrap": "node scripts/build-deepseek-chat-task-bootstrap-crn-compact.mjs",
|
||||
"check:data:deepseek-chat-task-bootstrap": "node scripts/check-deepseek-chat-task-bootstrap-crn-data.mjs",
|
||||
"check:site": "node scripts/check-site.mjs",
|
||||
"check:moe-browser": "node scripts/check-moe-browser.mjs",
|
||||
"check:reasoning-browser": "node scripts/check-reasoning-browser.mjs",
|
||||
@@ -35,6 +37,7 @@
|
||||
"check:deepseek-browser": "node scripts/check-deepseek-browser.mjs",
|
||||
"check:deepseek-sampling-browser": "node scripts/check-deepseek-sampling-browser.mjs",
|
||||
"check:deepseek-cross-source-sampling-browser": "node scripts/check-deepseek-cross-source-sampling-browser.mjs",
|
||||
"check:deepseek-task-bootstrap-browser": "node scripts/check-deepseek-task-bootstrap-browser.mjs",
|
||||
"check:k3-browser": "node scripts/check-k3-browser.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
|
||||
@@ -0,0 +1,786 @@
|
||||
# DeepSeek-V2-Lite-Chat：32 题任务 bootstrap 与显式共同随机数审计
|
||||
|
||||
> 协议：`llm-atlas-deepseek-chat-task-bootstrap-crn-v1`
|
||||
>
|
||||
> 模型：`deepseek-ai/DeepSeek-V2-Lite-Chat`
|
||||
>
|
||||
> revision：`85864749cd611b4353ce1decdb286193298f64c7`
|
||||
>
|
||||
> 预注册协议：
|
||||
> `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md`
|
||||
>
|
||||
> 冻结清单：
|
||||
> `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json`
|
||||
>
|
||||
> 执行日期：2026-07-30
|
||||
|
||||
## 0. 先说结论
|
||||
|
||||
Round 08 把 Round 07 的“每域 4 道题 × 4 seeds”扩成：
|
||||
|
||||
```text
|
||||
主分析：
|
||||
HumanEval 32 题 × T0 × 4 conditions = 128 outputs
|
||||
GSM8K 32 题 × T0 × 4 conditions = 128 outputs
|
||||
|
||||
随机带敏感性：
|
||||
每域 4 道固定题 × T0–T3 × 4 conditions
|
||||
其中 T1–T3 新增 96 outputs
|
||||
|
||||
正式总计：
|
||||
64 tasks × 4 conditions on T0
|
||||
+ 8 diagnostic tasks × 3 extra tapes × 4 conditions
|
||||
= 352 outputs
|
||||
```
|
||||
|
||||
四个条件不再只是“同 seed 对齐”，而是在同一道题、同一随机带、同一个生成步显式读取
|
||||
完全相同的均匀数 `u_t`。`.3/.95` 的温度和 nucleus 分布来自固定 checkpoint 与
|
||||
Transformers warper，离散取样则由本协议定义的 token-ID 顺序逆 CDF 完成。
|
||||
|
||||
最稳的观察不是 correctness，而是**输出长度具有明显的 domain × prompt 条件交互**：
|
||||
|
||||
- Code：在 period 边界下，打开 system 后平均缩短 `130.875` tokens；当前 32 题、T0
|
||||
selected-task resampling band 为 `[-178.688, -83.218]`；
|
||||
- Math：同一对比反而平均延长 `25.125` tokens；对应 band 为
|
||||
`[7.874, 44.531]`；
|
||||
- EOS 边界下的 system 长度差在两个 domain 都没有得到同样清楚的非零带；
|
||||
- 所有四个预注册 correctness contrast 的 selected-task band 在 Code 和 Math 中都跨
|
||||
0，不能写成稳定能力提升或下降；
|
||||
- 64 条独立跨进程 replay 在 12 个冻结字段上全部 `64 / 64 exact`。
|
||||
|
||||
因此，本轮最合理的结论是：
|
||||
|
||||
> 同一个 system 文本的生成长度效应并不是孤立、固定方向的“system 开关效应”；它依赖
|
||||
> 历史边界 token，并且在 Code 与 Math 两类任务上方向相反。correctness 的方向则仍有
|
||||
> 明显任务与随机带不确定性。
|
||||
|
||||
---
|
||||
|
||||
## 1. 为什么 Round 07 还不够
|
||||
|
||||
Round 07 已经避免了最明显的伪样本量错误：它没有把同一道题的四个 seed 当成四道独立
|
||||
题。但每域只有 4 道题：
|
||||
|
||||
```text
|
||||
4 tasks × 4 seeds × 4 conditions
|
||||
```
|
||||
|
||||
这仍有三个缺口：
|
||||
|
||||
1. 4 道题上的方向很容易被单题主导；
|
||||
2. 同一个 batch seed 不等于四行使用同一个随机数；
|
||||
3. 任务差异与 sampling 随机带差异没有被正交地拆开。
|
||||
|
||||
Round 08 因此把预算优先给更多任务，并额外留下一个小型交叉设计：
|
||||
|
||||
```text
|
||||
32 tasks × T0 → 主 selected-task 分析
|
||||
4 tasks × T0–T3 → tape sensitivity
|
||||
```
|
||||
|
||||
`4 tasks × 4 tapes` 是 crossed repeated measures，不是 16 道独立题。
|
||||
|
||||
---
|
||||
|
||||
## 2. 预注册与预运行修订
|
||||
|
||||
### 2.1 先冻结、后生成
|
||||
|
||||
首版协议与 64 题选择清单在任何本轮模型输出产生前提交：
|
||||
|
||||
```text
|
||||
6181af9 research: preregister task bootstrap CRN protocol
|
||||
```
|
||||
|
||||
实现与 256 个 Chat prompt hash 在 smoke 前提交：
|
||||
|
||||
```text
|
||||
a09e6b1 research: freeze Chat prompts and CRN runner
|
||||
```
|
||||
|
||||
评估、重放与 bootstrap 的指标口径已经写入首版协议；对应计算脚本在 formal 运行期间实现
|
||||
并提交：
|
||||
|
||||
```text
|
||||
65a91ae research: add task bootstrap evaluation pipeline
|
||||
```
|
||||
|
||||
脚本实现没有根据 correctness 结果替换题、随机带、contrast、bootstrap seed 或主指标。
|
||||
|
||||
### 2.2 修订 1：固定四行 batch
|
||||
|
||||
最初草案写成四条 trajectory 逐条 forward。实现审查发现这会把同一模型 forward 重复四
|
||||
次，预计增加约四倍 GPU/CPU-offload 时间。任何 smoke 输出之前，合同修订为：
|
||||
|
||||
- 固定四行 batch；
|
||||
- 行序 `s0_eos, s1_eos, s0_period, s1_period`；
|
||||
- 四行第 `t` 步读取同一个 `u_t`；
|
||||
- 某行 EOS 后，后续追加 PAD=EOS、attention mask 为 0；
|
||||
- batch 内不存在跨行 attention；
|
||||
- sampling 不调用 PyTorch RNG。
|
||||
|
||||
这项修订只改变执行效率与完成行的 padding 合同，没有看结果。
|
||||
|
||||
### 2.3 修订 2：纠正 Chat prompt hash
|
||||
|
||||
tokenizer-only 闸门在模型加载前发现，首版 manifest 把路由探针的短输入 hash 错标成了
|
||||
Chat 生成 prompt hash。例如 `HumanEval/31`：
|
||||
|
||||
```text
|
||||
路由探针 s0_eos：48 tokens
|
||||
Chat 生成 s0_eos：145 tokens
|
||||
```
|
||||
|
||||
当时：
|
||||
|
||||
- 本轮模型输出为 0；
|
||||
- checkpoint 尚未加载；
|
||||
- correctness 未计算。
|
||||
|
||||
清单随后：
|
||||
|
||||
1. 保留原字段，并改名为 `routing_probe_prompt_token_ids_sha256`；
|
||||
2. 用正式 Chat renderer 重新产生 64 × 4 = 256 个 hash；
|
||||
3. 写入 `chat_generation_prompt_token_ids_sha256`；
|
||||
4. runner 只核验后者。
|
||||
|
||||
最终：
|
||||
|
||||
| 项目 | SHA-256 |
|
||||
|---|---|
|
||||
| manifest 文件 | `6313e70536c464fe598a93035576752418f08016dfd60ac246437c3b43bf2ae1` |
|
||||
| manifest 规范内容 | `d15303e18345f6dec2aaf891ba812c0a0232b86f416f8f685eabbe818249e838` |
|
||||
| 256 Chat prompt 合同 | `7f766be54463c7f513948dd6d63a55e12b85b9220ee177361c743cd9997f6c3b` |
|
||||
|
||||
---
|
||||
|
||||
## 3. source 与执行网格
|
||||
|
||||
### 3.1 source 选择
|
||||
|
||||
两个 domain 都使用前序 routing control 中已冻结的全部 32 条：
|
||||
|
||||
```text
|
||||
sample salt = llm-atlas-deepseek-routing-template-control-v1
|
||||
within_domain_index = 0..31
|
||||
```
|
||||
|
||||
没有按 Round 07 pass、EOS、长度、文本质量或 selection rank 的“好看程度”二次筛选。
|
||||
完整 source ID、source hash、selection rank、四格 prompt hash 与 tape 分配见 manifest。
|
||||
|
||||
### 3.2 主随机带与诊断随机带
|
||||
|
||||
| tape | display seed |
|
||||
|---|---:|
|
||||
| T0 | 2,572,353,518 |
|
||||
| T1 | 399,507,326 |
|
||||
| T2 | 405,284,229 |
|
||||
| T3 | 568,701,915 |
|
||||
|
||||
全部 64 题运行 T0。每域 index `0, 8, 16, 24` 再运行 T1–T3。
|
||||
|
||||
### 3.3 独立 replay 子集
|
||||
|
||||
每域固定：
|
||||
|
||||
```text
|
||||
0, 4, 8, 12, 16, 20, 24, 28
|
||||
```
|
||||
|
||||
只 replay T0 × 四条件，共：
|
||||
|
||||
```text
|
||||
16 tasks × 4 conditions = 64 cells
|
||||
```
|
||||
|
||||
这不是看完 formal 后挑的“容易复现样本”。
|
||||
|
||||
---
|
||||
|
||||
## 4. 显式共同随机数取样器
|
||||
|
||||
### 4.1 `u_t` 的定义
|
||||
|
||||
对同一 `source_id + tape + step`：
|
||||
|
||||
```text
|
||||
H = SHA256(
|
||||
protocol_id + "\0uniform\0"
|
||||
+ tape_label + "\0"
|
||||
+ source_id + "\0"
|
||||
+ decimal(step)
|
||||
)
|
||||
|
||||
z_t = H[0:8] as big-endian uint64
|
||||
u_t = (z_t + 0.5) / 2^64
|
||||
```
|
||||
|
||||
四个 condition 使用完全相同的 `z_t / u_t`。每条 output 都记录：
|
||||
|
||||
-实际消费步数；
|
||||
-消费前缀的规范 SHA-256；
|
||||
-前 8 个 uint64 hex；
|
||||
- prompt、trajectory 与文本 hash。
|
||||
|
||||
离线审计从协议重新派生全部 352 条 output 的 uniform 前缀：
|
||||
|
||||
```text
|
||||
352 / 352 exact
|
||||
```
|
||||
|
||||
### 4.2 概率分布
|
||||
|
||||
每步：
|
||||
|
||||
1. fixed checkpoint 产生 BF16 logits；
|
||||
2. `TemperatureLogitsWarper(0.3)`；
|
||||
3. `TopPLogitsWarper(0.95, min_tokens_to_keep=1)`；
|
||||
4. float32 softmax；
|
||||
5. token ID 顺序 float32 CDF；
|
||||
6. CDF 最后一个值强制为 1；
|
||||
7. `u_t` 转 float32并 clamp 到相邻可表示开区间端点；
|
||||
8. `torch.searchsorted(..., right=False)` 选 token。
|
||||
|
||||
必须区分：
|
||||
|
||||
```text
|
||||
模型与 .3/.95 概率分布：官方固定对象
|
||||
离散 sampler：本协议的显式 inverse CDF
|
||||
```
|
||||
|
||||
本轮没有调用：
|
||||
|
||||
```text
|
||||
transformers.generate
|
||||
torch.multinomial
|
||||
```
|
||||
|
||||
所以不能把 trajectory 写成“官方 Transformers sampler 的 exact 输出”。
|
||||
|
||||
### 4.3 共同随机数不等于共同 token
|
||||
|
||||
共享的是概率积分尺度上的 `u_t`。prompt 条件改变 logits 后，同一个 `u_t` 会落入不同
|
||||
token 的 CDF 区间。
|
||||
|
||||
这正是想要的配对：
|
||||
|
||||
```text
|
||||
随机冲击相同
|
||||
条件分布不同
|
||||
→ token 可以不同
|
||||
```
|
||||
|
||||
八个 domain × contrast 的主分析中：
|
||||
|
||||
```text
|
||||
32 sources × 8 domain-contrasts
|
||||
= 256 / 256 shared-uniform-prefix audits exact
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. smoke 闸门
|
||||
|
||||
smoke：
|
||||
|
||||
```text
|
||||
每域 index 0,8,16,24
|
||||
× T0/T1
|
||||
× 4 conditions
|
||||
× 16-token cap
|
||||
= 64 short outputs
|
||||
```
|
||||
|
||||
结果：
|
||||
|
||||
| 闸门 | 结果 |
|
||||
|---|---:|
|
||||
| Chat prompt hash | 256 / 256 exact |
|
||||
| short outputs | 64 / 64 |
|
||||
| 同进程 T0 replay | 32 / 32 exact |
|
||||
| T0/T1 可比格 | 32 |
|
||||
| T0/T1 trajectory 分叉 | 17 / 32 |
|
||||
| RNG state 前后不变 | 16 / 16 runs |
|
||||
| synthetic inverse-CDF test | passed |
|
||||
| OOM / NaN / exception | 0 |
|
||||
|
||||
smoke 文件：
|
||||
|
||||
```text
|
||||
/tmp/deepseek-v2-lite-chat-task-bootstrap-smoke.json
|
||||
SHA-256 c868e74f88e39343f95524b4b38d0c1503f727a22d4dfa3937532514205110b5
|
||||
```
|
||||
|
||||
smoke 不进入正式统计。
|
||||
|
||||
---
|
||||
|
||||
## 6. 正式执行账
|
||||
|
||||
### 6.1 总数
|
||||
|
||||
| 项目 | 数值 |
|
||||
|---|---:|
|
||||
| tasks | 64 |
|
||||
| source × tape runs | 88 |
|
||||
| outputs | 352 |
|
||||
| T0 主 outputs | 256 |
|
||||
| T1–T3 新增诊断 outputs | 96 |
|
||||
| natural EOS | 343 |
|
||||
| budget truncated | 9 |
|
||||
| unique trajectory hashes | 320 |
|
||||
| RNG state 前后不变 | 88 / 88 runs |
|
||||
|
||||
### 6.2 分 domain 的全部 formal outputs
|
||||
|
||||
这里含诊断题 T1–T3，只用于执行账，不是主通过率。
|
||||
|
||||
| domain | outputs | natural EOS | truncated | mean tokens |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Code | 176 | 168 | 8 | 277.278 |
|
||||
| Math | 176 | 175 | 1 | 183.188 |
|
||||
|
||||
### 6.3 资源
|
||||
|
||||
| 项目 | 数值 |
|
||||
|---|---:|
|
||||
| checkpoint load | 8.344 s |
|
||||
| formal generation sum | 4,336.699 s |
|
||||
| peak CUDA allocated | 29,919,644,672 bytes |
|
||||
| peak CUDA allocated | 27.864 GiB |
|
||||
| GPU | NVIDIA GeForce RTX 5090 |
|
||||
| dtype | BF16 |
|
||||
| CUDA resident | embedding + layers 0–23 |
|
||||
| CPU offload | layers 24–26 + norm + lm_head |
|
||||
|
||||
这不是 serving benchmark。CPU-offloaded eager latency 不能外推成吞吐。
|
||||
|
||||
---
|
||||
|
||||
## 7. 独立 evaluator
|
||||
|
||||
### 7.1 HumanEval sandbox
|
||||
|
||||
固定：
|
||||
|
||||
```text
|
||||
python:3.11-alpine
|
||||
@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4
|
||||
```
|
||||
|
||||
约束：
|
||||
|
||||
- network none；
|
||||
- read-only root filesystem；
|
||||
- user `65534:65534`；
|
||||
- drop ALL capabilities；
|
||||
- no-new-privileges；
|
||||
- 256 MiB memory / swap；
|
||||
- 64 PIDs；
|
||||
- 0.5 CPU；
|
||||
- 5 秒 timeout；
|
||||
- host mounts 0。
|
||||
|
||||
### 7.2 T0 主评估总账
|
||||
|
||||
| domain | outputs | fixed-budget pass | strict pass | natural EOS | truncated | mean tokens |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| Code | 128 | 59 | 59 | 122 | 6 | 284.227 |
|
||||
| Math | 128 | 71 | 71 | 127 | 1 | 185.969 |
|
||||
|
||||
不要把 `59 / 128` 或 `71 / 128` 当成标准 benchmark 分数。分母是：
|
||||
|
||||
```text
|
||||
32 frozen tasks × 4 prompt conditions
|
||||
```
|
||||
|
||||
同一道题在四条件中重复出现。
|
||||
|
||||
### 7.3 失败分类
|
||||
|
||||
HumanEval T0：
|
||||
|
||||
| outcome | count |
|
||||
|---|---:|
|
||||
| passed | 59 |
|
||||
| assertion failed | 47 |
|
||||
| runtime error | 16 |
|
||||
| timeout | 3 |
|
||||
| extract failed | 2 |
|
||||
| syntax error | 1 |
|
||||
|
||||
GSM8K T0：
|
||||
|
||||
| outcome | count |
|
||||
|---|---:|
|
||||
| numeric exact | 71 |
|
||||
| wrong numeric answer | 57 |
|
||||
| no numeric answer | 0 |
|
||||
|
||||
通过 HumanEval tests 只是功能证据，不是安全证据。
|
||||
|
||||
---
|
||||
|
||||
## 8. T0 四条件：先看原始格
|
||||
|
||||
### 8.1 Code
|
||||
|
||||
| condition | pass / 32 | natural EOS / 32 | mean tokens |
|
||||
|---|---:|---:|---:|
|
||||
| `s0_eos` | 16 | 30 | 330.094 |
|
||||
| `s1_eos` | 16 | 30 | 308.938 |
|
||||
| `s0_period` | 15 | 31 | 314.375 |
|
||||
| `s1_period` | 12 | 31 | 183.500 |
|
||||
|
||||
### 8.2 Math
|
||||
|
||||
| condition | pass / 32 | natural EOS / 32 | mean tokens |
|
||||
|---|---:|---:|---:|
|
||||
| `s0_eos` | 19 | 31 | 181.406 |
|
||||
| `s1_eos` | 19 | 32 | 190.969 |
|
||||
| `s0_period` | 17 | 32 | 173.188 |
|
||||
| `s1_period` | 16 | 32 | 198.313 |
|
||||
|
||||
同样的 `s1_period`，在 Code 中最短，在 Math 中却最长。这比一个全域平均值更重要。
|
||||
|
||||
---
|
||||
|
||||
## 9. selected-task bootstrap
|
||||
|
||||
### 9.1 它是什么
|
||||
|
||||
每个 domain：
|
||||
|
||||
-固定 T0；
|
||||
- 32 题有放回重采样 32 题；
|
||||
-四条件保持题级配对；
|
||||
- 10,000 次；
|
||||
- `numpy.random.default_rng`；
|
||||
- seed `1,364,512,825`；
|
||||
-报告 mean contrast 的 percentile 2.5 / 50 / 97.5%。
|
||||
|
||||
准确标签：
|
||||
|
||||
> 当前冻结 32 题、固定 T0 的 selected-task resampling band。
|
||||
|
||||
它不是：
|
||||
|
||||
-完整 benchmark population CI；
|
||||
-模型能力 CI；
|
||||
- generation-seed uncertainty CI；
|
||||
-因果效应 CI。
|
||||
|
||||
### 9.2 correctness：所有带都跨 0
|
||||
|
||||
右减左，单位为每题成功率：
|
||||
|
||||
| domain | contrast | point | selected-task band |
|
||||
|---|---|---:|---:|
|
||||
| Code | period at s0 | -0.0313 | [-0.1563, 0.0938] |
|
||||
| Code | period at s1 | -0.1250 | [-0.3438, 0.0938] |
|
||||
| Code | system at EOS | 0.0000 | [-0.1250, 0.1250] |
|
||||
| Code | system at period | -0.0938 | [-0.2500, 0.0625] |
|
||||
| Math | period at s0 | -0.0625 | [-0.2188, 0.0938] |
|
||||
| Math | period at s1 | -0.0938 | [-0.2500, 0.0625] |
|
||||
| Math | system at EOS | 0.0000 | [-0.1563, 0.1563] |
|
||||
| Math | system at period | -0.0313 | [-0.1563, 0.0938] |
|
||||
|
||||
点估计里 period 条件经常更低，但任务重采样带都覆盖 0。不能把点估计写成稳定能力损失。
|
||||
|
||||
### 9.3 correctness 转移表
|
||||
|
||||
`right` 相对 `left`：
|
||||
|
||||
| domain | contrast | fail→pass | pass→fail | both pass | both fail |
|
||||
|---|---|---:|---:|---:|---:|
|
||||
| Code | period at s0 | 2 | 3 | 13 | 14 |
|
||||
| Code | period at s1 | 4 | 8 | 8 | 12 |
|
||||
| Code | system at EOS | 2 | 2 | 14 | 14 |
|
||||
| Code | system at period | 2 | 5 | 10 | 15 |
|
||||
| Math | period at s0 | 3 | 5 | 14 | 10 |
|
||||
| Math | period at s1 | 2 | 5 | 14 | 11 |
|
||||
| Math | system at EOS | 3 | 3 | 16 | 10 |
|
||||
| Math | system at period | 2 | 3 | 14 | 13 |
|
||||
|
||||
即使平均差是 0，也可能同时有 fail→pass 与 pass→fail。例如 system at EOS 在两个 domain
|
||||
都恰好净 0，但不是每题完全不变。
|
||||
|
||||
### 9.4 输出长度
|
||||
|
||||
右减左，单位 tokens：
|
||||
|
||||
| domain | contrast | point | selected-task band |
|
||||
|---|---|---:|---:|
|
||||
| Code | period at s0 | -15.719 | [-57.627, 24.938] |
|
||||
| Code | period at s1 | -125.438 | [-165.031, -85.125] |
|
||||
| Code | system at EOS | -21.156 | [-47.846, 2.438] |
|
||||
| Code | system at period | -130.875 | [-178.688, -83.218] |
|
||||
| Math | period at s0 | -8.219 | [-27.563, 8.688] |
|
||||
| Math | period at s1 | 7.344 | [-9.407, 26.094] |
|
||||
| Math | system at EOS | 9.563 | [-4.938, 25.188] |
|
||||
| Math | system at period | 25.125 | [7.874, 44.531] |
|
||||
|
||||
两个清楚不跨 0 的 system-at-period 长度带方向相反：
|
||||
|
||||
```text
|
||||
Code → system on 更短
|
||||
Math → system on 更长
|
||||
```
|
||||
|
||||
这是一条比“system prompt 会让输出变短/变长”更精确的结论：
|
||||
|
||||
> system 的长度关联依赖 boundary 与 domain。
|
||||
|
||||
### 9.5 EOS
|
||||
|
||||
EOS contrast 很小：
|
||||
|
||||
- Code：四个 contrast 的 point 在 `0` 或 `+1/32`；
|
||||
- Math：同样只在 `0` 或 `+1/32`；
|
||||
-大部分 source 的 EOS 状态不变。
|
||||
|
||||
因此，Code 中 `-130.875` token 的 system-at-period 差异主要不是简单的“更多题命中 EOS”
|
||||
造成，而是在都自然结束的 trajectory 内结束得更早。
|
||||
|
||||
---
|
||||
|
||||
## 10. trajectory 与 first divergence
|
||||
|
||||
### 10.1 共同随机数审计
|
||||
|
||||
| domain | contrast | uniform exact | exact full trajectory | mean common-prefix tokens |
|
||||
|---|---|---:|---:|---:|
|
||||
| Code | period at s0 | 32 / 32 | 0 | 13.25 |
|
||||
| Code | period at s1 | 32 / 32 | 0 | 3.69 |
|
||||
| Code | system at EOS | 32 / 32 | 0 | 23.91 |
|
||||
| Code | system at period | 32 / 32 | 1 | 11.88 |
|
||||
| Math | period at s0 | 32 / 32 | 6 | 52.53 |
|
||||
| Math | period at s1 | 32 / 32 | 1 | 22.91 |
|
||||
| Math | system at EOS | 32 / 32 | 6 | 59.44 |
|
||||
| Math | system at period | 32 / 32 | 1 | 27.75 |
|
||||
|
||||
同一个 `u_t` 没有让四格“黏”成相同 token。特别是 Code 的 `s1` period contrast 平均只
|
||||
共享约 3.7 个前缀 token 就分叉。
|
||||
|
||||
Math 的共同前缀更长、exact trajectory 更多，说明 prompt 条件的作用方式也有 domain
|
||||
差异。
|
||||
|
||||
---
|
||||
|
||||
## 11. 四随机带诊断
|
||||
|
||||
每域固定四题，不外推为 32 题总体。
|
||||
|
||||
### 11.1 correctness contrast 的 tape means
|
||||
|
||||
右减左；每个值是 4 道诊断题的均值：
|
||||
|
||||
#### Code
|
||||
|
||||
| contrast | T0 | T1 | T2 | T3 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| period at s0 | 0.00 | 0.00 | 0.00 | 0.00 |
|
||||
| period at s1 | -0.25 | 0.00 | 0.00 | 0.00 |
|
||||
| system at EOS | 0.00 | 0.00 | 0.00 | 0.00 |
|
||||
| system at period | -0.25 | 0.00 | 0.00 | 0.00 |
|
||||
|
||||
Code 的四题诊断子集中，T0 上的两个 `-0.25` 没有在 T1–T3 重现。它支持谨慎解释主
|
||||
correctness 点估计。
|
||||
|
||||
#### Math
|
||||
|
||||
| contrast | T0 | T1 | T2 | T3 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| period at s0 | 0.00 | 0.00 | -0.25 | +0.25 |
|
||||
| period at s1 | -0.50 | -0.75 | -0.25 | -0.50 |
|
||||
| system at EOS | 0.00 | +0.50 | -0.25 | +0.25 |
|
||||
| system at period | -0.50 | -0.25 | -0.25 | -0.50 |
|
||||
|
||||
Math 四题上，方向与幅度都明显依赖 tape；四题太少，不能把这些数当成稳定概率。
|
||||
|
||||
### 11.2 长度 contrast 的 tape means
|
||||
|
||||
#### Code
|
||||
|
||||
| contrast | T0 | T1 | T2 | T3 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| period at s0 | -14.50 | +44.50 | +13.50 | -16.75 |
|
||||
| period at s1 | -108.50 | -155.00 | -21.00 | -163.25 |
|
||||
| system at EOS | -59.00 | +6.00 | -90.00 | -39.50 |
|
||||
| system at period | -153.00 | -193.50 | -124.50 | -186.00 |
|
||||
|
||||
Code 的 system-at-period 在四条 tape 上都明显为负；period-at-s1 也都为负，但 T2 幅度
|
||||
较小。
|
||||
|
||||
#### Math
|
||||
|
||||
| contrast | T0 | T1 | T2 | T3 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| period at s0 | -5.25 | +31.25 | -10.00 | +4.00 |
|
||||
| period at s1 | +0.50 | +0.25 | +3.25 | -2.50 |
|
||||
| system at EOS | -3.00 | +54.75 | -7.50 | +32.75 |
|
||||
| system at period | +2.75 | +23.75 | +5.75 | +26.25 |
|
||||
|
||||
Math 四题上的 system-at-period 四条 tape 都为正，但幅度从 `+2.75` 到 `+26.25`。
|
||||
|
||||
诊断支持长度交互的方向，但仍只覆盖 4 道题 × 4 条 tape。
|
||||
|
||||
---
|
||||
|
||||
## 12. 独立跨进程 replay
|
||||
|
||||
重放结果：
|
||||
|
||||
```text
|
||||
16 sources × T0 × 4 conditions = 64 cells
|
||||
```
|
||||
|
||||
| 字段 | exact |
|
||||
|---|---:|
|
||||
| run seed | 64 / 64 |
|
||||
| prompt hash | 64 / 64 |
|
||||
| full run uniform hash | 64 / 64 |
|
||||
| output uniform-prefix hash | 64 / 64 |
|
||||
| uniform steps | 64 / 64 |
|
||||
| generated token IDs | 64 / 64 |
|
||||
| decoded text | 64 / 64 |
|
||||
| EOS state | 64 / 64 |
|
||||
| truncation state | 64 / 64 |
|
||||
| CPU RNG pre-state | 64 / 64 |
|
||||
| CUDA RNG pre-state | 64 / 64 |
|
||||
| RNG unchanged flag | 64 / 64 |
|
||||
|
||||
总闸门：
|
||||
|
||||
```text
|
||||
64 / 64 all preregistered fields exact
|
||||
```
|
||||
|
||||
这证明固定硬件/软件/精度/执行合同下的逐 token 可重放，不证明换 GPU、PyTorch、
|
||||
Transformers、kernel 或 dtype 后仍 exact。
|
||||
|
||||
---
|
||||
|
||||
## 13. 已知偏离与限制
|
||||
|
||||
### 13.1 gold 文件打开时机偏离
|
||||
|
||||
runner 复用了前序 `behavior.task_score` 路径：
|
||||
|
||||
1. 在模型加载前读取 HumanEval / GSM8K gold；
|
||||
2. 每条文本生成结束后附加一个 narrow `task_score`；
|
||||
3. authoritative evaluator 在全部 formal 输出冻结后另行运行。
|
||||
|
||||
这不满足协议中最严格的“gold 只在生成进程结束后打开”表述。
|
||||
|
||||
为什么它不改变 trajectory：
|
||||
|
||||
- gold 不进入 message / prompt；
|
||||
- gold 不进入 tokenizer 输入；
|
||||
- gold 不进入 logits、warper、CDF 或 `u_t`；
|
||||
- gold 不改变 source、tape、max tokens 或 stop；
|
||||
- `task_score` 在 token 序列生成结束后才调用；
|
||||
-独立 evaluator 的 352 行结果不读取 runner 的 `task_score`；
|
||||
- 64-cell replay exact。
|
||||
|
||||
因此它是**真实的流程隔离偏离**，但没有已知的生成因果路径。后续 runner 应删除
|
||||
`task_score`，让生成进程完全不打开 gold。
|
||||
|
||||
### 13.2 不是完整 benchmark
|
||||
|
||||
HumanEval 只使用冻结 32 / 164 tasks；GSM8K 只使用冻结 32 / 1,319 test tasks。source
|
||||
selection 是前序 SHA 排序合同，不是完整 benchmark。
|
||||
|
||||
### 13.3 一条主随机带
|
||||
|
||||
32 题主分析只用 T0。selected-task band 不包含 generation-tape uncertainty。
|
||||
|
||||
### 13.4 多随机带只有四题
|
||||
|
||||
T0–T3 诊断只覆盖每域四题。它揭示 sensitivity，但不能稳定估计 tape 方差。
|
||||
|
||||
### 13.5 counterfactual 不可部署
|
||||
|
||||
period cell 是把一个官方 EOS boundary ID 改为普通句点 ID。它不是官方有效聊天格式。
|
||||
|
||||
### 13.6 成功不是安全
|
||||
|
||||
HumanEval pass 只说明固定 tests 通过。它不说明：
|
||||
|
||||
-安全；
|
||||
-鲁棒；
|
||||
-无未测 bug；
|
||||
-可部署。
|
||||
|
||||
---
|
||||
|
||||
## 14. 证据文件与 SHA-256
|
||||
|
||||
| 文件 | SHA-256 |
|
||||
|---|---|
|
||||
| formal sampling | `ea0607f2b197fac3f794655c1538ce1f9a1cb072d637eb31d35573682e311809` |
|
||||
| independent evaluation | `82b2fc5d1f854a7e0cd7aba7c70ff74d733d24221522a4f92b962478c76177ab` |
|
||||
| replay generation | `6519947e2fa4327c1ba2cc506b0861f172a6bdbd4f2d6787447edaf8fbcac508` |
|
||||
| reproduction comparison | `63ed39e5dcdbc2a30e516172f3657dfa453ff3b23d5bd5c49c734939242a70f5` |
|
||||
| bootstrap analysis | `9ab17561ced930a668c082141f7c6e013cbda70e42b09de63d41f1b82c01a6ae` |
|
||||
| compact website data | `082e3c56373e5f91d51d88b1430d14ad3b68b0f200e80901171c2fb62a1e57bf` |
|
||||
| frozen manifest | `6313e70536c464fe598a93035576752418f08016dfd60ac246437c3b43bf2ae1` |
|
||||
|
||||
离线一致性检查：
|
||||
|
||||
```bash
|
||||
node scripts/check-deepseek-chat-task-bootstrap-crn-data.mjs
|
||||
```
|
||||
|
||||
通过：
|
||||
|
||||
```text
|
||||
formal sources 64
|
||||
formal runs 88
|
||||
formal outputs 352
|
||||
prompt hashes exact 256
|
||||
uniform output hashes exact 352
|
||||
RNG unchanged runs 88
|
||||
evaluation rows 352
|
||||
reproduction exact cells 64 / 64
|
||||
CRN contrast checks 256
|
||||
bootstrap resamples 10,000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 15. 最终结论该怎么写
|
||||
|
||||
### 可以写
|
||||
|
||||
1. 在当前固定 32 道 HumanEval、T0 下，system-at-period 的平均输出长度比 system-off
|
||||
短约 131 tokens，selected-task band 不跨 0；
|
||||
2. 在当前固定 32 道 GSM8K、T0 下，同一 contrast 平均长约 25 tokens，band 方向相反；
|
||||
3. 两个 domain 的 correctness contrast bands 都跨 0；
|
||||
4. 每域四题的 T0–T3 诊断显示 correctness 对 tape 敏感，长度交互的方向相对更一致；
|
||||
5. 显式 CRN 与 64-cell 跨进程 replay 都通过 exact audit。
|
||||
|
||||
### 不可以写
|
||||
|
||||
1. “system prompt 普遍让代码更短、数学更长”；
|
||||
2. “period 会降低 DeepSeek 的总体能力”；
|
||||
3. “这些是完整 HumanEval / GSM8K 分数”；
|
||||
4. “selected-task band 是模型总体 95% CI”；
|
||||
5. “period 是官方聊天格式”；
|
||||
6. “通过 HumanEval tests 就安全”；
|
||||
7. “本协议 trajectory 等于官方 `torch.multinomial` trajectory”。
|
||||
|
||||
本轮真正增加的不是一个更大的数字，而是一条更干净的推理链：
|
||||
|
||||
```text
|
||||
固定任务
|
||||
→ 固定概率分布
|
||||
→ 显式共享随机冲击
|
||||
→ 条件内逐题配对
|
||||
→ 任务重采样
|
||||
→ 独立随机带诊断
|
||||
→ 跨进程逐 token 重放
|
||||
```
|
||||
|
||||
它让“prompt 边界如何改变生成行为”从单题现象，前进到一个仍有限、但可审计的任务级机制
|
||||
实验。
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,396 @@
|
||||
# DeepSeek-V2-Lite-Chat 任务 bootstrap 与显式共同随机数协议
|
||||
|
||||
> 状态：**已预注册，尚未运行本协议的任何生成输出**
|
||||
>
|
||||
> 预运行修订 1：实现审查后把“四条 trajectory 逐条执行”改为固定四行 batch；取样仍由
|
||||
> 每步显式共享的 `u_t` 完成。修订发生在任何 smoke / 正式输出之前，只为避免四倍重复
|
||||
> forward；finished row 的后续 PAD=EOS 与 mask=0 合同同时冻结在下文。
|
||||
>
|
||||
> 预运行修订 2：tokenizer-only 闸门发现首版 manifest 误把路由探针的短输入 hash 标成
|
||||
> Chat 生成 prompt hash。此时模型尚未加载、协议输出仍为 0。清单保留路由 hash，并新增
|
||||
> 由正式 Chat renderer 产生的 256 个 prompt hash；runner 只核验后者。
|
||||
>
|
||||
> 注册日期：2026-07-30
|
||||
>
|
||||
> 协议 ID：`llm-atlas-deepseek-chat-task-bootstrap-crn-v1`
|
||||
>
|
||||
> 冻结清单：
|
||||
> `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json`
|
||||
>
|
||||
> 清单文件 SHA-256：
|
||||
> `6313e70536c464fe598a93035576752418f08016dfd60ac246437c3b43bf2ae1`
|
||||
>
|
||||
> 清单规范内容 SHA-256：
|
||||
> `d15303e18345f6dec2aaf891ba812c0a0232b86f416f8f685eabbe818249e838`
|
||||
>
|
||||
> 256 个 Chat prompt 合同 hash：
|
||||
> `7f766be54463c7f513948dd6d63a55e12b85b9220ee177361c743cd9997f6c3b`
|
||||
|
||||
## 0. 一句话说明这一轮
|
||||
|
||||
Round 07 用 4 道 HumanEval、4 道 GSM8K 和每题 4 个 seed 回答了“换几道题后，现象还在
|
||||
不在”。但 4 道题仍太少，而且把同题多个随机输出当成多道题会制造伪样本量。
|
||||
|
||||
Round 08 改为：
|
||||
|
||||
```text
|
||||
主分析：
|
||||
32 HumanEval × 1 冻结随机带 × 4 条件 = 128 outputs
|
||||
32 GSM8K × 1 冻结随机带 × 4 条件 = 128 outputs
|
||||
|
||||
随机带敏感性诊断：
|
||||
每域 4 道预选题 × 额外 3 条随机带 × 4 条件 = 48 outputs / domain
|
||||
|
||||
正式总计：
|
||||
256 + 96 = 352 outputs
|
||||
```
|
||||
|
||||
它第一次允许在当前预选的 32 道题内做**按题重采样**，并且让同题四个条件在第 `t` 个
|
||||
生成步真正读取同一个均匀随机数。它仍然不是完整 HumanEval / GSM8K benchmark，不估计
|
||||
模型总体能力，也不把 counterfactual `period` 序列称为官方聊天格式。
|
||||
|
||||
---
|
||||
|
||||
## 1. 研究问题
|
||||
|
||||
### 1.1 主要问题
|
||||
|
||||
HumanEval 与 GSM8K 分开回答：
|
||||
|
||||
1. 在固定 T0 随机带下，`period - EOS` 是否改变选定任务上的通过率、自然 EOS 率和输出
|
||||
长度？
|
||||
2. `system on - off` 是否在 EOS 与 period 边界下方向一致？
|
||||
3. 任务级差异是否足以让 Round 07 的 4 题方向翻转？
|
||||
4. 同一均匀随机带下，四个条件从哪一步开始分叉；分叉前是否逐步读取完全相同的
|
||||
`u_t`？
|
||||
|
||||
### 1.2 诊断问题
|
||||
|
||||
每域固定 index `0, 8, 16, 24` 四题，额外运行 T1、T2、T3：
|
||||
|
||||
1. T0 上的任务级差异相对随机带差异有多大？
|
||||
2. 主要 contrast 的方向是否依赖单条随机带？
|
||||
3. 对同一道题，四条随机带带来的成功/失败与长度变化有多大？
|
||||
|
||||
这些是敏感性诊断，不与 32 题 T0 主分析合并成一个更大的“样本”。
|
||||
|
||||
---
|
||||
|
||||
## 2. 模型、软件与提示合同
|
||||
|
||||
冻结：
|
||||
|
||||
- checkpoint：`deepseek-ai/DeepSeek-V2-Lite-Chat`；
|
||||
- revision：`85864749cd611b4353ce1decdb286193298f64c7`；
|
||||
- checkpoint 身份：SFT Chat；
|
||||
- Transformers `4.41.2`；
|
||||
- PyTorch `2.11.0+cu128`；
|
||||
- BF16、官方 remote modeling code、eager attention；
|
||||
- embedding + layers 0–23 放 CUDA；
|
||||
- layers 24–26、final norm 与 `lm_head` 放 CPU；
|
||||
- `device_map=auto`，GPU budget `28GiB`，CPU budget `80GiB`；
|
||||
- `use_cache=true`，每条 trajectory 的 `max_new_tokens=512`；
|
||||
- BOS `100000`，EOS `100001`，PAD alias EOS。
|
||||
|
||||
固定四条件与行序：
|
||||
|
||||
```text
|
||||
s0_eos, s1_eos, s0_period, s1_period
|
||||
```
|
||||
|
||||
- `s0/s1`：system off/on；
|
||||
- `eos`：官方历史 assistant 边界；
|
||||
- `period`：把该位置的单个 EOS ID 换成 ordinary period ID `13`；
|
||||
- 其余 prompt token 与前序冻结模板相同；
|
||||
- 每条 prompt token hash 必须与冻结清单逐格 exact。
|
||||
|
||||
正式生成使用固定四行 batch，行序即上述条件序。batch 内没有跨行 attention；取样不调用
|
||||
PyTorch RNG。同一行结束后，后续步追加 PAD=EOS 且该位置 attention mask 为 0，其他仍活跃
|
||||
行继续 forward。这个合同避免四倍重复 forward，也把执行形状固定下来。
|
||||
|
||||
---
|
||||
|
||||
## 3. 任务如何在结果前冻结
|
||||
|
||||
输入来源：
|
||||
|
||||
```text
|
||||
src/data/deepseek-v2-lite-routing-special-token-family-control.json
|
||||
SHA-256 c372c1b03a8b15f615b54ded5d9257a8fc2cdb7728001735d3d4c1d8534af5bf
|
||||
sample salt llm-atlas-deepseek-routing-template-control-v1
|
||||
```
|
||||
|
||||
每个领域使用其中按 `within_domain_index=0..31` 已冻结的全部 32 条，不根据 Round 07
|
||||
正确率、输出长度、EOS 或文本质量二次筛选。完整 source ID、selection rank、源文本 hash、
|
||||
四格 prompt hash、随机带分配和重放分配均在冻结清单内。
|
||||
|
||||
### 3.1 HumanEval
|
||||
|
||||
```text
|
||||
00 HumanEval/31 01 HumanEval/44 02 HumanEval/133 03 HumanEval/23
|
||||
04 HumanEval/147 05 HumanEval/34 06 HumanEval/123 07 HumanEval/77
|
||||
08 HumanEval/22 09 HumanEval/127 10 HumanEval/37 11 HumanEval/48
|
||||
12 HumanEval/158 13 HumanEval/70 14 HumanEval/89 15 HumanEval/9
|
||||
16 HumanEval/53 17 HumanEval/145 18 HumanEval/35 19 HumanEval/103
|
||||
20 HumanEval/25 21 HumanEval/98 22 HumanEval/115 23 HumanEval/10
|
||||
24 HumanEval/141 25 HumanEval/91 26 HumanEval/122 27 HumanEval/75
|
||||
28 HumanEval/134 29 HumanEval/142 30 HumanEval/149 31 HumanEval/102
|
||||
```
|
||||
|
||||
### 3.2 GSM8K
|
||||
|
||||
```text
|
||||
00 1069 01 1228 02 0144 03 1251 04 0676 05 0126 06 0438 07 1299
|
||||
08 0546 09 0537 10 0226 11 1128 12 0760 13 1173 14 0452 15 0357
|
||||
16 1050 17 0951 18 0906 19 0092 20 0372 21 0361 22 0733 23 0687
|
||||
24 0584 25 1152 26 1021 27 1194 28 0399 29 0214 30 1084 31 0964
|
||||
```
|
||||
|
||||
GSM8K 行均指 `gsm8k/test/NNNN`。
|
||||
|
||||
gold 只进入生成后的 evaluator：
|
||||
|
||||
| 输入 | SHA-256 |
|
||||
|---|---|
|
||||
| HumanEval JSONL.gz | `b796127e635a67f93fb35c04f4cb03cf06f38c8072ee7cee8833d7bee06979ef` |
|
||||
| GSM8K test JSONL | `3730d312f6e3440559ace48831e51066acaca737f6eabec99bccb9e4b3c39d14` |
|
||||
|
||||
---
|
||||
|
||||
## 4. 真正的共同随机数如何构造
|
||||
|
||||
### 4.1 为什么不再只重置 seed
|
||||
|
||||
Transformers `generate()` 的常规 sampling 最终调用 `torch.multinomial`。即使四行共享 batch
|
||||
seed，每行也消费不同 RNG 子流；如果逐行重置同 seed，分布内部的实现细节仍不等于我们
|
||||
明确定义并能逐步审计的共同随机数合同。
|
||||
|
||||
本轮不调用 `torch.multinomial`，也不把输出称作“Transformers 官方 sampler 的 exact
|
||||
trajectory”。模型、logits 和官方 `.3/.95` 解码分布仍来自固定 checkpoint；离散取样器改为
|
||||
本协议定义的**显式均匀数 + token-ID 顺序逆 CDF**。
|
||||
|
||||
### 4.2 冻结随机带
|
||||
|
||||
四条带 T0–T3 的 display seed：
|
||||
|
||||
| 带 | display seed | 派生字符串 SHA-256 |
|
||||
|---|---:|---|
|
||||
| T0 | 2,572,353,518 | `9952ffeec840649d2654d35b9a86c97db3737b70826c366e3f5a7633ec26d0c4` |
|
||||
| T1 | 399,507,326 | `17cfff7e5b3b6d011f4d1b8076637d4bdfb0ebd8a6021beb2a0011a8f6d520c6` |
|
||||
| T2 | 405,284,229 | `1828258525ea1b450d6de0ba0d9ff12f737b048b1cef554d7dd2ba90d25e5f4d` |
|
||||
| T3 | 568,701,915 | `21e5b3dbe506f1b7e9d483318195fc5cb4990c518850a52d40b5eff225491bd2` |
|
||||
|
||||
对 `tape_label, source_id, step`：
|
||||
|
||||
```text
|
||||
H = SHA256(
|
||||
protocol_id + "\0uniform\0"
|
||||
+ tape_label + "\0"
|
||||
+ source_id + "\0"
|
||||
+ decimal(step)
|
||||
)
|
||||
|
||||
z_t = H[0:8]，big-endian unsigned uint64
|
||||
u_t = (z_t + 0.5) / 2^64
|
||||
```
|
||||
|
||||
`u_t` 位于开区间 `(0,1)`。同一 source + tape 的四个条件在相同生成步 `t` 使用完全相同
|
||||
的 `z_t`。某条件 EOS 后停止，不再产生“配对输出”；其他仍存活条件继续读取下一步相同
|
||||
定义的随机带。
|
||||
|
||||
### 4.3 每一步的概率与取样
|
||||
|
||||
对模型最后位置 logits：
|
||||
|
||||
1. 用 Transformers `TemperatureLogitsWarper(0.3)`；
|
||||
2. 用 Transformers `TopPLogitsWarper(0.95, min_tokens_to_keep=1)`；
|
||||
3. 以 `torch.float32` 做 softmax；
|
||||
4. 按 token ID `0..vocab-1` 做 `torch.float32` cumulative sum；
|
||||
5. 强制最后一个 CDF 元素为 `1.0`；
|
||||
6. 将 `u_t` cast 为 `torch.float32`；
|
||||
7. float32 转换若落在端点，则 clamp 到
|
||||
`torch.nextafter(0,1)` / `torch.nextafter(1,0)`；
|
||||
8. `torch.searchsorted(cdf, u_t, right=False)` 取得 token ID。
|
||||
|
||||
因此：
|
||||
|
||||
```text
|
||||
共享的是概率积分尺度上的 u_t
|
||||
≠ 强迫四格选择同一 token
|
||||
```
|
||||
|
||||
prompt 改变 logits 后，同一个 `u_t` 可以落入不同 token 区间，这正是共同随机数配对所需
|
||||
的性质。
|
||||
|
||||
每条 output 记录：
|
||||
|
||||
- 前 8 个 `z_t` / `u_t`；
|
||||
-实际消费的 `z_t` 序列规范 hash；
|
||||
- prompt hash、generated token IDs 与 hash；
|
||||
- EOS / truncation；
|
||||
-生成前后 CPU/CUDA RNG state hash；
|
||||
- PyTorch RNG 是否完全未被取样器消费。
|
||||
|
||||
---
|
||||
|
||||
## 5. 执行网格、smoke 与独立重放
|
||||
|
||||
### 5.1 16-token smoke
|
||||
|
||||
每域 index `0,8,16,24`，T0/T1，四条件：
|
||||
|
||||
```text
|
||||
8 sources × 2 tapes × 4 conditions = 64 short outputs
|
||||
```
|
||||
|
||||
通过闸门：
|
||||
|
||||
1. 正式 64 题 × 4 条件的 256 个 prompt hash 全部与清单 exact；
|
||||
2. 64 条短输出无 OOM、NaN、exception；
|
||||
3. 同进程重放 8 sources × T0 × 4 = 32 cells 逐 token、文本与 stop state exact；
|
||||
4. T0/T1 至少一个 source-condition trajectory 分叉；
|
||||
5. 同一 source+tape 四条件的前 16 个 `z_t` 定义逐步 exact；
|
||||
6. 生成前后 PyTorch CPU/CUDA RNG state exact，证明 sampling 不消费隐式 RNG；
|
||||
7. synthetic distribution 单元测试覆盖 CDF 边界与被 top-p 过滤 token。
|
||||
|
||||
smoke 不进入正式统计。
|
||||
|
||||
### 5.2 正式运行
|
||||
|
||||
- 全部 64 题跑 T0 × 四条件：256 outputs；
|
||||
- 每域 index `0,8,16,24` 再跑 T1/T2/T3 × 四条件：96 outputs；
|
||||
- 合计 352 outputs；
|
||||
- 每题结束后写独立 journal；再次启动时必须核验协议、source、prompt 与输出内容 hash 后才
|
||||
能 resume；
|
||||
- journal 只用于可恢复执行，正式 JSON 由全部合格 journal 组装。
|
||||
- 四格固定同 batch；finished row 在后续步追加 PAD=EOS、对应 attention mask 为 0。
|
||||
|
||||
### 5.3 跨进程独立重放
|
||||
|
||||
每域 index：
|
||||
|
||||
```text
|
||||
0, 4, 8, 12, 16, 20, 24, 28
|
||||
```
|
||||
|
||||
只跑 T0 × 四条件：
|
||||
|
||||
```text
|
||||
16 sources × 1 tape × 4 conditions = 64 outputs
|
||||
```
|
||||
|
||||
逐格比较 protocol、model revision、generation contract、tape、prompt hash、uniform hash、
|
||||
generated token IDs、decoded text、EOS、truncation 和生成前 RNG state。目标为 64/64
|
||||
全部 exact。
|
||||
|
||||
---
|
||||
|
||||
## 6. evaluator 在输出前冻结
|
||||
|
||||
### 6.1 HumanEval
|
||||
|
||||
沿用前序四本账：
|
||||
|
||||
1. 代码抽取是否成功；
|
||||
2. Python AST 是否可解析；
|
||||
3. sandbox 是否真正执行；
|
||||
4. tests 是否通过。
|
||||
|
||||
固定 sandbox：
|
||||
|
||||
```text
|
||||
python:3.11-alpine
|
||||
@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4
|
||||
```
|
||||
|
||||
仍为 network none、read-only root filesystem、非 root、drop ALL capabilities、
|
||||
no-new-privileges、256 MiB、64 PIDs、0.5 CPU、5 秒 timeout。报告失败类型：
|
||||
|
||||
```text
|
||||
extract_failed / syntax_error / timeout / runtime_error / assertion_failed / passed
|
||||
```
|
||||
|
||||
主 correctness 是 fixed-budget tests pass；strict-complete pass 单独报告，不混写。
|
||||
|
||||
### 6.2 GSM8K
|
||||
|
||||
沿用冻结 numeric final-answer evaluator：
|
||||
|
||||
- gold：`####` 后最后一个规范化数字；
|
||||
- prediction：输出中最后一个规范化数字；
|
||||
- fixed-budget numeric exact；
|
||||
- strict-complete numeric exact；
|
||||
- 无数字单独记为 `no_numeric_answer`；
|
||||
- 有数字但不等于 gold 记为 `wrong_numeric_answer`。
|
||||
|
||||
---
|
||||
|
||||
## 7. 统计合同
|
||||
|
||||
### 7.1 主要 contrast
|
||||
|
||||
在每个 domain 内、每道题、T0 上计算右减左：
|
||||
|
||||
```text
|
||||
period_at_s0 = s0_period - s0_eos
|
||||
period_at_s1 = s1_period - s1_eos
|
||||
system_at_eos = s1_eos - s0_eos
|
||||
system_at_period = s1_period - s0_period
|
||||
```
|
||||
|
||||
success 用 `{-1,0,+1}` 任务级转移；长度与 EOS 保留配对差。两因子 main effect 与
|
||||
interaction 可以作为 secondary descriptive summary，但不能替代四个预注册 contrast。
|
||||
|
||||
### 7.2 任务 bootstrap
|
||||
|
||||
HumanEval 32 题与 GSM8K 32 题分别进行：
|
||||
|
||||
- 10,000 次；
|
||||
- 每次在当前 domain 的 32 题中有放回抽 32 题；
|
||||
- 同一次 resample 对四条件保持题级配对；
|
||||
- 固定 seed `1,364,512,825`；
|
||||
- 报告 percentile 2.5% / 50% / 97.5%。
|
||||
|
||||
必须写成：
|
||||
|
||||
> 当前冻结 32 题、固定 T0 下的 selected-task resampling band。
|
||||
|
||||
不得写成：
|
||||
|
||||
- 完整 benchmark 总体 95% CI；
|
||||
- 模型真实能力 CI；
|
||||
- 同时覆盖 generation-seed uncertainty 的 CI；
|
||||
- 因果效应 CI。
|
||||
|
||||
### 7.3 多随机带诊断
|
||||
|
||||
对每域固定四题 × T0–T3：
|
||||
|
||||
- 分别报告每条带的 condition success / EOS / length；
|
||||
- 计算题内跨带 range 与带内跨题 range；
|
||||
- 分别做 selected-task 和 selected-tape 描述；
|
||||
- 不把 `4 tasks × 4 tapes = 16` 当成 16 道独立题。
|
||||
|
||||
---
|
||||
|
||||
## 8. 结论边界
|
||||
|
||||
允许：
|
||||
|
||||
- 描述这 32 道预选题、T0 下四个条件的配对差；
|
||||
- 描述任务重采样带与 8 道诊断题上的 tape sensitivity；
|
||||
- 描述显式共同随机数下的 first-divergence、转移表和长度差；
|
||||
- 报告 exact replay 是否通过。
|
||||
|
||||
禁止：
|
||||
|
||||
- 把 period 当成官方可部署格式；
|
||||
- 把 selected-task band 外推为 benchmark / 用户分布总体；
|
||||
- 把一条 tape 的结果写成 sampling robustness；
|
||||
- 把测试通过写成代码安全；
|
||||
- 把同一 `u_t` 错写成“同一 token”；
|
||||
- 把本协议 sampler 写成 Transformers `generate()` / `torch.multinomial` 的 exact
|
||||
trajectory；
|
||||
- 根据正式输出修改 source、tape、contrast、bootstrap seed、sandbox 或主指标。
|
||||
|
||||
任何偏离必须在结果审计中逐项列出；不能静默修订本文件。
|
||||
@@ -0,0 +1,347 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import process from "node:process";
|
||||
|
||||
const CONDITIONS = ["s0_eos", "s1_eos", "s0_period", "s1_period"];
|
||||
const DOMAINS = ["code", "math"];
|
||||
const CONTRASTS = [
|
||||
"period_at_s0",
|
||||
"period_at_s1",
|
||||
"system_at_eos",
|
||||
"system_at_period",
|
||||
];
|
||||
const PATHS = {
|
||||
sampling: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json",
|
||||
evaluation: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json",
|
||||
reproduction: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-reproduction.json",
|
||||
analysis: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-analysis.json",
|
||||
output: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-compact.json",
|
||||
};
|
||||
|
||||
function sha256(value) {
|
||||
return createHash("sha256").update(value).digest("hex");
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const result = { ...PATHS };
|
||||
for (let index = 2; index < argv.length; index += 2) {
|
||||
const key = argv[index]?.replace(/^--/, "");
|
||||
const value = argv[index + 1];
|
||||
if (!(key in result) || value === undefined) {
|
||||
throw new Error(`Unknown or incomplete argument: ${argv[index]}`);
|
||||
}
|
||||
result[key] = value;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function success(row) {
|
||||
const evaluation = row.task_evaluation;
|
||||
return row.domain === "code"
|
||||
? evaluation.fixed_budget_tests_pass
|
||||
: evaluation.fixed_budget_numeric_exact;
|
||||
}
|
||||
|
||||
const paths = parseArgs(process.argv);
|
||||
const inputBytes = {};
|
||||
const input = {};
|
||||
for (const name of ["sampling", "evaluation", "reproduction", "analysis"]) {
|
||||
inputBytes[name] = await readFile(paths[name]);
|
||||
input[name] = JSON.parse(inputBytes[name]);
|
||||
}
|
||||
const { sampling, evaluation, reproduction, analysis } = input;
|
||||
const evaluationIndex = new Map(
|
||||
evaluation.rows.map((row) => [
|
||||
`${row.source_id}\0${row.tape_label}\0${row.condition}`,
|
||||
row,
|
||||
]),
|
||||
);
|
||||
const sampleIndex = new Map(
|
||||
sampling.sources.flatMap((source) =>
|
||||
source.runs.flatMap((run) =>
|
||||
run.outputs.map((output) => [
|
||||
`${source.id}\0${run.tape_label}\0${output.condition}`,
|
||||
output,
|
||||
]),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
const conditionTable = Object.fromEntries(
|
||||
DOMAINS.map((domain) => [
|
||||
domain,
|
||||
CONDITIONS.map((condition) => {
|
||||
const summary =
|
||||
evaluation.summary.main_t0_by_domain_condition[domain][condition];
|
||||
return {
|
||||
condition,
|
||||
outputs: summary.outputs,
|
||||
success: summary.fixed_budget_success,
|
||||
successRate: summary.fixed_budget_success / summary.outputs,
|
||||
naturalEos: summary.natural_eos,
|
||||
naturalEosRate: summary.natural_eos / summary.outputs,
|
||||
meanTokens: summary.mean_generated_tokens,
|
||||
outcomes: summary.task_outcomes,
|
||||
};
|
||||
}),
|
||||
]),
|
||||
);
|
||||
|
||||
const contrasts = Object.fromEntries(
|
||||
DOMAINS.map((domain) => [
|
||||
domain,
|
||||
Object.fromEntries(
|
||||
CONTRASTS.map((name) => {
|
||||
const row =
|
||||
analysis.main_t0_selected_task_analysis[domain].contrasts[name];
|
||||
const metrics = Object.fromEntries(
|
||||
Object.entries(row.metrics).map(([metric, value]) => [
|
||||
metric,
|
||||
{
|
||||
point: value.right_minus_left_point,
|
||||
band: value.selected_task_resampling_band,
|
||||
directions: value.direction_counts,
|
||||
transition: value.transition ?? null,
|
||||
},
|
||||
]),
|
||||
);
|
||||
return [
|
||||
name,
|
||||
{
|
||||
left: row.left,
|
||||
right: row.right,
|
||||
metrics,
|
||||
trajectory: {
|
||||
sources: row.trajectory.sources,
|
||||
sharedUniformExact:
|
||||
row.trajectory.shared_uniform_prefix_exact,
|
||||
exactTrajectories: row.trajectory.exact_trajectories,
|
||||
commonPrefixTokens: row.trajectory.common_prefix_tokens,
|
||||
},
|
||||
},
|
||||
];
|
||||
}),
|
||||
),
|
||||
]),
|
||||
);
|
||||
|
||||
const diagnostic = Object.fromEntries(
|
||||
DOMAINS.map((domain) => [
|
||||
domain,
|
||||
{
|
||||
sourceIds: analysis.multi_tape_diagnostic[domain].source_ids,
|
||||
tapes: analysis.multi_tape_diagnostic[domain].tapes,
|
||||
contrasts: Object.fromEntries(
|
||||
CONTRASTS.map((name) => {
|
||||
const row =
|
||||
analysis.multi_tape_diagnostic[domain].contrasts[name];
|
||||
return [
|
||||
name,
|
||||
{
|
||||
left: row.left,
|
||||
right: row.right,
|
||||
success: {
|
||||
matrix:
|
||||
row.metrics.fixed_budget_success.matrix_task_by_tape,
|
||||
tapeMeans:
|
||||
row.metrics.fixed_budget_success.tape_means,
|
||||
taskRangeWithinTape:
|
||||
row.metrics.fixed_budget_success.task_range_within_tape,
|
||||
tapeRangeWithinTask:
|
||||
row.metrics.fixed_budget_success.tape_range_within_task,
|
||||
},
|
||||
tokens: {
|
||||
matrix: row.metrics.generated_tokens.matrix_task_by_tape,
|
||||
tapeMeans: row.metrics.generated_tokens.tape_means,
|
||||
taskRangeWithinTape:
|
||||
row.metrics.generated_tokens.task_range_within_tape,
|
||||
tapeRangeWithinTask:
|
||||
row.metrics.generated_tokens.tape_range_within_task,
|
||||
},
|
||||
},
|
||||
];
|
||||
}),
|
||||
),
|
||||
},
|
||||
]),
|
||||
);
|
||||
|
||||
const tasks = Object.fromEntries(
|
||||
DOMAINS.map((domain) => {
|
||||
const domainSources = sampling.sources
|
||||
.filter((source) => source.domain === domain)
|
||||
.sort((left, right) => left.within_domain_index - right.within_domain_index);
|
||||
return [
|
||||
domain,
|
||||
domainSources.map((source) => {
|
||||
const conditions = Object.fromEntries(
|
||||
CONDITIONS.map((condition) => {
|
||||
const evaluationRow = evaluationIndex.get(
|
||||
`${source.id}\0T0\0${condition}`,
|
||||
);
|
||||
const samplingRow = sampleIndex.get(
|
||||
`${source.id}\0T0\0${condition}`,
|
||||
);
|
||||
return [
|
||||
condition,
|
||||
{
|
||||
success: success(evaluationRow),
|
||||
outcome: evaluationRow.task_outcome,
|
||||
tokens: evaluationRow.generated_tokens,
|
||||
naturalEos: evaluationRow.hit_eos,
|
||||
truncated: evaluationRow.stopped_at_max_new_tokens,
|
||||
trajectoryHash:
|
||||
evaluationRow.generated_token_ids_sha256.slice(0, 12),
|
||||
firstTokenIds: samplingRow.generated_token_ids.slice(0, 8),
|
||||
},
|
||||
];
|
||||
}),
|
||||
);
|
||||
const taskContrasts = Object.fromEntries(
|
||||
CONTRASTS.map((name) => {
|
||||
const sourceRow =
|
||||
analysis.main_t0_selected_task_analysis[domain].contrasts[name];
|
||||
const successRow =
|
||||
sourceRow.metrics.fixed_budget_success.by_source.find(
|
||||
(row) => row.source_id === source.id,
|
||||
);
|
||||
const tokenRow = sourceRow.metrics.generated_tokens.by_source.find(
|
||||
(row) => row.source_id === source.id,
|
||||
);
|
||||
const trajectoryRow = sourceRow.trajectory.rows.find(
|
||||
(row) => row.source_id === source.id,
|
||||
);
|
||||
return [
|
||||
name,
|
||||
{
|
||||
successDelta: successRow.right_minus_left,
|
||||
tokenDelta: tokenRow.right_minus_left,
|
||||
commonPrefixTokens: trajectoryRow.common_prefix_tokens,
|
||||
exactTrajectory: trajectoryRow.token_ids_exact,
|
||||
},
|
||||
];
|
||||
}),
|
||||
);
|
||||
return {
|
||||
id: source.id,
|
||||
index: source.within_domain_index,
|
||||
conditions,
|
||||
contrasts: taskContrasts,
|
||||
diagnosticTapes: source.runs.map((run) => run.tape_label),
|
||||
};
|
||||
}),
|
||||
];
|
||||
}),
|
||||
);
|
||||
|
||||
const exampleSource = sampling.sources.find(
|
||||
(source) => source.id === "HumanEval/31",
|
||||
);
|
||||
const exampleRun = exampleSource.runs.find((run) => run.tape_label === "T0");
|
||||
const compact = {
|
||||
schemaVersion: 1,
|
||||
protocolId: sampling.protocol_id,
|
||||
capturedAt: sampling.captured_at,
|
||||
model: {
|
||||
repo: sampling.model.repo,
|
||||
revision: sampling.model.revision,
|
||||
checkpointIdentity: sampling.model.checkpoint_identity,
|
||||
dtype: sampling.model.dtype,
|
||||
},
|
||||
artifactHashes: Object.fromEntries(
|
||||
Object.entries(inputBytes).map(([name, bytes]) => [name, sha256(bytes)]),
|
||||
),
|
||||
grid: {
|
||||
formalSources: sampling.summary.sources,
|
||||
formalRuns: sampling.summary.runs,
|
||||
formalOutputs: sampling.summary.outputs,
|
||||
mainT0Outputs: evaluation.summary.main_t0.outputs,
|
||||
diagnosticAdditionalOutputs:
|
||||
evaluation.summary.diagnostic_additional_t1_t3.outputs,
|
||||
naturalEos: sampling.summary.natural_eos,
|
||||
budgetTruncated: sampling.summary.budget_truncated,
|
||||
uniqueTrajectories: sampling.summary.unique_generated_token_hashes,
|
||||
promptHashesExact: sampling.source_contract.prompt_hash_audit.exact,
|
||||
torchRngUnchangedRuns: sampling.summary.torch_rng_unchanged_runs,
|
||||
},
|
||||
sampler: {
|
||||
name: sampling.generation_contract.decode,
|
||||
temperature: sampling.generation_contract.distribution_temperature,
|
||||
topP: sampling.generation_contract.distribution_top_p,
|
||||
softmaxDtype: sampling.generation_contract.softmax_dtype,
|
||||
cdfDtype: sampling.generation_contract.cdf_dtype,
|
||||
uniformDtype: sampling.generation_contract.uniform_dtype,
|
||||
transformersGenerateCalled:
|
||||
sampling.generation_contract.transformers_generate_called,
|
||||
torchMultinomialCalled:
|
||||
sampling.generation_contract.torch_multinomial_called,
|
||||
commonRandomNumbers:
|
||||
sampling.seed_contract.explicit_common_random_numbers,
|
||||
},
|
||||
conditionTable,
|
||||
contrasts,
|
||||
diagnostic,
|
||||
tasks,
|
||||
outcomes: {
|
||||
code: evaluation.summary.main_t0.by_domain.code.task_outcomes,
|
||||
math: evaluation.summary.main_t0.by_domain.math.task_outcomes,
|
||||
},
|
||||
reproduction: reproduction.summary,
|
||||
bootstrap: analysis.bootstrap_contract,
|
||||
uniformExample: {
|
||||
sourceId: exampleSource.id,
|
||||
tape: exampleRun.tape_label,
|
||||
uniformUint64FirstEightHex: exampleRun.uniform_uint64_first_eight_hex,
|
||||
uniformFloat32FirstEight: exampleRun.uniform_float32_first_eight,
|
||||
conditions: Object.fromEntries(
|
||||
exampleRun.outputs.map((output) => [
|
||||
output.condition,
|
||||
{
|
||||
generatedTokenIds: output.generated_token_ids.slice(0, 8),
|
||||
generatedTokens: output.generated_tokens,
|
||||
naturalEos: output.hit_eos,
|
||||
},
|
||||
]),
|
||||
),
|
||||
},
|
||||
deviations: [
|
||||
{
|
||||
id: "prompt-hash-correction",
|
||||
severity: "corrected-before-output",
|
||||
summary:
|
||||
"The first manifest mislabeled routing-probe hashes as Chat prompt hashes; all 256 Chat hashes were corrected before model output.",
|
||||
},
|
||||
{
|
||||
id: "gold-loaded-in-runner",
|
||||
severity: "reported-process-deviation",
|
||||
summary:
|
||||
"The reused runner loaded gold for a post-decode narrow task_score before generation finished. Gold never entered prompts, logits, sampling, selection, or the authoritative evaluator.",
|
||||
},
|
||||
],
|
||||
evidenceBoundary: [
|
||||
"HumanEval and GSM8K remain separate.",
|
||||
"Selected-task bands cover only the frozen 32-task frame under T0.",
|
||||
"T1-T3 are sensitivity diagnostics, not extra independent tasks.",
|
||||
"The explicit sampler uses the official .3/.95 distribution but is not a torch.multinomial trajectory.",
|
||||
"Period prompts are counterfactual and not official-valid chats.",
|
||||
],
|
||||
};
|
||||
|
||||
await writeFile(paths.output, `${JSON.stringify(compact, null, 2)}\n`);
|
||||
const outputBytes = await readFile(paths.output);
|
||||
process.stdout.write(
|
||||
`${JSON.stringify(
|
||||
{
|
||||
output: paths.output,
|
||||
bytes: outputBytes.length,
|
||||
sha256: sha256(outputBytes),
|
||||
tasks: Object.values(tasks).reduce((sum, rows) => sum + rows.length, 0),
|
||||
formalOutputs: compact.grid.formalOutputs,
|
||||
reproduction: compact.reproduction,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
);
|
||||
@@ -0,0 +1,203 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import { resolve } from "node:path";
|
||||
import process from "node:process";
|
||||
|
||||
const PROTOCOL_ID = "llm-atlas-deepseek-chat-task-bootstrap-crn-v1";
|
||||
const CONDITIONS = ["s0_eos", "s1_eos", "s0_period", "s1_period"];
|
||||
const DOMAINS = ["code", "math"];
|
||||
const DIAGNOSTIC_INDICES = new Set([0, 8, 16, 24]);
|
||||
const REPLAY_INDICES = new Set([0, 4, 8, 12, 16, 20, 24, 28]);
|
||||
|
||||
function sha256(value) {
|
||||
return createHash("sha256").update(value).digest("hex");
|
||||
}
|
||||
|
||||
function uint32(label) {
|
||||
return createHash("sha256").update(label).digest().readUInt32BE(0);
|
||||
}
|
||||
|
||||
function canonical(value) {
|
||||
if (Array.isArray(value)) {
|
||||
return `[${value.map(canonical).join(",")}]`;
|
||||
}
|
||||
if (value !== null && typeof value === "object") {
|
||||
return `{${Object.keys(value)
|
||||
.sort()
|
||||
.map((key) => `${JSON.stringify(key)}:${canonical(value[key])}`)
|
||||
.join(",")}}`;
|
||||
}
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = {};
|
||||
for (let index = 2; index < argv.length; index += 2) {
|
||||
const key = argv[index];
|
||||
const value = argv[index + 1];
|
||||
if (!key?.startsWith("--") || value === undefined) {
|
||||
throw new Error(`Expected --name value arguments; got ${key ?? "<end>"}`);
|
||||
}
|
||||
args[key.slice(2)] = value;
|
||||
}
|
||||
if (!args.reference || !args.output) {
|
||||
throw new Error("Usage: build-deepseek-chat-task-bootstrap-manifest.mjs --reference FILE --output FILE");
|
||||
}
|
||||
return args;
|
||||
}
|
||||
|
||||
const args = parseArgs(process.argv);
|
||||
const referencePath = resolve(args.reference);
|
||||
const outputPath = resolve(args.output);
|
||||
const referenceBytes = await readFile(referencePath);
|
||||
const reference = JSON.parse(referenceBytes);
|
||||
const selected = reference.corpus_contract?.selected ?? [];
|
||||
|
||||
const sources = DOMAINS.flatMap((domain) => {
|
||||
const rows = selected
|
||||
.filter((row) => row.domain === domain)
|
||||
.sort((left, right) => left.within_domain_index - right.within_domain_index);
|
||||
if (rows.length !== 32) {
|
||||
throw new Error(`${domain}: expected 32 frozen sources, observed ${rows.length}`);
|
||||
}
|
||||
return rows.map((row, domainIndex) => {
|
||||
if (row.within_domain_index !== domainIndex) {
|
||||
throw new Error(`${domain}: non-contiguous within_domain_index at ${domainIndex}`);
|
||||
}
|
||||
const promptHashes = Object.fromEntries(
|
||||
CONDITIONS.map((condition) => {
|
||||
const hash = row.conditions?.[condition]?.token_ids_sha256;
|
||||
if (!/^[0-9a-f]{64}$/.test(hash ?? "")) {
|
||||
throw new Error(`${row.id}/${condition}: missing prompt hash`);
|
||||
}
|
||||
return [condition, hash];
|
||||
}),
|
||||
);
|
||||
return {
|
||||
id: row.id,
|
||||
domain,
|
||||
domain_index: domainIndex,
|
||||
selection_rank: row.selection_rank,
|
||||
source_text_sha256: row.text_sha256,
|
||||
routing_probe_prompt_token_ids_sha256: promptHashes,
|
||||
main_tapes: ["T0"],
|
||||
diagnostic_tapes: DIAGNOSTIC_INDICES.has(domainIndex)
|
||||
? ["T1", "T2", "T3"]
|
||||
: [],
|
||||
independent_replay: REPLAY_INDICES.has(domainIndex),
|
||||
};
|
||||
});
|
||||
});
|
||||
|
||||
const tapes = Array.from({ length: 4 }, (_, index) => {
|
||||
const label = `T${index}`;
|
||||
const derivation = `${PROTOCOL_ID}/tape/${index}`;
|
||||
return {
|
||||
label,
|
||||
index,
|
||||
display_seed: uint32(derivation),
|
||||
derivation,
|
||||
derivation_sha256: sha256(derivation),
|
||||
};
|
||||
});
|
||||
|
||||
const manifest = {
|
||||
schema_version: 1,
|
||||
protocol_id: PROTOCOL_ID,
|
||||
status: "preregistered_before_any_protocol_output",
|
||||
source_reference: {
|
||||
path: args.reference,
|
||||
sha256: sha256(referenceBytes),
|
||||
sample_salt: reference.corpus_contract.sample_salt,
|
||||
selection_rule:
|
||||
"For each domain, all 32 rows ordered by frozen within_domain_index; no outcome-based selection.",
|
||||
},
|
||||
model: {
|
||||
repo: "deepseek-ai/DeepSeek-V2-Lite-Chat",
|
||||
revision: "85864749cd611b4353ce1decdb286193298f64c7",
|
||||
},
|
||||
conditions: CONDITIONS,
|
||||
domains: DOMAINS,
|
||||
sources,
|
||||
source_counts: Object.fromEntries(
|
||||
DOMAINS.map((domain) => [
|
||||
domain,
|
||||
sources.filter((source) => source.domain === domain).length,
|
||||
]),
|
||||
),
|
||||
tape_contract: {
|
||||
tapes,
|
||||
main_tape: "T0",
|
||||
diagnostic_source_indices_per_domain: [...DIAGNOSTIC_INDICES],
|
||||
diagnostic_tapes: ["T0", "T1", "T2", "T3"],
|
||||
uniform_uint64:
|
||||
"first 8 SHA-256 bytes, big-endian, of protocol_id + '\\0uniform\\0' + tape_label + '\\0' + source_id + '\\0' + decimal(step)",
|
||||
uniform_open_interval:
|
||||
"u_t = (uniform_uint64 + 0.5) / 2^64; cast to torch.float32 before searchsorted",
|
||||
common_random_number_scope:
|
||||
"For the same source and tape, all four conditions consume the identical u_t at generation step t while each trajectory remains active.",
|
||||
},
|
||||
execution_grid: {
|
||||
main: {
|
||||
sources: 64,
|
||||
tapes_per_source: 1,
|
||||
conditions: 4,
|
||||
outputs: 256,
|
||||
},
|
||||
additional_tape_diagnostic: {
|
||||
sources: 8,
|
||||
additional_tapes_per_source: 3,
|
||||
conditions: 4,
|
||||
outputs: 96,
|
||||
},
|
||||
formal_outputs: 352,
|
||||
independent_replay: {
|
||||
source_indices_per_domain: [...REPLAY_INDICES],
|
||||
tapes: ["T0"],
|
||||
conditions: 4,
|
||||
outputs: 64,
|
||||
},
|
||||
},
|
||||
bootstrap_contract: {
|
||||
resamples: 10000,
|
||||
domain_separated: true,
|
||||
main_task_bootstrap_seed: uint32(
|
||||
`${PROTOCOL_ID}/bootstrap/task-bootstrap`,
|
||||
),
|
||||
diagnostic_task_bootstrap_seed: uint32(
|
||||
`${PROTOCOL_ID}/bootstrap/diagnostic-task-bootstrap`,
|
||||
),
|
||||
tape_bootstrap_seed: uint32(
|
||||
`${PROTOCOL_ID}/bootstrap/tape-bootstrap`,
|
||||
),
|
||||
estimand_boundary:
|
||||
"Selected-task resampling bands for this fixed 32-task frame and one frozen tape; not benchmark-population or generation-seed confidence intervals.",
|
||||
},
|
||||
prompt_contract: {
|
||||
routing_probe_hashes_are_not_chat_generation_hashes: true,
|
||||
chat_generation_hashes:
|
||||
"Added by freeze-deepseek-chat-task-bootstrap-prompts.py before any protocol output.",
|
||||
},
|
||||
};
|
||||
|
||||
manifest.canonical_content_sha256 = sha256(canonical(manifest));
|
||||
await writeFile(outputPath, `${JSON.stringify(manifest, null, 2)}\n`);
|
||||
|
||||
const outputBytes = await readFile(outputPath);
|
||||
process.stdout.write(
|
||||
`${JSON.stringify(
|
||||
{
|
||||
output: outputPath,
|
||||
bytes: outputBytes.length,
|
||||
sha256: sha256(outputBytes),
|
||||
canonical_content_sha256: manifest.canonical_content_sha256,
|
||||
sources: sources.length,
|
||||
formal_outputs: manifest.execution_grid.formal_outputs,
|
||||
replay_outputs: manifest.execution_grid.independent_replay.outputs,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
);
|
||||
@@ -1212,7 +1212,7 @@ console.log(JSON.stringify(report, null, 2));
|
||||
const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
|
||||
const failures = [];
|
||||
if (!overview.title.includes("为什么转向")) failures.push("专题标题异常");
|
||||
if (overview.sections !== 30 || overview.tocLinks !== 30) failures.push("二十九个编号专题加阅读链的目录结构异常");
|
||||
if (overview.sections !== 31 || overview.tocLinks !== 31) failures.push("三十个编号专题加阅读链的目录结构异常");
|
||||
if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常");
|
||||
if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
@@ -1220,7 +1220,7 @@ if (overview.artifactTabs !== 13 || overview.artifactPanels !== 13 || overview.a
|
||||
if (overview.behaviorTabs !== 4 || overview.behaviorPanels !== 4 || overview.behaviorSources !== 16 || overview.behaviorEdges !== 10) failures.push("Chat 行为实验结构异常");
|
||||
if (overview.completionDepthTabs !== 4 || overview.completionDepthPanels !== 4 || overview.hiddenStages !== 29 || overview.routerLayers !== 26) failures.push("Chat 完成度与全深度实验结构异常");
|
||||
if (overview.crossSourceTabs !== 4 || overview.crossSourcePanels !== 4) failures.push("跨来源采样实验结构异常");
|
||||
if (overview.heroLabs !== "21 个可操作实验") failures.push("DeepSeek 实验总数账异常");
|
||||
if (overview.heroLabs !== "22 个可操作实验") failures.push("DeepSeek 实验总数账异常");
|
||||
if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
|
||||
if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常");
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import process from "node:process";
|
||||
|
||||
const PROTOCOL_ID = "llm-atlas-deepseek-chat-task-bootstrap-crn-v1";
|
||||
const CONDITIONS = ["s0_eos", "s1_eos", "s0_period", "s1_period"];
|
||||
const EXPECTED_HASHES = {
|
||||
sampling: "ea0607f2b197fac3f794655c1538ce1f9a1cb072d637eb31d35573682e311809",
|
||||
evaluation: "82b2fc5d1f854a7e0cd7aba7c70ff74d733d24221522a4f92b962478c76177ab",
|
||||
replay: "6519947e2fa4327c1ba2cc506b0861f172a6bdbd4f2d6787447edaf8fbcac508",
|
||||
reproduction: "63ed39e5dcdbc2a30e516172f3657dfa453ff3b23d5bd5c49c734939242a70f5",
|
||||
analysis: "9ab17561ced930a668c082141f7c6e013cbda70e42b09de63d41f1b82c01a6ae",
|
||||
manifest: "6313e70536c464fe598a93035576752418f08016dfd60ac246437c3b43bf2ae1",
|
||||
};
|
||||
const DEFAULT_PATHS = {
|
||||
sampling: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json",
|
||||
evaluation: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json",
|
||||
replay: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-replay.json",
|
||||
reproduction: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-reproduction.json",
|
||||
analysis: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-analysis.json",
|
||||
manifest: "research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json",
|
||||
};
|
||||
|
||||
function sha256(value) {
|
||||
return createHash("sha256").update(value).digest("hex");
|
||||
}
|
||||
|
||||
function canonicalIntegerArray(values) {
|
||||
return `[${values.map((value) => value.toString()).join(",")}]`;
|
||||
}
|
||||
|
||||
function tapeUint64(tape, sourceId, step) {
|
||||
const payload = `${PROTOCOL_ID}\0uniform\0${tape}\0${sourceId}\0${step}`;
|
||||
return createHash("sha256").update(payload).digest().readBigUInt64BE(0);
|
||||
}
|
||||
|
||||
function assert(value, message) {
|
||||
if (!value) {
|
||||
throw new Error(message);
|
||||
}
|
||||
}
|
||||
|
||||
const loaded = {};
|
||||
for (const [name, path] of Object.entries(DEFAULT_PATHS)) {
|
||||
const bytes = await readFile(path);
|
||||
assert(
|
||||
sha256(bytes) === EXPECTED_HASHES[name],
|
||||
`${name}: file SHA-256 differs`,
|
||||
);
|
||||
loaded[name] = JSON.parse(bytes);
|
||||
}
|
||||
const { sampling, evaluation, replay, reproduction, analysis, manifest } = loaded;
|
||||
for (const [name, payload] of Object.entries(loaded)) {
|
||||
assert(payload.protocol_id === PROTOCOL_ID, `${name}: protocol ID differs`);
|
||||
}
|
||||
|
||||
assert(sampling.execution_mode === "formal", "sampling: not formal mode");
|
||||
assert(sampling.sources.length === 64, "sampling: source count differs");
|
||||
assert(sampling.summary.runs === 88, "sampling: run count differs");
|
||||
assert(sampling.summary.outputs === 352, "sampling: output count differs");
|
||||
assert(sampling.summary.natural_eos === 343, "sampling: EOS count differs");
|
||||
assert(sampling.summary.budget_truncated === 9, "sampling: truncation count differs");
|
||||
assert(
|
||||
sampling.summary.torch_rng_unchanged_runs === 88,
|
||||
"sampling: RNG nonconsumption count differs",
|
||||
);
|
||||
assert(
|
||||
sampling.source_contract.prompt_hash_audit.exact === 256,
|
||||
"sampling: prompt audit differs",
|
||||
);
|
||||
|
||||
const manifestBySource = new Map(
|
||||
manifest.sources.map((source) => [source.id, source]),
|
||||
);
|
||||
const samplingKeys = new Set();
|
||||
let uniformOutputHashesExact = 0;
|
||||
let diagnosticSources = 0;
|
||||
for (const source of sampling.sources) {
|
||||
const frozen = manifestBySource.get(source.id);
|
||||
assert(frozen, `${source.id}: absent from manifest`);
|
||||
assert(source.domain === frozen.domain, `${source.id}: domain differs`);
|
||||
assert(
|
||||
source.within_domain_index === frozen.domain_index,
|
||||
`${source.id}: domain index differs`,
|
||||
);
|
||||
const expectedTapes = [...frozen.main_tapes, ...frozen.diagnostic_tapes];
|
||||
const observedTapes = source.runs.map((run) => run.tape_label);
|
||||
assert(
|
||||
JSON.stringify(observedTapes) === JSON.stringify(expectedTapes),
|
||||
`${source.id}: tape assignment differs`,
|
||||
);
|
||||
diagnosticSources += source.runs.length === 4;
|
||||
for (const run of source.runs) {
|
||||
assert(run.torch_rng_unchanged, `${source.id}/${run.tape_label}: RNG changed`);
|
||||
assert(
|
||||
JSON.stringify(run.outputs.map((output) => output.condition))
|
||||
=== JSON.stringify(CONDITIONS),
|
||||
`${source.id}/${run.tape_label}: condition order differs`,
|
||||
);
|
||||
for (const output of run.outputs) {
|
||||
const key = `${source.id}\0${run.tape_label}\0${output.condition}`;
|
||||
assert(!samplingKeys.has(key), `${key}: duplicate sampling key`);
|
||||
samplingKeys.add(key);
|
||||
assert(
|
||||
output.prompt_token_ids_sha256
|
||||
=== frozen.chat_generation_prompt_token_ids_sha256[output.condition],
|
||||
`${key}: prompt hash differs`,
|
||||
);
|
||||
const uniforms = Array.from(
|
||||
{ length: output.uniform_steps_consumed },
|
||||
(_, step) => tapeUint64(run.tape_label, source.id, step),
|
||||
);
|
||||
assert(
|
||||
sha256(canonicalIntegerArray(uniforms))
|
||||
=== output.uniform_uint64_prefix_sha256,
|
||||
`${key}: uniform prefix hash differs`,
|
||||
);
|
||||
uniformOutputHashesExact += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
assert(diagnosticSources === 8, "sampling: diagnostic source count differs");
|
||||
assert(samplingKeys.size === 352, "sampling: unique key count differs");
|
||||
assert(uniformOutputHashesExact === 352, "sampling: uniform audit differs");
|
||||
|
||||
assert(evaluation.rows.length === 352, "evaluation: row count differs");
|
||||
const evaluationKeys = new Set(
|
||||
evaluation.rows.map(
|
||||
(row) => `${row.source_id}\0${row.tape_label}\0${row.condition}`,
|
||||
),
|
||||
);
|
||||
assert(evaluationKeys.size === 352, "evaluation: duplicate keys");
|
||||
assert(
|
||||
[...evaluationKeys].every((key) => samplingKeys.has(key)),
|
||||
"evaluation: cell key absent from sampling",
|
||||
);
|
||||
assert(
|
||||
evaluation.summary.main_t0.outputs === 256,
|
||||
"evaluation: T0 output count differs",
|
||||
);
|
||||
assert(
|
||||
evaluation.summary.main_t0.by_domain.code.fixed_budget_success === 59,
|
||||
"evaluation: code success count differs",
|
||||
);
|
||||
assert(
|
||||
evaluation.summary.main_t0.by_domain.math.fixed_budget_success === 71,
|
||||
"evaluation: math success count differs",
|
||||
);
|
||||
|
||||
assert(replay.execution_mode === "replay", "replay: mode differs");
|
||||
assert(replay.sources.length === 16, "replay: source count differs");
|
||||
assert(replay.summary.outputs === 64, "replay: output count differs");
|
||||
assert(
|
||||
replay.summary.torch_rng_unchanged_runs === 16,
|
||||
"replay: RNG nonconsumption count differs",
|
||||
);
|
||||
assert(
|
||||
reproduction.summary.cells === 64
|
||||
&& reproduction.summary.all_preregistered_fields_exact === 64,
|
||||
"reproduction: exact cell count differs",
|
||||
);
|
||||
assert(
|
||||
Object.values(reproduction.summary.by_field).every((count) => count === 64),
|
||||
"reproduction: a field is not 64/64 exact",
|
||||
);
|
||||
|
||||
assert(
|
||||
analysis.bootstrap_contract.resamples === 10000
|
||||
&& analysis.bootstrap_contract.seed === 1364512825,
|
||||
"analysis: bootstrap contract differs",
|
||||
);
|
||||
let crnContrastChecks = 0;
|
||||
for (const domain of ["code", "math"]) {
|
||||
const domainAnalysis = analysis.main_t0_selected_task_analysis[domain];
|
||||
assert(domainAnalysis.tasks === 32, `${domain}: task count differs`);
|
||||
for (const contrast of Object.values(domainAnalysis.contrasts)) {
|
||||
assert(
|
||||
contrast.trajectory.shared_uniform_prefix_exact === 32,
|
||||
`${domain}: contrast CRN audit differs`,
|
||||
);
|
||||
crnContrastChecks += contrast.trajectory.shared_uniform_prefix_exact;
|
||||
}
|
||||
}
|
||||
assert(crnContrastChecks === 256, "analysis: CRN contrast total differs");
|
||||
|
||||
process.stdout.write(
|
||||
`${JSON.stringify(
|
||||
{
|
||||
passed: true,
|
||||
protocol_id: PROTOCOL_ID,
|
||||
files: EXPECTED_HASHES,
|
||||
formal: {
|
||||
sources: sampling.sources.length,
|
||||
runs: sampling.summary.runs,
|
||||
outputs: sampling.summary.outputs,
|
||||
prompt_hashes_exact: sampling.source_contract.prompt_hash_audit.exact,
|
||||
uniform_output_hashes_exact: uniformOutputHashesExact,
|
||||
torch_rng_unchanged_runs: sampling.summary.torch_rng_unchanged_runs,
|
||||
},
|
||||
evaluation: {
|
||||
rows: evaluation.rows.length,
|
||||
main_t0_outputs: evaluation.summary.main_t0.outputs,
|
||||
code_success: evaluation.summary.main_t0.by_domain.code.fixed_budget_success,
|
||||
math_success: evaluation.summary.main_t0.by_domain.math.fixed_budget_success,
|
||||
},
|
||||
reproduction: reproduction.summary,
|
||||
analysis: {
|
||||
bootstrap_resamples: analysis.bootstrap_contract.resamples,
|
||||
crn_contrast_checks: crnContrastChecks,
|
||||
},
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
);
|
||||
@@ -240,8 +240,8 @@ assert(overview.facts["SAMPLED OUTPUTS"] === "256", "output headline 异常");
|
||||
assert(overview.facts["NATURAL EOS"] === "250 / 256", "EOS headline 异常");
|
||||
assert(overview.facts["MATH · STRICT"] === "47 / 64", "Math headline 异常");
|
||||
assert(overview.facts["CODE · TESTS"] === "52 / 64", "Code headline 异常");
|
||||
assert(overview.labs === "21 个可操作实验", "DeepSeek LABS 总账异常");
|
||||
assert(overview.status === "七轮 · 512 条采样", "DeepSeek STATUS 总账异常");
|
||||
assert(overview.labs === "22 个可操作实验", "DeepSeek LABS 总账异常");
|
||||
assert(overview.status === "八轮 · 864 条采样", "DeepSeek STATUS 总账异常");
|
||||
assert(overview.overflow <= 1, `桌面横向溢出 ${overview.overflow}px`);
|
||||
|
||||
assert(hierarchySwitch.cards.length === 4, "Code source cards 数量异常");
|
||||
|
||||
@@ -223,7 +223,7 @@ assert(overview.panels === 4, `sampling panels=${overview.panels}`);
|
||||
assert(overview.seedBars === 8, `initial seed bars=${overview.seedBars}`);
|
||||
assert(overview.conditions === 8, `condition rows=${overview.conditions}`);
|
||||
assert(overview.edges === 10, `edge rows=${overview.edges}`);
|
||||
assert(overview.heroLabs?.startsWith("20"), `hero labs=${overview.heroLabs}`);
|
||||
assert(overview.heroLabs === "22 个可操作实验", `hero labs=${overview.heroLabs}`);
|
||||
assert(overview.activePanel === "trajectories", `active=${overview.activePanel}`);
|
||||
assert(overview.unique === "8 / 8", `initial unique=${overview.unique}`);
|
||||
assert(overview.eos === "5 / 8", `initial eos=${overview.eos}`);
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
import { writeFileSync } from "node:fs";
|
||||
|
||||
const cdpPort = process.env.CDP_PORT ?? "9231";
|
||||
const baseUrl = process.env.SITE_URL ?? "http://127.0.0.1:4327";
|
||||
const pages = await fetch(`http://127.0.0.1:${cdpPort}/json/list`)
|
||||
.then((response) => response.json());
|
||||
const page = pages.find((entry) => entry.type === "page");
|
||||
if (!page) throw new Error(`CDP ${cdpPort} 没有可用页面`);
|
||||
|
||||
const socket = new WebSocket(page.webSocketDebuggerUrl);
|
||||
await new Promise((resolve, reject) => {
|
||||
socket.addEventListener("open", resolve, { once: true });
|
||||
socket.addEventListener("error", reject, { once: true });
|
||||
});
|
||||
|
||||
let nextId = 0;
|
||||
const pending = new Map();
|
||||
const exceptions = [];
|
||||
socket.addEventListener("message", (event) => {
|
||||
const message = JSON.parse(event.data);
|
||||
if (message.id && pending.has(message.id)) {
|
||||
const { resolve, reject } = pending.get(message.id);
|
||||
pending.delete(message.id);
|
||||
if (message.error) reject(new Error(message.error.message));
|
||||
else resolve(message.result);
|
||||
}
|
||||
if (message.method === "Runtime.exceptionThrown") {
|
||||
exceptions.push(
|
||||
message.params.exceptionDetails.exception?.description
|
||||
?? message.params.exceptionDetails.text,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
const command = (method, params = {}) => new Promise((resolve, reject) => {
|
||||
const id = ++nextId;
|
||||
pending.set(id, { resolve, reject });
|
||||
socket.send(JSON.stringify({ id, method, params }));
|
||||
});
|
||||
const pause = (milliseconds) => new Promise(
|
||||
(resolve) => setTimeout(resolve, milliseconds),
|
||||
);
|
||||
const evaluate = async (expression) => {
|
||||
const result = await command("Runtime.evaluate", {
|
||||
expression,
|
||||
returnByValue: true,
|
||||
awaitPromise: true,
|
||||
});
|
||||
if (result.exceptionDetails) {
|
||||
throw new Error(
|
||||
result.exceptionDetails.exception?.description
|
||||
?? result.exceptionDetails.text,
|
||||
);
|
||||
}
|
||||
return result.result.value;
|
||||
};
|
||||
const screenshot = async (path) => {
|
||||
const result = await command("Page.captureScreenshot", {
|
||||
format: "png",
|
||||
captureBeyondViewport: false,
|
||||
});
|
||||
writeFileSync(path, Buffer.from(result.data, "base64"));
|
||||
};
|
||||
const assert = (condition, message) => {
|
||||
if (!condition) throw new Error(message);
|
||||
};
|
||||
|
||||
await command("Page.enable");
|
||||
await command("Runtime.enable");
|
||||
await command("Emulation.setDeviceMetricsOverride", {
|
||||
width: 1440,
|
||||
height: 1100,
|
||||
deviceScaleFactor: 1,
|
||||
mobile: false,
|
||||
});
|
||||
await command("Page.navigate", {
|
||||
url: `${baseUrl}/deepseek/#task-bootstrap-crn`,
|
||||
});
|
||||
for (let attempt = 0; attempt < 100; attempt += 1) {
|
||||
await pause(100);
|
||||
if (await evaluate("document.readyState === 'complete'")) break;
|
||||
}
|
||||
|
||||
const overview = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-task-bootstrap-lab]");
|
||||
if (!root) return null;
|
||||
document.documentElement.style.scrollBehavior = "auto";
|
||||
window.scrollTo(0, root.getBoundingClientRect().top + window.scrollY);
|
||||
const facts = Object.fromEntries(
|
||||
[...root.querySelectorAll(".tb-ledger article")].map((node) => [
|
||||
node.querySelector("span").textContent.trim(),
|
||||
node.querySelector("b").textContent.trim(),
|
||||
]),
|
||||
);
|
||||
return {
|
||||
tabs: root.querySelectorAll("[data-tb-tab]").length,
|
||||
panels: root.querySelectorAll("[data-tb-panel]").length,
|
||||
active: root.querySelector("[data-tb-panel]:not([hidden])")?.dataset.tbPanel,
|
||||
bars: root.querySelectorAll(".uniform-bars article").length,
|
||||
lanes: root.querySelectorAll(".token-lanes article").length,
|
||||
laneTokens: root.querySelectorAll(".token-lanes article i").length,
|
||||
facts,
|
||||
labs: [...document.querySelectorAll(".page-facts > div")]
|
||||
.find((node) => node.querySelector("dt")?.textContent.trim() === "LABS")
|
||||
?.querySelector("dd")?.textContent.trim(),
|
||||
status: [...document.querySelectorAll(".page-facts > div")]
|
||||
.find((node) => node.querySelector("dt")?.textContent.trim() === "STATUS")
|
||||
?.querySelector("dd")?.textContent.trim(),
|
||||
documentOverflow: document.documentElement.scrollWidth
|
||||
- document.documentElement.clientWidth,
|
||||
};
|
||||
})()`);
|
||||
await pause(250);
|
||||
await screenshot("/tmp/llm-atlas-task-bootstrap-sampler-desktop.png");
|
||||
|
||||
const bootstrap = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-task-bootstrap-lab]");
|
||||
root.querySelector('[data-tb-tab="bootstrap"]').click();
|
||||
const read = () => ({
|
||||
active: root.querySelector("[data-tb-panel]:not([hidden])").dataset.tbPanel,
|
||||
conditionCards: [...root.querySelectorAll("[data-tb-condition-cards] article")]
|
||||
.map((node) => node.querySelector("b").textContent.trim()),
|
||||
forest: [...root.querySelectorAll("[data-tb-forest] > article")].map((node) => ({
|
||||
label: node.querySelector("span").textContent.trim(),
|
||||
point: node.querySelector(":scope > b").textContent.trim(),
|
||||
band: node.querySelector("small").textContent.trim(),
|
||||
clear: node.querySelector("u").classList.contains("clear"),
|
||||
})),
|
||||
robust: root.querySelector("[data-tb-robust-count]").textContent.trim(),
|
||||
});
|
||||
const codeCorrectness = read();
|
||||
root.querySelector('[data-tb-metric="generated_tokens"]').click();
|
||||
const codeLength = read();
|
||||
root.querySelector('[data-tb-domain="math"]').click();
|
||||
const mathLength = read();
|
||||
return { codeCorrectness, codeLength, mathLength };
|
||||
})()`);
|
||||
await pause(150);
|
||||
await screenshot("/tmp/llm-atlas-task-bootstrap-forest-desktop.png");
|
||||
|
||||
const taskExplorer = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-task-bootstrap-lab]");
|
||||
root.querySelector('[data-tb-tab="tasks"]').click();
|
||||
const domain = root.querySelector("[data-tb-task-domain]");
|
||||
const contrast = root.querySelector("[data-tb-task-contrast]");
|
||||
domain.value = "math";
|
||||
domain.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
contrast.value = "system_at_eos";
|
||||
contrast.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
root.querySelector('[data-tb-task-page="2"]').click();
|
||||
return {
|
||||
active: root.querySelector("[data-tb-panel]:not([hidden])").dataset.tbPanel,
|
||||
rows: root.querySelectorAll("[data-tb-task-rows] > article").length,
|
||||
page: root.querySelector("[data-tb-task-page-label]").textContent.trim(),
|
||||
selectedPage: root.querySelector('[data-tb-task-page][aria-pressed="true"]')
|
||||
.dataset.tbTaskPage,
|
||||
firstTask: root.querySelector("[data-tb-task-rows] > article span b")
|
||||
.textContent.trim(),
|
||||
transition: [...root.querySelectorAll("[data-tb-transition-cards] article")]
|
||||
.map((node) => node.querySelector("b").textContent.trim()),
|
||||
};
|
||||
})()`);
|
||||
|
||||
const tape = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-task-bootstrap-lab]");
|
||||
root.querySelector('[data-tb-tab="tapes"]').click();
|
||||
const domain = root.querySelector("[data-tb-tape-domain]");
|
||||
const contrast = root.querySelector("[data-tb-tape-contrast]");
|
||||
domain.value = "math";
|
||||
domain.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
contrast.value = "system_at_eos";
|
||||
contrast.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
return {
|
||||
active: root.querySelector("[data-tb-panel]:not([hidden])").dataset.tbPanel,
|
||||
rows: root.querySelectorAll("[data-tb-tape-rows] > article").length,
|
||||
cells: root.querySelectorAll("[data-tb-tape-rows] > article > b").length,
|
||||
means: [...root.querySelectorAll("[data-tb-tape-means] > b")]
|
||||
.map((node) => node.textContent.trim()),
|
||||
taskRange: root.querySelector("[data-tb-task-range]").textContent.trim(),
|
||||
tapeRange: root.querySelector("[data-tb-tape-range]").textContent.trim(),
|
||||
};
|
||||
})()`);
|
||||
|
||||
const audit = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-task-bootstrap-lab]");
|
||||
root.querySelector('[data-tb-tab="audit"]').click();
|
||||
return {
|
||||
active: root.querySelector("[data-tb-panel]:not([hidden])").dataset.tbPanel,
|
||||
fields: [...root.querySelectorAll(".replay-fields article")]
|
||||
.map((node) => node.querySelector("b").textContent.trim()),
|
||||
codeOutcomes: root.querySelectorAll(".failure-ledger > article:first-child > div").length,
|
||||
mathOutcomes: root.querySelectorAll(".failure-ledger > article:last-child > div").length,
|
||||
deviations: root.querySelectorAll(".deviation-ledger article").length,
|
||||
artifacts: root.querySelectorAll(".artifact-chain article").length,
|
||||
};
|
||||
})()`);
|
||||
|
||||
const keyboard = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-task-bootstrap-lab]");
|
||||
const first = root.querySelector('[data-tb-tab="sampler"]');
|
||||
first.click();
|
||||
first.focus();
|
||||
first.dispatchEvent(new KeyboardEvent("keydown", {
|
||||
key: "ArrowRight",
|
||||
bubbles: true,
|
||||
}));
|
||||
return {
|
||||
selected: root.querySelector('[data-tb-tab][aria-selected="true"]').dataset.tbTab,
|
||||
active: root.querySelector("[data-tb-panel]:not([hidden])").dataset.tbPanel,
|
||||
focused: document.activeElement.dataset.tbTab,
|
||||
};
|
||||
})()`);
|
||||
|
||||
await command("Emulation.setDeviceMetricsOverride", {
|
||||
width: 390,
|
||||
height: 844,
|
||||
deviceScaleFactor: 1,
|
||||
mobile: true,
|
||||
});
|
||||
await pause(300);
|
||||
const mobile = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-task-bootstrap-lab]");
|
||||
root.querySelector('[data-tb-tab="tasks"]').click();
|
||||
root.scrollIntoView();
|
||||
return {
|
||||
documentOverflow: document.documentElement.scrollWidth
|
||||
- document.documentElement.clientWidth,
|
||||
rootOverflow: root.scrollWidth - root.clientWidth,
|
||||
tableWidth: root.querySelector(".task-table").getBoundingClientRect().width,
|
||||
tableScrollWidth: root.querySelector(".task-table").scrollWidth,
|
||||
viewport: window.innerWidth,
|
||||
};
|
||||
})()`);
|
||||
await screenshot("/tmp/llm-atlas-task-bootstrap-mobile.png");
|
||||
|
||||
assert(overview, "找不到任务 bootstrap 实验");
|
||||
assert(overview.tabs === 5 && overview.panels === 5, "五页签/面板合同异常");
|
||||
assert(overview.active === "sampler", "初始面板不是 sampler");
|
||||
assert(overview.bars === 8, "uniform tape 前八步渲染异常");
|
||||
assert(overview.lanes === 4 && overview.laneTokens === 32, "四条件 token lane 异常");
|
||||
assert(overview.facts.TASKS === "32 + 32", "任务 headline 异常");
|
||||
assert(overview.facts["FORMAL GRID"] === "352", "formal headline 异常");
|
||||
assert(overview.facts["UNIFORM AUDIT"] === "352 / 352", "uniform headline 异常");
|
||||
assert(overview.facts["FRESH PROCESS"] === "64 / 64", "replay headline 异常");
|
||||
assert(overview.labs === "22 个可操作实验", "DeepSeek LABS 总账异常");
|
||||
assert(overview.status === "八轮 · 864 条采样", "DeepSeek STATUS 总账异常");
|
||||
assert(overview.documentOverflow <= 1, `桌面横向溢出 ${overview.documentOverflow}px`);
|
||||
|
||||
assert(bootstrap.codeCorrectness.active === "bootstrap", "bootstrap 面板切换异常");
|
||||
assert(bootstrap.codeCorrectness.conditionCards.join("|") === "16 / 32 pass|16 / 32 pass|15 / 32 pass|12 / 32 pass", "Code 条件 pass 表异常");
|
||||
assert(bootstrap.codeCorrectness.forest.length === 4, "forest contrast 数异常");
|
||||
assert(bootstrap.codeCorrectness.robust === "0 / 4 bands 不跨 0", "Code correctness 带数异常");
|
||||
assert(bootstrap.codeLength.robust === "2 / 4 bands 不跨 0", "Code length 带数异常");
|
||||
assert(bootstrap.codeLength.forest.at(-1).point === "-130.9 tok", "Code system-period 长度异常");
|
||||
assert(bootstrap.mathLength.robust === "1 / 4 bands 不跨 0", "Math length 带数异常");
|
||||
assert(bootstrap.mathLength.forest.at(-1).point === "+25.1 tok", "Math system-period 长度异常");
|
||||
|
||||
assert(taskExplorer.active === "tasks", "逐题面板切换异常");
|
||||
assert(taskExplorer.rows === 8 && taskExplorer.page === "17–24 / 32", "逐题分页异常");
|
||||
assert(taskExplorer.selectedPage === "2", "逐题页按钮状态异常");
|
||||
assert(taskExplorer.firstTask === "gsm8k/test/1050", "逐题页首任务异常");
|
||||
assert(taskExplorer.transition.join("|") === "3 / 32|3 / 32|16 / 32|10 / 32", "Math system-EOS 转移异常");
|
||||
|
||||
assert(tape.active === "tapes", "tape 面板切换异常");
|
||||
assert(tape.rows === 4 && tape.cells === 16, "4×4 tape matrix 异常");
|
||||
assert(tape.means.join("|") === "0.00|+0.50|-0.25|+0.25", "Math system-EOS tape means 异常");
|
||||
assert(tape.taskRange === "1.25" && tape.tapeRange === "1.00", "tape/task range 异常");
|
||||
|
||||
assert(audit.active === "audit", "审计面板切换异常");
|
||||
assert(audit.fields.length === 12, "重放字段数不为 12");
|
||||
assert(audit.fields.every((value) => value === "64 / 64"), "重放字段未全部 exact");
|
||||
assert(audit.codeOutcomes === 6 && audit.mathOutcomes === 2, "失败分类数量异常");
|
||||
assert(audit.deviations === 2, "偏离账数量异常");
|
||||
assert(audit.artifacts === 4, "工件 hash 数量异常");
|
||||
assert(
|
||||
keyboard.selected === "bootstrap"
|
||||
&& keyboard.active === "bootstrap"
|
||||
&& keyboard.focused === "bootstrap",
|
||||
"页签键盘导航异常",
|
||||
);
|
||||
assert(mobile.documentOverflow <= 1, `移动端 document 横向溢出 ${mobile.documentOverflow}px`);
|
||||
assert(mobile.rootOverflow <= 1, `移动端实验横向溢出 ${mobile.rootOverflow}px`);
|
||||
assert(mobile.tableWidth <= mobile.viewport, "移动端任务表容器超出 viewport");
|
||||
assert(mobile.tableScrollWidth > mobile.tableWidth, "移动端任务表没有内部横向滚动");
|
||||
assert(exceptions.length === 0, `浏览器异常：${exceptions.join(" | ")}`);
|
||||
|
||||
console.log(JSON.stringify({
|
||||
overview,
|
||||
bootstrap,
|
||||
taskExplorer,
|
||||
tape,
|
||||
audit,
|
||||
keyboard,
|
||||
mobile,
|
||||
screenshots: [
|
||||
"/tmp/llm-atlas-task-bootstrap-sampler-desktop.png",
|
||||
"/tmp/llm-atlas-task-bootstrap-forest-desktop.png",
|
||||
"/tmp/llm-atlas-task-bootstrap-mobile.png",
|
||||
],
|
||||
}, null, 2));
|
||||
|
||||
socket.close();
|
||||
@@ -0,0 +1,172 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Freeze all 256 Chat-generation prompt hashes into the Round 08 manifest."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
DEEPSEEK_EXPERIMENTS = ROOT / "experiments" / "deepseek"
|
||||
sys.path.insert(0, str(DEEPSEEK_EXPERIMENTS))
|
||||
|
||||
from transformers import AutoTokenizer # noqa: E402
|
||||
|
||||
import v2_lite_chat_special_token_behavior_probe as behavior # noqa: E402
|
||||
import v2_lite_routing_special_token_family_control as special # noqa: E402
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-deepseek-chat-task-bootstrap-crn-v1"
|
||||
CONDITIONS = (
|
||||
"s0_eos",
|
||||
"s1_eos",
|
||||
"s0_period",
|
||||
"s1_period",
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--artifact-dir", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--reference-routing-json",
|
||||
type=Path,
|
||||
required=True,
|
||||
)
|
||||
parser.add_argument("--human-eval", type=Path, required=True)
|
||||
parser.add_argument("--gsm8k", type=Path, required=True)
|
||||
parser.add_argument("--tnews", type=Path, required=True)
|
||||
parser.add_argument("--tnews-archive", type=Path, required=True)
|
||||
parser.add_argument("--wikitext", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def canonical_hash(value: Any) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def install_condition_contract() -> None:
|
||||
factors = {
|
||||
condition: special.FACTORS[condition]
|
||||
for condition in CONDITIONS
|
||||
}
|
||||
special.BOUNDARY_LEVELS = ("eos", "period")
|
||||
special.CONDITIONS = CONDITIONS
|
||||
special.FACTORS = factors
|
||||
special.SYSTEM_CELLS = {
|
||||
"eos": ("s0_eos", "s1_eos"),
|
||||
"period": ("s0_period", "s1_period"),
|
||||
}
|
||||
special.SYSTEM_EDGE_CONTRASTS = {
|
||||
"period_minus_eos": ("eos", "period"),
|
||||
}
|
||||
special.install_control_contract()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
manifest = json.loads(args.manifest.read_text(encoding="utf-8"))
|
||||
if manifest["protocol_id"] != PROTOCOL_ID:
|
||||
raise RuntimeError("manifest protocol ID differs")
|
||||
install_condition_contract()
|
||||
tokenizer = AutoTokenizer.from_pretrained(
|
||||
args.artifact_dir,
|
||||
trust_remote_code=True,
|
||||
local_files_only=True,
|
||||
use_fast=True,
|
||||
)
|
||||
if tokenizer.pad_token_id is None:
|
||||
tokenizer.pad_token_id = tokenizer.eos_token_id
|
||||
args.domains = ["code", "math"]
|
||||
args.per_domain = 32
|
||||
source_rows, _ = behavior.selected_sources(args, tokenizer)
|
||||
frozen_rows = {
|
||||
row["id"]: row for row in manifest["sources"]
|
||||
}
|
||||
audit = []
|
||||
for source in source_rows:
|
||||
frozen = frozen_rows.get(source["id"])
|
||||
if frozen is None:
|
||||
raise RuntimeError(f"{source['id']} is absent from manifest")
|
||||
hashes = {}
|
||||
tokens = {}
|
||||
for condition in CONDITIONS:
|
||||
rendered = special.prior.render_boundary_variant(
|
||||
tokenizer,
|
||||
source["content"],
|
||||
condition,
|
||||
)
|
||||
hashes[condition] = rendered["token_ids_sha256"]
|
||||
tokens[condition] = rendered["tokens"]
|
||||
audit.append(
|
||||
{
|
||||
"source_id": source["id"],
|
||||
"condition": condition,
|
||||
"tokens": rendered["tokens"],
|
||||
"sha256": rendered["token_ids_sha256"],
|
||||
}
|
||||
)
|
||||
frozen["chat_generation_prompt_token_ids_sha256"] = hashes
|
||||
frozen["chat_generation_prompt_tokens"] = tokens
|
||||
if len(audit) != 256 or len(source_rows) != 64:
|
||||
raise RuntimeError(
|
||||
f"expected 64 sources / 256 cells, got "
|
||||
f"{len(source_rows)} / {len(audit)}"
|
||||
)
|
||||
manifest["status"] = (
|
||||
"preregistered_and_chat_prompt_hashes_corrected_before_"
|
||||
"any_protocol_output"
|
||||
)
|
||||
manifest["prompt_contract"] = {
|
||||
"routing_probe_hashes_are_not_chat_generation_hashes": True,
|
||||
"chat_generation_hashes_frozen": True,
|
||||
"chat_generation_prompt_cells": len(audit),
|
||||
"chat_generation_prompt_hashes_sha256": canonical_hash(audit),
|
||||
"renderer": (
|
||||
"official apply_chat_template followed by the frozen "
|
||||
"single boundary-ID edit for period cells"
|
||||
),
|
||||
}
|
||||
manifest.pop("canonical_content_sha256", None)
|
||||
manifest["canonical_content_sha256"] = canonical_hash(manifest)
|
||||
args.manifest.write_text(
|
||||
json.dumps(manifest, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
payload = args.manifest.read_bytes()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"manifest": str(args.manifest),
|
||||
"bytes": len(payload),
|
||||
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||
"canonical_content_sha256": manifest[
|
||||
"canonical_content_sha256"
|
||||
],
|
||||
"sources": len(source_rows),
|
||||
"prompt_cells": len(audit),
|
||||
"prompt_hashes_sha256": manifest["prompt_contract"][
|
||||
"chat_generation_prompt_hashes_sha256"
|
||||
],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,827 @@
|
||||
---
|
||||
import rawLab from "@/data/deepseek-v2-lite-chat-task-bootstrap-crn-compact.json";
|
||||
|
||||
const lab = rawLab as any;
|
||||
const json = JSON.stringify(lab).replaceAll("<", "\\u003c");
|
||||
const example = lab.uniformExample;
|
||||
const conditions = ["s0_eos", "s1_eos", "s0_period", "s1_period"];
|
||||
const conditionLabels: Record<string, string> = {
|
||||
s0_eos: "无 system · EOS",
|
||||
s1_eos: "有 system · EOS",
|
||||
s0_period: "无 system · 句点",
|
||||
s1_period: "有 system · 句点",
|
||||
};
|
||||
---
|
||||
|
||||
<figure class="task-bootstrap-lab" data-task-bootstrap-lab>
|
||||
<header class="tb-head">
|
||||
<div>
|
||||
<p>ROUND 08 / TASK BOOTSTRAP × EXPLICIT CRN</p>
|
||||
<h3>把“换题”和“换随机数”拆开，再问 prompt 到底改变了什么</h3>
|
||||
</div>
|
||||
<p>
|
||||
主分析固定 T0，在 HumanEval 与 GSM8K 各 32 道预选题上逐题配对；
|
||||
另取每域 4 题跑 T0–T3。四格在第 <code>t</code> 步读取同一个显式
|
||||
<code>uₜ</code>，再各自穿过不同的 token CDF。
|
||||
</p>
|
||||
</header>
|
||||
|
||||
<div class="tb-ledger">
|
||||
<article><span>TASKS</span><b>32 + 32</b><p>Code / Math 始终分开</p></article>
|
||||
<article><span>FORMAL GRID</span><b>352</b><p>256 主分析 + 96 额外 tape</p></article>
|
||||
<article class="pass"><span>PROMPT HASH</span><b>256 / 256</b><p>输出前冻结并逐格 exact</p></article>
|
||||
<article><span>UNIFORM AUDIT</span><b>352 / 352</b><p>每条消费前缀重新派生</p></article>
|
||||
<article><span>TASK BOOTSTRAP</span><b>10,000×</b><p>固定 32 题框，不外推总体</p></article>
|
||||
<article class="pass"><span>FRESH PROCESS</span><b>64 / 64</b><p>十二项字段全部 exact</p></article>
|
||||
</div>
|
||||
|
||||
<div class="tb-tabs" role="tablist" aria-label="选择任务 bootstrap 实验视图">
|
||||
<button type="button" role="tab" data-tb-tab="sampler" aria-selected="true">
|
||||
<span>01</span><b>真正的共同随机数</b><small>uniform tape → four CDFs</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-tb-tab="bootstrap" aria-selected="false" tabindex="-1">
|
||||
<span>02</span><b>32 题重采样带</b><small>paired task bootstrap</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-tb-tab="tasks" aria-selected="false" tabindex="-1">
|
||||
<span>03</span><b>逐题看正负抵消</b><small>task × condition explorer</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-tb-tab="tapes" aria-selected="false" tabindex="-1">
|
||||
<span>04</span><b>换题还是换 tape</b><small>4 tasks × 4 tapes</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-tb-tab="audit" aria-selected="false" tabindex="-1">
|
||||
<span>05</span><b>重放、失败与偏离</b><small>evidence boundary</small>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<section class="tb-panel" data-tb-panel="sampler">
|
||||
<div class="tb-panel-lead">
|
||||
<div><span>I / COMMON RANDOM NUMBERS</span><h4>同一个 seed，不一定是同一个随机冲击</h4></div>
|
||||
<p>
|
||||
旧实验把四行放在同一个 seeded batch，<code>torch.multinomial</code> 为不同
|
||||
行消费不同 RNG 子流。本轮直接定义每一步的均匀数，所以配对对象终于可见、可重建。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="pairing-compare">
|
||||
<article>
|
||||
<span>ROUND 06–07 · BATCH SEED</span>
|
||||
<div class="stream-row"><i>seed</i><b>→</b><u>r₀</u><u>r₁</u><u>r₂</u><u>r₃</u></div>
|
||||
<p>同一 seed 与调用时序，但四行不是同一概率分位。</p>
|
||||
</article>
|
||||
<article class="active">
|
||||
<span>ROUND 08 · EXPLICIT TAPE</span>
|
||||
<div class="stream-row"><i>uₜ</i><b>→</b><u>uₜ</u><u>uₜ</u><u>uₜ</u><u>uₜ</u></div>
|
||||
<p>同题、同 tape、同 step 的四格读取完全相同的 <code>uₜ</code>。</p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<div class="sampler-pipeline" aria-label="显式共同随机数采样流程">
|
||||
<article><span>01 / HASH</span><b>SHA-256</b><p>protocol · tape · source · step</p></article>
|
||||
<i>→</i>
|
||||
<article class="uniform"><span>02 / SHARED</span><b>uₜ ∈ (0,1)</b><p>四格同一个概率分位</p></article>
|
||||
<i>→</i>
|
||||
<article><span>03 / FOUR DISTRIBUTIONS</span><b>T .3 · P .95</b><p>prompt 改变各自 logits / CDF</p></article>
|
||||
<i>→</i>
|
||||
<article><span>04 / TOKEN</span><b>searchsorted</b><p>同 uₜ 可以落入不同 token</p></article>
|
||||
</div>
|
||||
|
||||
<div class="uniform-demo">
|
||||
<header>
|
||||
<div><span>REAL T0 TAPE / {example.sourceId}</span><b>前 8 个生成步</b></div>
|
||||
<p>柱高是 float32 <code>uₜ</code>；hex 是冻结的 uint64 前缀。</p>
|
||||
</header>
|
||||
<div class="uniform-bars">
|
||||
{example.uniformFloat32FirstEight.map((value: number, index: number) => (
|
||||
<article style={`--u:${Math.max(0.025, value)}`}>
|
||||
<i></i>
|
||||
<span>t{index}</span>
|
||||
<b>{value.toFixed(3)}</b>
|
||||
<code>{example.uniformUint64FirstEightHex[index].slice(0, 6)}</code>
|
||||
</article>
|
||||
))}
|
||||
</div>
|
||||
<div class="token-lanes">
|
||||
{conditions.map((condition) => (
|
||||
<article>
|
||||
<header><span>{condition}</span><b>{conditionLabels[condition]}</b></header>
|
||||
<div>
|
||||
{example.conditions[condition].generatedTokenIds.map((token: number, index: number) => (
|
||||
<i><small>t{index}</small>{token}</i>
|
||||
))}
|
||||
</div>
|
||||
</article>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<aside class="tb-note">
|
||||
<b>读图关键：四条 lane 上方的随机柱完全相同，token ID 却会分叉</b>
|
||||
<p>
|
||||
共同随机数控制的是 sampling noise，不是把四个条件钉成同一输出。prompt 一旦改变
|
||||
概率分布，同一分位自然可以映射到不同 token。
|
||||
</p>
|
||||
</aside>
|
||||
</section>
|
||||
|
||||
<section class="tb-panel" data-tb-panel="bootstrap" hidden>
|
||||
<div class="tb-panel-lead">
|
||||
<div><span>II / SELECTED-TASK BOOTSTRAP</span><h4>带宽回答“换这 32 道题的权重会怎样”</h4></div>
|
||||
<p>
|
||||
每次在固定 32 题中有放回抽 32 题，四条件保持题级配对。它不包含换随机带的不确定性，
|
||||
也不是完整 benchmark population confidence interval。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="tb-switch-row">
|
||||
<div role="group" aria-label="选择 bootstrap 任务域">
|
||||
<button type="button" data-tb-domain="code" aria-pressed="true">CODE · HUMANEVAL</button>
|
||||
<button type="button" data-tb-domain="math" aria-pressed="false">MATH · GSM8K</button>
|
||||
</div>
|
||||
<div role="group" aria-label="选择 bootstrap 指标">
|
||||
<button type="button" data-tb-metric="fixed_budget_success" aria-pressed="true">CORRECTNESS</button>
|
||||
<button type="button" data-tb-metric="generated_tokens" aria-pressed="false">LENGTH</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="condition-cards" data-tb-condition-cards></div>
|
||||
|
||||
<div class="forest">
|
||||
<header><span>RIGHT LOWER / SHORTER</span><b>0 · NO MEAN DIFFERENCE</b><span>RIGHT HIGHER / LONGER</span></header>
|
||||
<div data-tb-forest></div>
|
||||
</div>
|
||||
|
||||
<div class="bootstrap-reading">
|
||||
<article>
|
||||
<span>WHAT IS RESAMPLED</span>
|
||||
<b>32 selected tasks</b>
|
||||
<p>同一次抽样中，四个 prompt condition 保持配对。</p>
|
||||
</article>
|
||||
<article>
|
||||
<span>WHAT IS FIXED</span>
|
||||
<b>T0 · checkpoint · prompt</b>
|
||||
<p>这条带不覆盖 generation-tape uncertainty。</p>
|
||||
</article>
|
||||
<article class="result">
|
||||
<span data-tb-robust-label>LENGTH / CODE</span>
|
||||
<b data-tb-robust-count>—</b>
|
||||
<p data-tb-robust-copy>—</p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<aside class="tb-note dark">
|
||||
<b>正确率的八条带都跨 0；长度出现 domain 反向交互</b>
|
||||
<p>
|
||||
Code 的 system-at-period 为 −130.9 tokens，Math 为 +25.1 tokens，两个
|
||||
selected-task bands 都不跨 0、方向却相反。“system 会让输出更短”不是可跨域外推的结论。
|
||||
</p>
|
||||
</aside>
|
||||
</section>
|
||||
|
||||
<section class="tb-panel" data-tb-panel="tasks" hidden>
|
||||
<div class="tb-panel-lead">
|
||||
<div><span>III / TASK EXPLORER</span><h4>平均差为 0，也可能是 fail→pass 与 pass→fail 抵消</h4></div>
|
||||
<p>
|
||||
每页 8 道题。P/F 是 T0 上的独立 evaluator 结果；末列显示所选 contrast 的
|
||||
success 差、长度差与共同 token 前缀。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="task-controls">
|
||||
<label><span>DOMAIN</span>
|
||||
<select data-tb-task-domain aria-label="选择逐题任务域">
|
||||
<option value="code">Code · HumanEval</option>
|
||||
<option value="math">Math · GSM8K</option>
|
||||
</select>
|
||||
</label>
|
||||
<label><span>CONTRAST</span>
|
||||
<select data-tb-task-contrast aria-label="选择逐题 contrast">
|
||||
<option value="period_at_s0">句点 − EOS · 无 system</option>
|
||||
<option value="period_at_s1">句点 − EOS · 有 system</option>
|
||||
<option value="system_at_eos">system on − off · EOS</option>
|
||||
<option value="system_at_period">system on − off · 句点</option>
|
||||
</select>
|
||||
</label>
|
||||
<article><span>VISIBLE TASKS</span><b data-tb-task-page-label>01–08 / 32</b></article>
|
||||
</div>
|
||||
|
||||
<div class="task-pages" role="group" aria-label="选择逐题页">
|
||||
{[0, 1, 2, 3].map((page) => (
|
||||
<button type="button" data-tb-task-page={page} aria-pressed={page === 0 ? "true" : "false"}>
|
||||
{String(page * 8 + 1).padStart(2, "0")}–{String(page * 8 + 8).padStart(2, "0")}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="task-table">
|
||||
<header><b>TASK</b>{conditions.map((condition) => <b>{condition}</b>)}<b>SELECTED CONTRAST</b></header>
|
||||
<div data-tb-task-rows></div>
|
||||
</div>
|
||||
|
||||
<div class="transition-cards" data-tb-transition-cards></div>
|
||||
|
||||
<aside class="tb-note">
|
||||
<b>四格合计 pass 不是模型标准分数</b>
|
||||
<p>
|
||||
Code 的 59/128 与 Math 的 71/128 都来自 <code>32 tasks × 4 conditions</code>；
|
||||
同一道题出现四次。逐题转移表才保留条件改变的方向。
|
||||
</p>
|
||||
</aside>
|
||||
</section>
|
||||
|
||||
<section class="tb-panel" data-tb-panel="tapes" hidden>
|
||||
<div class="tb-panel-lead">
|
||||
<div><span>IV / CROSSED TAPE DIAGNOSTIC</span><h4>4 道题 × 4 条 tape，不是 16 道独立题</h4></div>
|
||||
<p>
|
||||
行是预先固定的题，列是 T0–T3。先在题内横向看 tape range，再在 tape 内纵向看
|
||||
task range；两种变化不能揉成一个普通样本方差。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="tape-controls">
|
||||
<label><span>DOMAIN</span>
|
||||
<select data-tb-tape-domain aria-label="选择随机带诊断域">
|
||||
<option value="code">Code · HumanEval</option>
|
||||
<option value="math">Math · GSM8K</option>
|
||||
</select>
|
||||
</label>
|
||||
<label><span>CONTRAST</span>
|
||||
<select data-tb-tape-contrast aria-label="选择随机带诊断 contrast">
|
||||
<option value="period_at_s0">句点 − EOS · 无 system</option>
|
||||
<option value="period_at_s1">句点 − EOS · 有 system</option>
|
||||
<option value="system_at_eos">system on − off · EOS</option>
|
||||
<option value="system_at_period">system on − off · 句点</option>
|
||||
</select>
|
||||
</label>
|
||||
<label><span>METRIC</span>
|
||||
<select data-tb-tape-metric aria-label="选择随机带诊断指标">
|
||||
<option value="success">Correctness Δ</option>
|
||||
<option value="tokens">Length Δ</option>
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="tape-matrix">
|
||||
<header><b>TASK ↓ / TAPE →</b><b>T0</b><b>T1</b><b>T2</b><b>T3</b><b>TAPE RANGE</b></header>
|
||||
<div data-tb-tape-rows></div>
|
||||
<footer data-tb-tape-means></footer>
|
||||
</div>
|
||||
|
||||
<div class="tape-reading">
|
||||
<article><span>MEAN TASK RANGE WITHIN TAPE</span><b data-tb-task-range>—</b><p>固定一条 tape，四题之间的 contrast 跨度</p></article>
|
||||
<article><span>MEAN TAPE RANGE WITHIN TASK</span><b data-tb-tape-range>—</b><p>固定一道题，四条 tape 之间的 contrast 跨度</p></article>
|
||||
<article class="result"><span>INDEPENDENCE</span><b>4 crossed tasks</b><p>不是 16 个独立观测；只做敏感性诊断</p></article>
|
||||
</div>
|
||||
|
||||
<aside class="tb-note dark">
|
||||
<b>Code 长度方向更稳，Math correctness 对 tape 更敏感</b>
|
||||
<p>
|
||||
在四题诊断子集上，Code 的 system-at-period 四条 tape 都为负；Math 同一长度
|
||||
contrast 四条都为正。但 Math correctness 的 system-at-EOS 在 T0–T3 间从 −0.25 到 +0.50。
|
||||
</p>
|
||||
</aside>
|
||||
</section>
|
||||
|
||||
<section class="tb-panel" data-tb-panel="audit" hidden>
|
||||
<div class="tb-panel-lead">
|
||||
<div><span>V / EVIDENCE AUDIT</span><h4>先锁 trajectory，再打开 gold；偏离也写进证据链</h4></div>
|
||||
<p>
|
||||
formal、独立 evaluator、replay 与 analysis 各自有文件 hash。重放检查的不只是
|
||||
headline，而是每格 12 个冻结字段。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="evidence-pipeline">
|
||||
<article><span>01 / FREEZE</span><b>64 tasks · 256 prompts</b><p>source、tape、contrast、bootstrap seed</p></article>
|
||||
<i>→</i>
|
||||
<article><span>02 / GENERATE</span><b>352 trajectories</b><p>88/88 runs 不消费 PyTorch RNG</p></article>
|
||||
<i>→</i>
|
||||
<article><span>03 / EVALUATE</span><b>networkless sandbox</b><p>停止、覆盖、正确、失败分账</p></article>
|
||||
<i>→</i>
|
||||
<article class="result"><span>04 / REPLAY</span><b>64 / 64 exact</b><p>全新进程 · 十二字段</p></article>
|
||||
</div>
|
||||
|
||||
<div class="replay-fields">
|
||||
{Object.entries(lab.reproduction.by_field).map(([field, count]) => (
|
||||
<article>
|
||||
<span>{String(field).replaceAll("_", " ").toUpperCase()}</span>
|
||||
<b>{String(count)} / 64</b><i>EXACT</i>
|
||||
</article>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="failure-ledger">
|
||||
<article>
|
||||
<header><span>CODE / 128 T0 OUTPUTS</span><b>59 pass</b></header>
|
||||
{Object.entries(lab.outcomes.code).map(([name, count]) => (
|
||||
<div><span>{name.replaceAll("_", " ")}</span><i><u style={`--share:${Number(count) / 128}`}></u></i><b>{String(count)}</b></div>
|
||||
))}
|
||||
</article>
|
||||
<article>
|
||||
<header><span>MATH / 128 T0 OUTPUTS</span><b>71 exact</b></header>
|
||||
{Object.entries(lab.outcomes.math).map(([name, count]) => (
|
||||
<div><span>{name.replaceAll("_", " ")}</span><i><u style={`--share:${Number(count) / 128}`}></u></i><b>{String(count)}</b></div>
|
||||
))}
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<div class="deviation-ledger">
|
||||
{lab.deviations.map((deviation: any, index: number) => (
|
||||
<article class={deviation.severity}>
|
||||
<span>{String(index + 1).padStart(2, "0")} / {deviation.severity.replaceAll("-", " ").toUpperCase()}</span>
|
||||
<b>{deviation.id.replaceAll("-", " ")}</b>
|
||||
<p>{deviation.summary}</p>
|
||||
</article>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="artifact-chain">
|
||||
{Object.entries(lab.artifactHashes).map(([name, hash], index) => (
|
||||
<article>
|
||||
<span>{String(index + 1).padStart(2, "0")} / {name.toUpperCase()}</span>
|
||||
<b>{String(hash).slice(0, 12)}…{String(hash).slice(-8)}</b>
|
||||
</article>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<aside class="tb-note">
|
||||
<b>gold 加载时机是本轮明确报告的流程偏离</b>
|
||||
<p>
|
||||
复用 runner 在生成进程开始前加载 gold，只用于文本生成结束后的窄
|
||||
<code>task_score</code>；gold 不进入 prompt、logits、CDF、tape 或任务选择，权威
|
||||
evaluator 仍独立运行。它没有已知 trajectory 因果路径，但后续 runner 应彻底删除这条依赖。
|
||||
</p>
|
||||
</aside>
|
||||
</section>
|
||||
|
||||
<figcaption>
|
||||
<b>证据边界</b>
|
||||
<span>
|
||||
两个 domain 各 32 道预选题，不是完整 benchmark；selected-task band 固定 T0，
|
||||
不覆盖 generation-tape uncertainty；句点是 counterfactual，不是官方聊天格式。
|
||||
</span>
|
||||
<code>FORMAL ea0607…1809 · EVAL 82b2fc…77ab · REPLAY 64/64</code>
|
||||
</figcaption>
|
||||
|
||||
<script is:inline type="application/json" data-tb-data set:html={json}></script>
|
||||
</figure>
|
||||
|
||||
<script>
|
||||
document.querySelectorAll<HTMLElement>("[data-task-bootstrap-lab]").forEach((root) => {
|
||||
const payload = root.querySelector<HTMLScriptElement>("[data-tb-data]");
|
||||
if (!payload) return;
|
||||
const data = JSON.parse(payload.textContent ?? "{}");
|
||||
const conditionLabels: Record<string, string> = {
|
||||
s0_eos: "无 system · EOS",
|
||||
s1_eos: "有 system · EOS",
|
||||
s0_period: "无 system · 句点",
|
||||
s1_period: "有 system · 句点",
|
||||
};
|
||||
const contrastLabels: Record<string, string> = {
|
||||
period_at_s0: "句点 − EOS · 无 system",
|
||||
period_at_s1: "句点 − EOS · 有 system",
|
||||
system_at_eos: "system on − off · EOS",
|
||||
system_at_period: "system on − off · 句点",
|
||||
};
|
||||
const one = <T extends Element>(selector: string) => root.querySelector<T>(selector);
|
||||
const all = <T extends Element>(selector: string) => [...root.querySelectorAll<T>(selector)];
|
||||
const set = (selector: string, value: string) => {
|
||||
const node = one<HTMLElement>(selector);
|
||||
if (node) node.textContent = value;
|
||||
};
|
||||
const signed = (value: number, digits = 2) => (
|
||||
`${value > 0 ? "+" : ""}${value.toFixed(digits)}`
|
||||
);
|
||||
|
||||
const tabButtons = all<HTMLButtonElement>("[data-tb-tab]");
|
||||
const panels = all<HTMLElement>("[data-tb-panel]");
|
||||
tabButtons.forEach((button, index) => {
|
||||
button.addEventListener("click", () => {
|
||||
const target = button.dataset.tbTab;
|
||||
tabButtons.forEach((candidate) => {
|
||||
const active = candidate === button;
|
||||
candidate.setAttribute("aria-selected", String(active));
|
||||
candidate.tabIndex = active ? 0 : -1;
|
||||
});
|
||||
panels.forEach((panel) => {
|
||||
panel.hidden = panel.dataset.tbPanel !== target;
|
||||
});
|
||||
});
|
||||
button.addEventListener("keydown", (event) => {
|
||||
if (!["ArrowLeft", "ArrowRight"].includes(event.key)) return;
|
||||
event.preventDefault();
|
||||
const delta = event.key === "ArrowRight" ? 1 : -1;
|
||||
const target = tabButtons[(index + delta + tabButtons.length) % tabButtons.length];
|
||||
target.click();
|
||||
target.focus();
|
||||
});
|
||||
});
|
||||
|
||||
let bootstrapDomain: "code" | "math" = "code";
|
||||
let bootstrapMetric = "fixed_budget_success";
|
||||
const domainButtons = all<HTMLButtonElement>("[data-tb-domain]");
|
||||
const metricButtons = all<HTMLButtonElement>("[data-tb-metric]");
|
||||
const renderBootstrap = () => {
|
||||
const cards = one<HTMLElement>("[data-tb-condition-cards]");
|
||||
if (cards) {
|
||||
cards.replaceChildren();
|
||||
data.conditionTable[bootstrapDomain].forEach((row: any) => {
|
||||
const article = document.createElement("article");
|
||||
const label = document.createElement("span");
|
||||
label.textContent = conditionLabels[row.condition];
|
||||
const value = document.createElement("b");
|
||||
value.textContent = `${row.success} / 32 pass`;
|
||||
const length = document.createElement("p");
|
||||
length.textContent = `${row.meanTokens.toFixed(1)} tokens · ${row.naturalEos}/32 EOS`;
|
||||
article.append(label, value, length);
|
||||
cards.append(article);
|
||||
});
|
||||
}
|
||||
const forest = one<HTMLElement>("[data-tb-forest]");
|
||||
if (forest) {
|
||||
forest.replaceChildren();
|
||||
Object.entries(data.contrasts[bootstrapDomain]).forEach(([name, contrast]: [string, any]) => {
|
||||
const metric = contrast.metrics[bootstrapMetric];
|
||||
const scale = bootstrapMetric === "generated_tokens" ? 210 : 0.42;
|
||||
const position = (value: number) => Math.max(1, Math.min(99, 50 + value / scale * 50));
|
||||
const row = document.createElement("article");
|
||||
const label = document.createElement("span");
|
||||
label.textContent = contrastLabels[name];
|
||||
const track = document.createElement("i");
|
||||
const band = document.createElement("u");
|
||||
band.style.setProperty("--band-left", `${position(metric.band.p2_5)}%`);
|
||||
band.style.setProperty("--band-right", `${position(metric.band.p97_5)}%`);
|
||||
band.className = metric.band.p2_5 > 0 || metric.band.p97_5 < 0 ? "clear" : "crosses";
|
||||
const point = document.createElement("em");
|
||||
point.style.setProperty("--point", `${position(metric.point)}%`);
|
||||
track.append(band, point);
|
||||
const value = document.createElement("b");
|
||||
value.textContent = bootstrapMetric === "generated_tokens"
|
||||
? `${signed(metric.point, 1)} tok`
|
||||
: signed(metric.point, 3);
|
||||
const interval = document.createElement("small");
|
||||
interval.textContent = `[${signed(metric.band.p2_5, bootstrapMetric === "generated_tokens" ? 1 : 3)}, ${signed(metric.band.p97_5, bootstrapMetric === "generated_tokens" ? 1 : 3)}]`;
|
||||
row.append(label, track, value, interval);
|
||||
forest.append(row);
|
||||
});
|
||||
}
|
||||
const clear = Object.values(data.contrasts[bootstrapDomain]).filter((contrast: any) => {
|
||||
const band = contrast.metrics[bootstrapMetric].band;
|
||||
return band.p2_5 > 0 || band.p97_5 < 0;
|
||||
}).length;
|
||||
set(
|
||||
"[data-tb-robust-label]",
|
||||
`${bootstrapMetric === "generated_tokens" ? "LENGTH" : "CORRECTNESS"} / ${bootstrapDomain.toUpperCase()}`,
|
||||
);
|
||||
set("[data-tb-robust-count]", `${clear} / 4 bands 不跨 0`);
|
||||
set(
|
||||
"[data-tb-robust-copy]",
|
||||
bootstrapMetric === "fixed_budget_success"
|
||||
? "所有 correctness 带都跨 0，点估计不能升级成稳定能力结论。"
|
||||
: bootstrapDomain === "code"
|
||||
? "period-at-s1 与 system-at-period 明确偏负；Code 输出显著缩短。"
|
||||
: "只有 system-at-period 明确偏正；Math 输出反而延长。",
|
||||
);
|
||||
};
|
||||
domainButtons.forEach((button) => button.addEventListener("click", () => {
|
||||
bootstrapDomain = (button.dataset.tbDomain ?? "code") as "code" | "math";
|
||||
domainButtons.forEach((candidate) => candidate.setAttribute(
|
||||
"aria-pressed",
|
||||
String(candidate === button),
|
||||
));
|
||||
renderBootstrap();
|
||||
}));
|
||||
metricButtons.forEach((button) => button.addEventListener("click", () => {
|
||||
bootstrapMetric = button.dataset.tbMetric ?? "fixed_budget_success";
|
||||
metricButtons.forEach((candidate) => candidate.setAttribute(
|
||||
"aria-pressed",
|
||||
String(candidate === button),
|
||||
));
|
||||
renderBootstrap();
|
||||
}));
|
||||
renderBootstrap();
|
||||
|
||||
const taskDomainSelect = one<HTMLSelectElement>("[data-tb-task-domain]");
|
||||
const taskContrastSelect = one<HTMLSelectElement>("[data-tb-task-contrast]");
|
||||
const taskPageButtons = all<HTMLButtonElement>("[data-tb-task-page]");
|
||||
let taskPage = 0;
|
||||
const renderTasks = () => {
|
||||
const domain = taskDomainSelect?.value ?? "code";
|
||||
const contrast = taskContrastSelect?.value ?? "period_at_s0";
|
||||
const rows = data.tasks[domain].slice(taskPage * 8, taskPage * 8 + 8);
|
||||
set(
|
||||
"[data-tb-task-page-label]",
|
||||
`${String(taskPage * 8 + 1).padStart(2, "0")}–${String(taskPage * 8 + 8).padStart(2, "0")} / 32`,
|
||||
);
|
||||
const container = one<HTMLElement>("[data-tb-task-rows]");
|
||||
if (container) {
|
||||
container.replaceChildren();
|
||||
rows.forEach((task: any) => {
|
||||
const row = document.createElement("article");
|
||||
const label = document.createElement("span");
|
||||
label.innerHTML = `<small>${String(task.index + 1).padStart(2, "0")}</small><b>${task.id}</b>`;
|
||||
row.append(label);
|
||||
Object.values(task.conditions).forEach((condition: any) => {
|
||||
const cell = document.createElement("i");
|
||||
cell.className = condition.success ? "pass" : "fail";
|
||||
cell.innerHTML = `<b>${condition.success ? "P" : "F"}</b><small>${condition.tokens}t</small>`;
|
||||
cell.title = `${condition.outcome} · ${condition.tokens} tokens`;
|
||||
row.append(cell);
|
||||
});
|
||||
const delta = task.contrasts[contrast];
|
||||
const summary = document.createElement("strong");
|
||||
summary.className = delta.successDelta > 0 ? "positive" : delta.successDelta < 0 ? "negative" : "zero";
|
||||
summary.innerHTML = `<b>${signed(delta.successDelta, 0)} pass · ${signed(delta.tokenDelta, 0)} tok</b><small>${delta.commonPrefixTokens} token 共同前缀</small>`;
|
||||
row.append(summary);
|
||||
container.append(row);
|
||||
});
|
||||
}
|
||||
const transition = data.contrasts[domain][contrast]
|
||||
.metrics.fixed_budget_success.transition;
|
||||
const cards = one<HTMLElement>("[data-tb-transition-cards]");
|
||||
if (cards && transition) {
|
||||
cards.replaceChildren();
|
||||
[
|
||||
["FAIL → PASS", transition.fail_to_pass, "positive"],
|
||||
["PASS → FAIL", transition.pass_to_fail, "negative"],
|
||||
["PASS → PASS", transition.pass_to_pass, "stable"],
|
||||
["FAIL → FAIL", transition.fail_to_fail, "stable"],
|
||||
].forEach(([label, count, className]) => {
|
||||
const article = document.createElement("article");
|
||||
article.className = String(className);
|
||||
article.innerHTML = `<span>${label}</span><b>${count} / 32</b>`;
|
||||
cards.append(article);
|
||||
});
|
||||
}
|
||||
};
|
||||
taskDomainSelect?.addEventListener("change", () => {
|
||||
taskPage = 0;
|
||||
taskPageButtons.forEach((button, index) => button.setAttribute("aria-pressed", String(index === 0)));
|
||||
renderTasks();
|
||||
});
|
||||
taskContrastSelect?.addEventListener("change", renderTasks);
|
||||
taskPageButtons.forEach((button) => button.addEventListener("click", () => {
|
||||
taskPage = Number(button.dataset.tbTaskPage ?? 0);
|
||||
taskPageButtons.forEach((candidate) => candidate.setAttribute(
|
||||
"aria-pressed",
|
||||
String(candidate === button),
|
||||
));
|
||||
renderTasks();
|
||||
}));
|
||||
renderTasks();
|
||||
|
||||
const tapeDomain = one<HTMLSelectElement>("[data-tb-tape-domain]");
|
||||
const tapeContrast = one<HTMLSelectElement>("[data-tb-tape-contrast]");
|
||||
const tapeMetric = one<HTMLSelectElement>("[data-tb-tape-metric]");
|
||||
const renderTapes = () => {
|
||||
const domain = tapeDomain?.value ?? "code";
|
||||
const contrast = tapeContrast?.value ?? "period_at_s0";
|
||||
const metricName = tapeMetric?.value ?? "success";
|
||||
const domainData = data.diagnostic[domain];
|
||||
const metric = domainData.contrasts[contrast][metricName];
|
||||
const rows = one<HTMLElement>("[data-tb-tape-rows]");
|
||||
if (rows) {
|
||||
rows.replaceChildren();
|
||||
metric.matrix.forEach((values: number[], index: number) => {
|
||||
const row = document.createElement("article");
|
||||
const label = document.createElement("span");
|
||||
label.textContent = domainData.sourceIds[index];
|
||||
row.append(label);
|
||||
values.forEach((value: number) => {
|
||||
const cell = document.createElement("b");
|
||||
cell.className = value > 0 ? "positive" : value < 0 ? "negative" : "zero";
|
||||
cell.style.setProperty("--strength", String(Math.min(1, Math.abs(value) / (metricName === "success" ? 1 : 200))));
|
||||
cell.textContent = metricName === "success" ? signed(value, 2) : signed(value, 0);
|
||||
row.append(cell);
|
||||
});
|
||||
const range = Math.max(...values) - Math.min(...values);
|
||||
const output = document.createElement("strong");
|
||||
output.textContent = metricName === "success" ? range.toFixed(2) : `${range.toFixed(0)} tok`;
|
||||
row.append(output);
|
||||
rows.append(row);
|
||||
});
|
||||
}
|
||||
const means = one<HTMLElement>("[data-tb-tape-means]");
|
||||
if (means) {
|
||||
means.replaceChildren();
|
||||
const label = document.createElement("span");
|
||||
label.textContent = "TAPE MEAN";
|
||||
means.append(label);
|
||||
domainData.tapes.forEach((tape: string) => {
|
||||
const value = metric.tapeMeans[tape];
|
||||
const cell = document.createElement("b");
|
||||
cell.textContent = metricName === "success" ? signed(value, 2) : signed(value, 1);
|
||||
means.append(cell);
|
||||
});
|
||||
const note = document.createElement("strong");
|
||||
note.textContent = "descriptive";
|
||||
means.append(note);
|
||||
}
|
||||
set(
|
||||
"[data-tb-task-range]",
|
||||
`${metric.taskRangeWithinTape.mean.toFixed(metricName === "success" ? 2 : 1)}${metricName === "tokens" ? " tok" : ""}`,
|
||||
);
|
||||
set(
|
||||
"[data-tb-tape-range]",
|
||||
`${metric.tapeRangeWithinTask.mean.toFixed(metricName === "success" ? 2 : 1)}${metricName === "tokens" ? " tok" : ""}`,
|
||||
);
|
||||
};
|
||||
[tapeDomain, tapeContrast, tapeMetric].forEach((control) => (
|
||||
control?.addEventListener("change", renderTapes)
|
||||
));
|
||||
renderTapes();
|
||||
});
|
||||
</script>
|
||||
|
||||
<style is:global>
|
||||
[data-task-bootstrap-lab] { margin: 1.6rem 0 0; overflow: hidden; border: 1px solid var(--line); background: #f7f3ea; }
|
||||
[data-task-bootstrap-lab] .tb-head { display: grid; grid-template-columns: 1.08fr .92fr; gap: 1.5rem; padding: 1.55rem; color: #edf3f1; background: #263f47; }
|
||||
[data-task-bootstrap-lab] .tb-head p { margin: 0; font-size: .62rem; line-height: 1.65; }
|
||||
[data-task-bootstrap-lab] .tb-head > div > p { color: #8fcbbf; font: 690 .48rem/1.2 var(--font-mono); letter-spacing: .08em; }
|
||||
[data-task-bootstrap-lab] .tb-head h3 { margin: .6rem 0 0; max-width: 28ch; color: #edf3f1; font: 760 1.22rem/1.14 var(--font-display); }
|
||||
[data-task-bootstrap-lab] .tb-head code { color: #f0c7aa; font-size: .53rem; }
|
||||
[data-task-bootstrap-lab] .tb-ledger { display: grid; grid-template-columns: repeat(6,1fr); gap: 1px; border-bottom: 1px solid var(--line); background: var(--line); }
|
||||
[data-task-bootstrap-lab] .tb-ledger article { min-width: 0; padding: .82rem; background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .tb-ledger article.pass { background: #dcebe5; }
|
||||
[data-task-bootstrap-lab] .tb-ledger span { display: block; color: #687772; font: .42rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .tb-ledger b { display: block; margin-top: .38rem; font: 760 .72rem/1.1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .tb-ledger p { margin: .32rem 0 0; color: #717975; font-size: .45rem; line-height: 1.4; }
|
||||
[data-task-bootstrap-lab] .tb-tabs { display: grid; grid-template-columns: repeat(5,1fr); gap: 1px; border-bottom: 1px solid var(--line); background: var(--line); }
|
||||
[data-task-bootstrap-lab] .tb-tabs button { min-width: 0; padding: .86rem; text-align: left; color: #52615e; border: 0; background: #e5e0d5; cursor: pointer; }
|
||||
[data-task-bootstrap-lab] .tb-tabs button[aria-selected="true"] { color: #f0f5f3; background: #2e776c; }
|
||||
[data-task-bootstrap-lab] .tb-tabs span, [data-task-bootstrap-lab] .tb-tabs b, [data-task-bootstrap-lab] .tb-tabs small { display: block; }
|
||||
[data-task-bootstrap-lab] .tb-tabs span { color: var(--orange); font: 730 .42rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .tb-tabs button[aria-selected="true"] span { color: #f2c6a8; }
|
||||
[data-task-bootstrap-lab] .tb-tabs b { margin-top: .38rem; font-size: .57rem; }
|
||||
[data-task-bootstrap-lab] .tb-tabs small { margin-top: .25rem; opacity: .65; font: .39rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .tb-panel { padding: 1.25rem; }
|
||||
[data-task-bootstrap-lab] .tb-panel-lead { display: grid; grid-template-columns: 1.08fr .92fr; gap: 1.4rem; margin-bottom: 1rem; }
|
||||
[data-task-bootstrap-lab] .tb-panel-lead span { color: var(--orange); font: 700 .46rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .tb-panel-lead h4 { margin: .42rem 0 0; font: 750 .96rem/1.15 var(--font-display); }
|
||||
[data-task-bootstrap-lab] .tb-panel-lead p { margin: 0; color: #63706c; font-size: .57rem; line-height: 1.65; }
|
||||
[data-task-bootstrap-lab] .pairing-compare { display: grid; grid-template-columns: 1fr 1fr; gap: .8rem; }
|
||||
[data-task-bootstrap-lab] .pairing-compare article { padding: .9rem; border: 1px solid var(--line); background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .pairing-compare article.active { color: #edf3f1; border: 0; background: #2f776c; }
|
||||
[data-task-bootstrap-lab] .pairing-compare > article > span { font: 690 .43rem/1.2 var(--font-mono); opacity: .72; }
|
||||
[data-task-bootstrap-lab] .pairing-compare p { margin: .55rem 0 0; opacity: .72; font-size: .49rem; }
|
||||
[data-task-bootstrap-lab] .stream-row { display: grid; grid-template-columns: .9fr auto repeat(4,1fr); gap: .35rem; align-items: center; margin-top: .65rem; }
|
||||
[data-task-bootstrap-lab] .stream-row i, [data-task-bootstrap-lab] .stream-row u { display: grid; place-items: center; height: 2.2rem; text-decoration: none; font: 730 .52rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .stream-row i { color: #f4f0e8; background: #b96b45; font-style: normal; }
|
||||
[data-task-bootstrap-lab] .pairing-compare article:not(.active) .stream-row i { color: #fff; }
|
||||
[data-task-bootstrap-lab] .stream-row u { color: #29434a; background: #dce8e4; }
|
||||
[data-task-bootstrap-lab] .stream-row b { color: #d89a76; text-align: center; }
|
||||
[data-task-bootstrap-lab] .sampler-pipeline, [data-task-bootstrap-lab] .evidence-pipeline { display: grid; grid-template-columns: 1fr auto 1fr auto 1fr auto 1fr; gap: .55rem; align-items: center; margin-top: 1rem; }
|
||||
[data-task-bootstrap-lab] .sampler-pipeline article, [data-task-bootstrap-lab] .evidence-pipeline article { min-width: 0; padding: .8rem; border: 1px solid var(--line); background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .sampler-pipeline article.uniform, [data-task-bootstrap-lab] .evidence-pipeline article.result { color: #edf4f1; border: 0; background: #2f776c; }
|
||||
[data-task-bootstrap-lab] .sampler-pipeline span, [data-task-bootstrap-lab] .evidence-pipeline span { font: .4rem/1.2 var(--font-mono); opacity: .7; }
|
||||
[data-task-bootstrap-lab] .sampler-pipeline b, [data-task-bootstrap-lab] .evidence-pipeline b { display: block; margin-top: .35rem; font: 720 .61rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .sampler-pipeline p, [data-task-bootstrap-lab] .evidence-pipeline p { margin: .32rem 0 0; opacity: .7; font-size: .43rem; }
|
||||
[data-task-bootstrap-lab] .sampler-pipeline > i, [data-task-bootstrap-lab] .evidence-pipeline > i { color: var(--orange); font: 760 .72rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .uniform-demo { margin-top: 1rem; overflow: hidden; border: 1px solid var(--line); background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .uniform-demo > header { display: flex; justify-content: space-between; gap: 1rem; padding: .75rem .85rem; color: #e8efed; background: #29434a; }
|
||||
[data-task-bootstrap-lab] .uniform-demo > header span { display: block; color: #91c9be; font: .4rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .uniform-demo > header b { display: block; margin-top: .3rem; font: 710 .58rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .uniform-demo > header p { margin: 0; font-size: .45rem; }
|
||||
[data-task-bootstrap-lab] .uniform-bars { display: grid; grid-template-columns: repeat(8,1fr); gap: 1px; height: 9rem; padding: .75rem; background: #d9d4ca; }
|
||||
[data-task-bootstrap-lab] .uniform-bars article { display: grid; grid-template-rows: 1fr auto auto auto; min-width: 0; padding: .3rem; background: #f7f3ea; text-align: center; }
|
||||
[data-task-bootstrap-lab] .uniform-bars i { align-self: end; width: 58%; height: calc(var(--u) * 100%); min-height: .25rem; margin: 0 auto; background: linear-gradient(#8fc9bd,#2f776c); }
|
||||
[data-task-bootstrap-lab] .uniform-bars span, [data-task-bootstrap-lab] .uniform-bars code { margin-top: .18rem; color: #6a7773; font: .35rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .uniform-bars b { margin-top: .18rem; font: 700 .43rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .token-lanes { display: grid; gap: 1px; padding: 0 .75rem .75rem; background: #d9d4ca; }
|
||||
[data-task-bootstrap-lab] .token-lanes article { display: grid; grid-template-columns: 1.2fr 4fr; gap: 1px; background: #d9d4ca; }
|
||||
[data-task-bootstrap-lab] .token-lanes header { padding: .55rem; background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .token-lanes header span, [data-task-bootstrap-lab] .token-lanes header b { display: block; }
|
||||
[data-task-bootstrap-lab] .token-lanes header span { color: var(--orange); font: .38rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .token-lanes header b { margin-top: .3rem; font-size: .48rem; }
|
||||
[data-task-bootstrap-lab] .token-lanes article > div { display: grid; grid-template-columns: repeat(8,1fr); gap: 1px; background: #d9d4ca; }
|
||||
[data-task-bootstrap-lab] .token-lanes i { display: grid; place-items: center; min-width: 0; padding: .45rem .15rem; background: #fffaf2; font: 680 .39rem/1 var(--font-mono); font-style: normal; }
|
||||
[data-task-bootstrap-lab] .token-lanes small { display: block; margin-bottom: .23rem; color: #77817d; font-size: .31rem; }
|
||||
[data-task-bootstrap-lab] .tb-note { margin-top: 1rem; padding: .85rem 1rem; border-left: .24rem solid var(--orange); background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .tb-note.dark { color: #e8efed; border-left-color: #e0a17c; background: #29434a; }
|
||||
[data-task-bootstrap-lab] .tb-note b { font: 710 .58rem/1.3 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .tb-note p { margin: .42rem 0 0; opacity: .76; font-size: .51rem; line-height: 1.6; }
|
||||
[data-task-bootstrap-lab] .tb-switch-row { display: flex; justify-content: space-between; gap: 1rem; padding: .65rem; border: 1px solid var(--line); background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .tb-switch-row > div { display: flex; gap: .35rem; }
|
||||
[data-task-bootstrap-lab] .tb-switch-row button, [data-task-bootstrap-lab] .task-pages button { padding: .55rem .75rem; color: #53625f; border: 1px solid var(--line); background: #fffaf2; font: 690 .45rem/1 var(--font-mono); cursor: pointer; }
|
||||
[data-task-bootstrap-lab] .tb-switch-row button[aria-pressed="true"], [data-task-bootstrap-lab] .task-pages button[aria-pressed="true"] { color: #edf3f1; border-color: #2f776c; background: #2f776c; }
|
||||
[data-task-bootstrap-lab] .condition-cards { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
|
||||
[data-task-bootstrap-lab] .condition-cards article { padding: .75rem; background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .condition-cards span { color: #66736f; font: .41rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .condition-cards b { display: block; margin-top: .38rem; color: #2d6d63; font: 740 .63rem/1.1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .condition-cards p { margin: .3rem 0 0; color: #717b77; font-size: .44rem; }
|
||||
[data-task-bootstrap-lab] .forest { margin-top: 1rem; overflow: hidden; border: 1px solid var(--line); }
|
||||
[data-task-bootstrap-lab] .forest > header { display: grid; grid-template-columns: 1fr auto 1fr; padding: .6rem .8rem; color: #e9f0ee; background: #29434a; font: .4rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .forest > header span:last-child { text-align: right; }
|
||||
[data-task-bootstrap-lab] .forest > header b { color: #a8d2c9; }
|
||||
[data-task-bootstrap-lab] .forest [data-tb-forest] > article { display: grid; grid-template-columns: 1.25fr 2.8fr .65fr 1.2fr; gap: .65rem; align-items: center; padding: .72rem .8rem; border-bottom: 1px solid var(--line); background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .forest [data-tb-forest] > article:last-child { border-bottom: 0; }
|
||||
[data-task-bootstrap-lab] .forest article > span { font: 650 .46rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .forest article > i { position: relative; height: .38rem; background: linear-gradient(to right,#deb194 0 49.7%,#29434a 49.7% 50.3%,#a9d3ca 50.3% 100%); }
|
||||
[data-task-bootstrap-lab] .forest article u { position: absolute; top: 50%; left: var(--band-left); width: calc(var(--band-right) - var(--band-left)); height: .44rem; background: #596d68; transform: translateY(-50%); text-decoration: none; }
|
||||
[data-task-bootstrap-lab] .forest article u.clear { background: #2f776c; }
|
||||
[data-task-bootstrap-lab] .forest article em { position: absolute; top: 50%; left: var(--point); width: .75rem; height: .75rem; border: .13rem solid #eee9df; border-radius: 50%; background: #bd6c46; box-shadow: 0 0 0 1px #29434a; transform: translate(-50%,-50%); }
|
||||
[data-task-bootstrap-lab] .forest article > b { font: 730 .48rem/1.2 var(--font-mono); text-align: right; }
|
||||
[data-task-bootstrap-lab] .forest article > small { color: #687570; font: .37rem/1.3 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .bootstrap-reading, [data-task-bootstrap-lab] .tape-reading { display: grid; grid-template-columns: repeat(3,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
|
||||
[data-task-bootstrap-lab] .bootstrap-reading article, [data-task-bootstrap-lab] .tape-reading article { padding: .8rem; background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .bootstrap-reading article.result, [data-task-bootstrap-lab] .tape-reading article.result { color: #edf4f1; background: #2f776c; }
|
||||
[data-task-bootstrap-lab] .bootstrap-reading span, [data-task-bootstrap-lab] .tape-reading span { font: .4rem/1.2 var(--font-mono); opacity: .7; }
|
||||
[data-task-bootstrap-lab] .bootstrap-reading b, [data-task-bootstrap-lab] .tape-reading b { display: block; margin-top: .4rem; font: 730 .62rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .bootstrap-reading p, [data-task-bootstrap-lab] .tape-reading p { margin: .35rem 0 0; opacity: .7; font-size: .46rem; line-height: 1.45; }
|
||||
[data-task-bootstrap-lab] .task-controls, [data-task-bootstrap-lab] .tape-controls { display: grid; grid-template-columns: repeat(3,1fr); gap: 1px; border: 1px solid var(--line); background: var(--line); }
|
||||
[data-task-bootstrap-lab] .task-controls label, [data-task-bootstrap-lab] .task-controls article, [data-task-bootstrap-lab] .tape-controls label { padding: .72rem; background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .task-controls span, [data-task-bootstrap-lab] .tape-controls span { display: block; color: #6e7975; font: .4rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .task-controls select, [data-task-bootstrap-lab] .tape-controls select { width: 100%; margin-top: .38rem; padding: .42rem; border: 1px solid var(--line); background: #fffaf2; font: 650 .5rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .task-controls b { display: block; margin-top: .45rem; font: 730 .61rem/1.1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .task-pages { display: grid; grid-template-columns: repeat(4,1fr); gap: .35rem; margin-top: .7rem; }
|
||||
[data-task-bootstrap-lab] .task-table, [data-task-bootstrap-lab] .tape-matrix { margin-top: .75rem; overflow: hidden; border: 1px solid var(--line); }
|
||||
[data-task-bootstrap-lab] .task-table > header, [data-task-bootstrap-lab] .task-table [data-tb-task-rows] > article { display: grid; grid-template-columns: 1.4fr repeat(4,.55fr) 1.75fr; gap: 1px; background: var(--line); }
|
||||
[data-task-bootstrap-lab] .task-table > header > * { padding: .55rem .35rem; color: #e9f0ee; background: #29434a; font: 620 .36rem/1.2 var(--font-mono); text-align: center; }
|
||||
[data-task-bootstrap-lab] .task-table > header > *:first-child, [data-task-bootstrap-lab] .task-table > header > *:last-child { text-align: left; }
|
||||
[data-task-bootstrap-lab] .task-table [data-tb-task-rows] article > span { display: flex; gap: .5rem; align-items: center; min-width: 0; padding: .55rem; background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .task-table article > span small { color: var(--orange); font: .36rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .task-table article > span b { overflow: hidden; font: 650 .43rem/1.2 var(--font-mono); text-overflow: ellipsis; white-space: nowrap; }
|
||||
[data-task-bootstrap-lab] .task-table article > i { display: grid; place-items: center; padding: .4rem; background: #f0dfd5; font-style: normal; }
|
||||
[data-task-bootstrap-lab] .task-table article > i.pass { background: #dcebe5; }
|
||||
[data-task-bootstrap-lab] .task-table article > i b { font: 760 .52rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .task-table article > i small { margin-top: .22rem; color: #6b7773; font: .32rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .task-table article > strong { display: grid; align-content: center; padding: .45rem .55rem; background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .task-table article > strong.positive { background: #dcebe5; }
|
||||
[data-task-bootstrap-lab] .task-table article > strong.negative { background: #f0dfd5; }
|
||||
[data-task-bootstrap-lab] .task-table article > strong b { font: 700 .43rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .task-table article > strong small { margin-top: .25rem; color: #67736f; font: .33rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .transition-cards { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: .75rem; border: 1px solid var(--line); background: var(--line); }
|
||||
[data-task-bootstrap-lab] .transition-cards article { padding: .7rem; background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .transition-cards article.positive { background: #dcebe5; }
|
||||
[data-task-bootstrap-lab] .transition-cards article.negative { background: #f0dfd5; }
|
||||
[data-task-bootstrap-lab] .transition-cards span { color: #65736e; font: .4rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .transition-cards b { display: block; margin-top: .35rem; font: 730 .6rem/1.1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .tape-matrix > header, [data-task-bootstrap-lab] .tape-matrix [data-tb-tape-rows] > article, [data-task-bootstrap-lab] .tape-matrix > footer { display: grid; grid-template-columns: 1.45fr repeat(4,.75fr) 1fr; gap: 1px; background: var(--line); }
|
||||
[data-task-bootstrap-lab] .tape-matrix > header > *, [data-task-bootstrap-lab] .tape-matrix > footer > * { padding: .58rem .4rem; color: #e8efed; background: #29434a; font: 650 .4rem/1.2 var(--font-mono); text-align: center; }
|
||||
[data-task-bootstrap-lab] .tape-matrix > header > *:first-child, [data-task-bootstrap-lab] .tape-matrix > footer > *:first-child { text-align: left; }
|
||||
[data-task-bootstrap-lab] .tape-matrix [data-tb-tape-rows] article > span, [data-task-bootstrap-lab] .tape-matrix [data-tb-tape-rows] article > strong { padding: .7rem .55rem; background: #eee9df; font: 650 .43rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .tape-matrix [data-tb-tape-rows] article > b { display: grid; place-items: center; color: #29433e; background: color-mix(in srgb,#8fc8bd calc(var(--strength) * 72%),#f7f3ea); font: 730 .5rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .tape-matrix [data-tb-tape-rows] article > b.negative { color: #5e3427; background: color-mix(in srgb,#dc9d78 calc(var(--strength) * 72%),#f7f3ea); }
|
||||
[data-task-bootstrap-lab] .tape-matrix [data-tb-tape-rows] article > b.zero { background: #f7f3ea; }
|
||||
[data-task-bootstrap-lab] .tape-matrix [data-tb-tape-rows] article > strong { text-align: center; }
|
||||
[data-task-bootstrap-lab] .replay-fields { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
|
||||
[data-task-bootstrap-lab] .replay-fields article { padding: .7rem; background: #dfece7; }
|
||||
[data-task-bootstrap-lab] .replay-fields span { display: block; color: #58716b; font: .36rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .replay-fields b { display: block; margin-top: .32rem; color: #28675c; font: 740 .58rem/1.1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .replay-fields i { display: block; margin-top: .23rem; color: #568178; font: .34rem/1 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .failure-ledger { display: grid; grid-template-columns: 1fr 1fr; gap: .8rem; margin-top: 1rem; }
|
||||
[data-task-bootstrap-lab] .failure-ledger > article { overflow: hidden; border: 1px solid var(--line); background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .failure-ledger header { display: flex; justify-content: space-between; padding: .7rem; color: #e8efed; background: #29434a; }
|
||||
[data-task-bootstrap-lab] .failure-ledger header span { font: .4rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .failure-ledger header b { font: 720 .5rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .failure-ledger article > div { display: grid; grid-template-columns: 1.2fr 2fr .35fr; gap: .55rem; align-items: center; padding: .55rem .7rem; border-bottom: 1px solid var(--line); }
|
||||
[data-task-bootstrap-lab] .failure-ledger article > div:last-child { border-bottom: 0; }
|
||||
[data-task-bootstrap-lab] .failure-ledger div > span { font: .4rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .failure-ledger div > i { height: .36rem; background: #d8d2c8; }
|
||||
[data-task-bootstrap-lab] .failure-ledger div > i u { display: block; width: calc(var(--share) * 100%); height: 100%; background: #2f776c; text-decoration: none; }
|
||||
[data-task-bootstrap-lab] .failure-ledger div > b { font: 700 .44rem/1 var(--font-mono); text-align: right; }
|
||||
[data-task-bootstrap-lab] .deviation-ledger { display: grid; grid-template-columns: 1fr 1fr; gap: .8rem; margin-top: 1rem; }
|
||||
[data-task-bootstrap-lab] .deviation-ledger article { padding: .85rem; border: 1px solid var(--line); background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .deviation-ledger article.reported-process-deviation { border-left: .25rem solid #bd6c46; }
|
||||
[data-task-bootstrap-lab] .deviation-ledger article.corrected-before-output { border-left: .25rem solid #2f776c; }
|
||||
[data-task-bootstrap-lab] .deviation-ledger span { color: var(--orange); font: .39rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .deviation-ledger b { display: block; margin-top: .38rem; font: 710 .58rem/1.2 var(--font-mono); text-transform: uppercase; }
|
||||
[data-task-bootstrap-lab] .deviation-ledger p { margin: .38rem 0 0; color: #69746f; font-size: .48rem; line-height: 1.55; }
|
||||
[data-task-bootstrap-lab] .artifact-chain { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
|
||||
[data-task-bootstrap-lab] .artifact-chain article { min-width: 0; padding: .65rem; background: #eee9df; }
|
||||
[data-task-bootstrap-lab] .artifact-chain span { color: var(--orange); font: .36rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] .artifact-chain b { display: block; margin-top: .32rem; overflow: hidden; font: 650 .4rem/1.2 var(--font-mono); text-overflow: ellipsis; }
|
||||
[data-task-bootstrap-lab] > figcaption { display: grid; grid-template-columns: auto 1fr auto; gap: 1rem; align-items: center; padding: .9rem 1.1rem; color: #e2eae8; background: #203a42; }
|
||||
[data-task-bootstrap-lab] > figcaption b { color: #8dc8bd; font: 720 .48rem/1.2 var(--font-mono); }
|
||||
[data-task-bootstrap-lab] > figcaption span { font-size: .5rem; line-height: 1.5; }
|
||||
[data-task-bootstrap-lab] > figcaption code { color: #dda988; font: .38rem/1.3 var(--font-mono); }
|
||||
@media (max-width: 900px) {
|
||||
[data-task-bootstrap-lab] .tb-head, [data-task-bootstrap-lab] .tb-panel-lead { grid-template-columns: 1fr; }
|
||||
[data-task-bootstrap-lab] .tb-ledger { grid-template-columns: repeat(3,1fr); }
|
||||
[data-task-bootstrap-lab] .tb-tabs { grid-template-columns: repeat(3,1fr); }
|
||||
[data-task-bootstrap-lab] .sampler-pipeline, [data-task-bootstrap-lab] .evidence-pipeline { grid-template-columns: 1fr; }
|
||||
[data-task-bootstrap-lab] .sampler-pipeline > i, [data-task-bootstrap-lab] .evidence-pipeline > i { transform: rotate(90deg); text-align: center; }
|
||||
[data-task-bootstrap-lab] .replay-fields, [data-task-bootstrap-lab] .artifact-chain { grid-template-columns: 1fr 1fr; }
|
||||
[data-task-bootstrap-lab] > figcaption { grid-template-columns: 1fr; }
|
||||
}
|
||||
@media (max-width: 640px) {
|
||||
[data-task-bootstrap-lab] .tb-panel { padding: .9rem; }
|
||||
[data-task-bootstrap-lab] .tb-head { padding: 1.2rem; }
|
||||
[data-task-bootstrap-lab] .tb-tabs, [data-task-bootstrap-lab] .pairing-compare,
|
||||
[data-task-bootstrap-lab] .tb-switch-row, [data-task-bootstrap-lab] .task-controls,
|
||||
[data-task-bootstrap-lab] .tape-controls, [data-task-bootstrap-lab] .bootstrap-reading,
|
||||
[data-task-bootstrap-lab] .tape-reading, [data-task-bootstrap-lab] .failure-ledger,
|
||||
[data-task-bootstrap-lab] .deviation-ledger { display: grid; grid-template-columns: 1fr; }
|
||||
[data-task-bootstrap-lab] .tb-switch-row > div { display: grid; grid-template-columns: 1fr 1fr; }
|
||||
[data-task-bootstrap-lab] .condition-cards, [data-task-bootstrap-lab] .transition-cards { grid-template-columns: 1fr 1fr; }
|
||||
[data-task-bootstrap-lab] .uniform-demo > header { display: grid; }
|
||||
[data-task-bootstrap-lab] .uniform-bars { grid-template-columns: repeat(4,1fr); height: auto; }
|
||||
[data-task-bootstrap-lab] .uniform-bars article { min-height: 6.5rem; }
|
||||
[data-task-bootstrap-lab] .token-lanes article { grid-template-columns: 1fr; }
|
||||
[data-task-bootstrap-lab] .token-lanes article > div { overflow-x: auto; }
|
||||
[data-task-bootstrap-lab] .forest [data-tb-forest] > article { grid-template-columns: 1fr; }
|
||||
[data-task-bootstrap-lab] .forest article > i { margin: .45rem 0; }
|
||||
[data-task-bootstrap-lab] .task-table, [data-task-bootstrap-lab] .tape-matrix { overflow-x: auto; }
|
||||
[data-task-bootstrap-lab] .task-table > header, [data-task-bootstrap-lab] .task-table [data-tb-task-rows] > article { min-width: 44rem; }
|
||||
[data-task-bootstrap-lab] .tape-matrix > header, [data-task-bootstrap-lab] .tape-matrix [data-tb-tape-rows] > article, [data-task-bootstrap-lab] .tape-matrix > footer { min-width: 36rem; }
|
||||
}
|
||||
</style>
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -7,6 +7,7 @@ import DeepSeekBehaviorLab from "@/components/DeepSeekBehaviorLab.astro";
|
||||
import DeepSeekCompletionDepthLab from "@/components/DeepSeekCompletionDepthLab.astro";
|
||||
import DeepSeekSamplingLab from "@/components/DeepSeekSamplingLab.astro";
|
||||
import DeepSeekCrossSourceSamplingLab from "@/components/DeepSeekCrossSourceSamplingLab.astro";
|
||||
import DeepSeekTaskBootstrapLab from "@/components/DeepSeekTaskBootstrapLab.astro";
|
||||
import { deepseekBranches, deepseekLedgers, deepseekPaperChain, deepseekWaves } from "@/data/deepseek";
|
||||
|
||||
const toc = [
|
||||
@@ -37,21 +38,22 @@ const toc = [
|
||||
["24", "completion-depth", "Chat：完成度与全深度"],
|
||||
["25", "sampling", "Chat：多种子采样稳健性"],
|
||||
["26", "cross-source-sampling", "Chat：跨题采样与统计单位"],
|
||||
["27", "branches", "别漏掉旁支"],
|
||||
["28", "audit", "事实、推导与教学模型"],
|
||||
["27", "task-bootstrap-crn", "Chat：任务 bootstrap 与共同随机数"],
|
||||
["28", "branches", "别漏掉旁支"],
|
||||
["29", "audit", "事实、推导与教学模型"],
|
||||
["↳", "papers", "六十节点阅读链"],
|
||||
];
|
||||
---
|
||||
|
||||
<BaseLayout
|
||||
title="DeepSeek 技术谱系与真实权重深读：从 Dense、MoE、MLA 到 R1 与 V4"
|
||||
description="用二十四张问题账、十次技术转向、二十一个交互实验、真实 V2-Lite Base / Chat 权重、512-token 完成度评测、29 阶段隐藏状态、26 层 MoE 路由追踪，以及单题与跨题两轮各 256 条采样，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
||||
description="用二十四张问题账、十次技术转向、二十二个交互实验、真实 V2-Lite Base / Chat 权重、512-token 完成度评测、29 阶段隐藏状态、26 层 MoE 路由追踪，以及 864 条分层采样与显式共同随机数审计，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
||||
section="deepseek"
|
||||
>
|
||||
<header class="page-hero deepseek-hero">
|
||||
<div class="page-hero-inner">
|
||||
<div>
|
||||
<p class="eyebrow"><span>SPOTLIGHT / DEEPSEEK · ROUND 07</span> SOURCE COVERAGE × SAMPLING × FULL DEPTH</p>
|
||||
<p class="eyebrow"><span>SPOTLIGHT / DEEPSEEK · ROUND 08</span> TASK BOOTSTRAP × COMMON RANDOM NUMBERS × FULL DEPTH</p>
|
||||
<h1>不要背模型名<br />要看懂每次为什么转向</h1>
|
||||
<p class="lead">
|
||||
这不是七篇报告的摘要，而是一套可追问、可计算、可反驳的技术谱系：
|
||||
@@ -63,9 +65,9 @@ const toc = [
|
||||
<div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
|
||||
<div><dt>LEDGERS</dt><dd>24 张问题账</dd></div>
|
||||
<div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div>
|
||||
<div><dt>LABS</dt><dd>21 个可操作实验</dd></div>
|
||||
<div><dt>LABS</dt><dd>22 个可操作实验</dd></div>
|
||||
<div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div>
|
||||
<div><dt>STATUS</dt><dd>七轮 · 512 条采样</dd></div>
|
||||
<div><dt>STATUS</dt><dd>八轮 · 864 条采样</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</header>
|
||||
@@ -840,8 +842,22 @@ const toc = [
|
||||
<DeepSeekCrossSourceSamplingLab />
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="task-bootstrap-crn">
|
||||
<p class="eyebrow"><span>27</span> TASKS ARE NOT RANDOM TAPES</p>
|
||||
<h2>换一道题与换一条随机带，不是同一种不确定性：把 32 题 bootstrap 与真正的共同随机数接起来</h2>
|
||||
<p class="lede">
|
||||
Round 07 把 source 提升为覆盖单位，却仍只有每域四题；四行也只是共享 batch
|
||||
seed，不是真正共享同一概率分位。Round 08 在 HumanEval 与 GSM8K 各冻结 32 题，
|
||||
主分析统一使用 T0；另取每域四题跑 T0–T3。每个 source、tape、step 的
|
||||
<code>uₜ</code> 由 SHA-256 显式派生，四个 prompt 条件读取同一个
|
||||
<code>uₜ</code>，再穿过各自的 <code>temperature=.3 / top_p=.95</code>
|
||||
CDF。这样可以把任务差异、sampling tape 差异与 prompt 条件差异放进不同账本。
|
||||
</p>
|
||||
<DeepSeekTaskBootstrapLab />
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="branches">
|
||||
<p class="eyebrow"><span>27</span> THE MAIN LINE IS NOT THE WHOLE TREE</p>
|
||||
<p class="eyebrow"><span>28</span> THE MAIN LINE IS NOT THE WHOLE TREE</p>
|
||||
<h2>如果只读 V2 → V3 → R1 → V4，会漏掉五条反过来影响主线的旁支</h2>
|
||||
<div class="branch-grid">
|
||||
{deepseekBranches.map(([name, line, text, url]) => (
|
||||
@@ -861,7 +877,7 @@ const toc = [
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="audit">
|
||||
<p class="eyebrow"><span>28</span> EVIDENCE AUDIT</p>
|
||||
<p class="eyebrow"><span>29</span> EVIDENCE AUDIT</p>
|
||||
<h2>同一张页面里有三种知识，它们的语气必须不同</h2>
|
||||
<div class="audit-grid">
|
||||
<article class="reported">
|
||||
|
||||
@@ -145,18 +145,18 @@ const paths = [
|
||||
</a>
|
||||
<a class="release-card deepseek-release" href="/deepseek/">
|
||||
<div>
|
||||
<p class="eyebrow"><span>NEW / DEEPSEEK ROUND 07</span> CROSS-SOURCE SAMPLING · SOURCE-BLOCKED AUDIT</p>
|
||||
<p class="eyebrow"><span>NEW / DEEPSEEK ROUND 08</span> TASK BOOTSTRAP · EXPLICIT COMMON RANDOM NUMBERS</p>
|
||||
<h2>从 Dense 到百万上下文：每次创新都在偿还上一代最贵的一张账</h2>
|
||||
<p>
|
||||
单题抽 64 次仍然只有一道题。新一轮保持 256 条预算不变，改用 16 条预先冻结的
|
||||
source：Math 四题从 8 / 16 到 16 / 16，Code 四题也从 8 / 16 到 16 / 16；
|
||||
English 甚至出现域均值与 3 / 4 source 方向相反。新进程 R0 仍 64 / 64 格 exact。
|
||||
从每域四题扩到 HumanEval / GSM8K 各 32 题，并把“同 seed”升级为显式共享
|
||||
uniform tape：352 条正式输出、10,000 次选定任务配对 bootstrap 与 64 条
|
||||
十二字段新进程重放。正确性区间都跨零，但输出长度揭示 Code 与 Math 方向相反。
|
||||
</p>
|
||||
</div>
|
||||
<dl>
|
||||
<div><dt>LINEAGE</dt><dd>1991 → 2026 · 10 次转向</dd></div>
|
||||
<div><dt>NODES</dt><dd>60 个一手 / 官方节点</dd></div>
|
||||
<div><dt>LAB</dt><dd>21 · Base / Chat / sampling</dd></div>
|
||||
<div><dt>LAB</dt><dd>22 · Base / Chat / sampling</dd></div>
|
||||
</dl>
|
||||
<span class="release-arrow" aria-hidden="true">进入 DeepSeek 完整技术谱系 →</span>
|
||||
</a>
|
||||
|
||||
@@ -15,7 +15,7 @@ const workstreams = [
|
||||
{ label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
|
||||
{ label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" },
|
||||
{ label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM，加入真实去重与 mixture traces" },
|
||||
{ label: "DeepSeek 专题", value: 99, next: "把跨题采样扩大到可做 task-level bootstrap，再推进 per-row RNG、干预式 mediation、SM90 FlashMLA 与 R1-like RL" },
|
||||
{ label: "DeepSeek 专题", value: 99, next: "推进干预式 mediation、SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现" },
|
||||
{ label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
|
||||
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
|
||||
{ label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
|
||||
@@ -97,12 +97,12 @@ const workstreams = [
|
||||
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
||||
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
||||
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
||||
<article><span>✓</span><h3>八十八个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验、十三联 Base 工件实验、Chat 行为、completion/full-depth、multi-seed 与 cross-source sampling 四轮实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>八十九个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验、十三联 Base 工件实验、Chat 行为、completion/full-depth、multi-seed、cross-source 与 task-bootstrap CRN 五轮实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
||||
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
||||
<article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
|
||||
<article><span>✓</span><h3>DeepSeek 七轮真实权重里程碑</h3><p>继单题多 seed 之后，保持 256 条预算并把覆盖扩大到 16 条预先冻结的 source：250 条 natural EOS、247 个 unique trajectories；Math 四题为 14/16、16/16、8/16、9/16，Code 四题为 16/16、8/16、12/16、16/16。source-blocked 方向揭示 English 均值与多数题相反，新进程 R0 八项合同字段 64 / 64 exact。</p></article>
|
||||
<article><span>✓</span><h3>DeepSeek 八轮真实权重里程碑</h3><p>把覆盖扩到 HumanEval / GSM8K 各 32 条冻结任务，用显式 SHA-256 uniform tape 驱动四个条件的共同随机数采样：352 条正式输出中 343 条 natural EOS、320 个 unique trajectories；10,000 次选定任务配对 bootstrap 的正确性区间均跨零，长度则揭示 Code 与 Math 的相反方向。新进程十二字段重放 64 / 64 exact。</p></article>
|
||||
<article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
|
||||
<article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
|
||||
<article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article>
|
||||
@@ -134,7 +134,7 @@ const workstreams = [
|
||||
<div class="queue-table">
|
||||
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
|
||||
<div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div>
|
||||
<div><span>P0</span><strong>DeepSeek 七轮后续</strong><p>扩大到 task-level bootstrap → per-row RNG 对照 → 干预式 mediation → SM90 FlashMLA / FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
||||
<div><span>P0</span><strong>DeepSeek 八轮后续</strong><p>干预式 mediation → SM90 FlashMLA / FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
||||
<div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div>
|
||||
<div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div>
|
||||
<div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div>
|
||||
@@ -237,6 +237,10 @@ const workstreams = [
|
||||
<div><time>2026-07-30</time><b>任务总数必须展开为逐题矩阵</b><p>Math 与 Code 的四题都从 8/16 跨到 16/16；跨不同 GSM8K gold 的 final-answer frequency 禁止聚合。</p></div>
|
||||
<div><time>2026-07-30</time><b>均值与 source 方向同时展示</b><p>English 句点 contrast 均值 −8.5，但 3/4 source 为正；Code 两道题 +1/−1 interaction 在域均值 0 中抵消。</p></div>
|
||||
<div><time>2026-07-30</time><b>跨题 sampling 仍要求精确复跑</b><p>R0/R1 63/64 同格分叉；新进程 R0 的 seed、prompt、token、text、stop 与 CPU/CUDA RNG pre-state 八字段 64/64 exact。</p></div>
|
||||
<div><time>2026-07-30</time><b>共同随机数必须共享 uniform tape</b><p>仅把生成器 seed 重置为同一个值并不等于 CRN；Round 08 用 SHA-256 逐 task/tape/step 生成显式 u，并让四个条件把同一 u 映射到各自 token CDF。</p></div>
|
||||
<div><time>2026-07-30</time><b>bootstrap 的统计单位是任务</b><p>HumanEval 与 GSM8K 各 32 题分开做 10,000 次成对任务重采样；这是冻结题集的敏感性带，不冒充 benchmark 总体或跨 seed 置信区间。</p></div>
|
||||
<div><time>2026-07-30</time><b>正确性未定，不等于没有行为效应</b><p>八个正确性 contrast 区间全部跨零；system-at-period 的长度带在 Code 为 −130.9 [−178.7,−83.2]，在 Math 为 +25.1 [7.9,44.5]，明确显示任务域交互。</p></div>
|
||||
<div><time>2026-07-30</time><b>复现合同扩到随机带本身</b><p>64 条新进程重放逐字段核对 uniform hash、token IDs、文本、停止状态与 RNG；十二字段全部 64/64 exact，并公开评分文件提前加载这一流程偏差。</p></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user