research: audit task bootstrap CRN experiment
This commit is contained in:
wuyang
@@ -853,3 +853,135 @@ See `research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_PROTOCOL.md` and
|
|||||||
`research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md` for the
|
`research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md` for the
|
||||||
preregistered source frame, task matrices, source-direction contrasts,
|
preregistered source frame, task matrices, source-direction contrasts,
|
||||||
failure identities, hash chain, exact replay scope, and non-claims.
|
failure identities, hash chain, exact replay scope, and non-claims.
|
||||||
|
|
||||||
|
## Selected-task bootstrap with explicit common random numbers
|
||||||
|
|
||||||
|
`v2_lite_chat_task_bootstrap_crn_probe.py` expands the task frame to all 32
|
||||||
|
frozen HumanEval and 32 frozen GSM8K sources. The primary analysis runs one
|
||||||
|
preregistered tape T0 on every task. Four tasks per domain additionally run
|
||||||
|
T1-T3 as a crossed tape-sensitivity diagnostic:
|
||||||
|
|
||||||
|
```text
|
||||||
|
64 tasks × T0 × 4 conditions = 256 outputs
|
||||||
|
8 diagnostic tasks × 3 additional tapes × 4 conditions = 96 outputs
|
||||||
|
formal total = 352 outputs
|
||||||
|
```
|
||||||
|
|
||||||
|
Unlike the preceding batch-seed-aligned studies, Round 08 constructs a true
|
||||||
|
step-index common-random-number tape. The same SHA-256-derived `u_t` is
|
||||||
|
mapped through each condition's official `.3/.95` temperature/top-p
|
||||||
|
distribution using a float32 token-ID-ordered inverse CDF. It does not call
|
||||||
|
`transformers.generate` or `torch.multinomial`.
|
||||||
|
|
||||||
|
The frozen manifest and all 256 Chat prompt hashes must be built before any
|
||||||
|
model outputs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node scripts/build-deepseek-chat-task-bootstrap-manifest.mjs \
|
||||||
|
--reference \
|
||||||
|
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||||
|
--output \
|
||||||
|
research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json
|
||||||
|
|
||||||
|
PYTHONPATH=/path/to/transformers-4.41.2-deps \
|
||||||
|
python -B scripts/freeze-deepseek-chat-task-bootstrap-prompts.py \
|
||||||
|
--artifact-dir /path/to/deepseek-v2-lite-chat \
|
||||||
|
--manifest \
|
||||||
|
research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json \
|
||||||
|
--reference-routing-json \
|
||||||
|
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||||
|
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||||
|
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||||
|
--tnews /path/to/tnews/test.json \
|
||||||
|
--tnews-archive /path/to/tnews_public.zip \
|
||||||
|
--wikitext /path/to/wikitext-validation.parquet
|
||||||
|
```
|
||||||
|
|
||||||
|
The formal runner writes one hash-checked journal per source and supports
|
||||||
|
`--resume`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||||
|
PYTHONPATH=/path/to/transformers-4.41.2-deps:experiments/deepseek \
|
||||||
|
python -B \
|
||||||
|
experiments/deepseek/v2_lite_chat_task_bootstrap_crn_probe.py \
|
||||||
|
--execution-mode formal \
|
||||||
|
--artifact-dir /path/to/deepseek-v2-lite-chat \
|
||||||
|
--manifest \
|
||||||
|
research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json \
|
||||||
|
--reference-routing-json \
|
||||||
|
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||||
|
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||||
|
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||||
|
--tnews /path/to/tnews/test.json \
|
||||||
|
--tnews-archive /path/to/tnews_public.zip \
|
||||||
|
--wikitext /path/to/wikitext-validation.parquet \
|
||||||
|
--journal-dir /path/to/formal-journal \
|
||||||
|
--resume \
|
||||||
|
--output \
|
||||||
|
src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Evaluation keeps T0 primary rows separate from T1-T3 diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PYTHONPATH=experiments/deepseek \
|
||||||
|
python -B \
|
||||||
|
experiments/deepseek/v2_lite_chat_task_bootstrap_crn_evaluator.py \
|
||||||
|
--sampling-json \
|
||||||
|
src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json \
|
||||||
|
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||||
|
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||||
|
--sandbox-image \
|
||||||
|
python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
|
||||||
|
--output \
|
||||||
|
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The analysis resamples the 32 selected tasks 10,000 times within each domain
|
||||||
|
while preserving condition pairing. These are selected-task resampling
|
||||||
|
bands under T0, not benchmark-population or generation-seed confidence
|
||||||
|
intervals:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -B \
|
||||||
|
experiments/deepseek/v2_lite_chat_task_bootstrap_crn_analysis.py \
|
||||||
|
--sampling-json \
|
||||||
|
src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json \
|
||||||
|
--evaluation-json \
|
||||||
|
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json \
|
||||||
|
--reproduction-json \
|
||||||
|
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-reproduction.json \
|
||||||
|
--output \
|
||||||
|
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-analysis.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The formal grid contains 343/352 natural EOS outputs and 320 unique complete
|
||||||
|
trajectories. T0 correctness totals are Code 59/128 and Math 71/128 across
|
||||||
|
32 tasks × four prompt conditions; they are not standard benchmark scores.
|
||||||
|
All correctness contrast bands cross zero. The clearest length interaction
|
||||||
|
is system-on minus system-off under the period boundary: Code is -130.875
|
||||||
|
tokens with band [-178.688, -83.218], while Math is +25.125 with band
|
||||||
|
[7.874, 44.531]. The preregistered 64-cell cross-process replay is exact in
|
||||||
|
all twelve audited fields.
|
||||||
|
|
||||||
|
```text
|
||||||
|
formal ea0607f2b197fac3f794655c1538ce1f9a1cb072d637eb31d35573682e311809
|
||||||
|
eval 82b2fc5d1f854a7e0cd7aba7c70ff74d733d24221522a4f92b962478c76177ab
|
||||||
|
replay 6519947e2fa4327c1ba2cc506b0861f172a6bdbd4f2d6787447edaf8fbcac508
|
||||||
|
compare 63ed39e5dcdbc2a30e516172f3657dfa453ff3b23d5bd5c49c734939242a70f5
|
||||||
|
analysis 9ab17561ced930a668c082141f7c6e013cbda70e42b09de63d41f1b82c01a6ae
|
||||||
|
compact 082e3c56373e5f91d51d88b1430d14ad3b68b0f200e80901171c2fb62a1e57bf
|
||||||
|
```
|
||||||
|
|
||||||
|
Run the complete offline hash and invariant audit with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node scripts/check-deepseek-chat-task-bootstrap-crn-data.mjs
|
||||||
|
```
|
||||||
|
|
||||||
|
See `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md` and
|
||||||
|
`research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_CRN_AUDIT.md` for the
|
||||||
|
pre-output corrections, explicit sampler, task bootstrap, transition tables,
|
||||||
|
multi-tape matrices, exact replay, reported gold-loading process deviation,
|
||||||
|
and claim boundaries.
|
||||||
|
|||||||
@@ -17,6 +17,8 @@
|
|||||||
"build:data:deepseek-chat-completion-depth": "node scripts/build-deepseek-chat-completion-depth-compact.mjs",
|
"build:data:deepseek-chat-completion-depth": "node scripts/build-deepseek-chat-completion-depth-compact.mjs",
|
||||||
"build:data:deepseek-chat-sampling": "node scripts/build-deepseek-chat-sampling-compact.mjs",
|
"build:data:deepseek-chat-sampling": "node scripts/build-deepseek-chat-sampling-compact.mjs",
|
||||||
"build:data:deepseek-chat-cross-source-sampling": "node scripts/build-deepseek-chat-cross-source-sampling-compact.mjs",
|
"build:data:deepseek-chat-cross-source-sampling": "node scripts/build-deepseek-chat-cross-source-sampling-compact.mjs",
|
||||||
|
"build:data:deepseek-chat-task-bootstrap": "node scripts/build-deepseek-chat-task-bootstrap-crn-compact.mjs",
|
||||||
|
"check:data:deepseek-chat-task-bootstrap": "node scripts/check-deepseek-chat-task-bootstrap-crn-data.mjs",
|
||||||
"check:site": "node scripts/check-site.mjs",
|
"check:site": "node scripts/check-site.mjs",
|
||||||
"check:moe-browser": "node scripts/check-moe-browser.mjs",
|
"check:moe-browser": "node scripts/check-moe-browser.mjs",
|
||||||
"check:reasoning-browser": "node scripts/check-reasoning-browser.mjs",
|
"check:reasoning-browser": "node scripts/check-reasoning-browser.mjs",
|
||||||
|
|||||||
@@ -0,0 +1,786 @@
|
|||||||
|
# DeepSeek-V2-Lite-Chat：32 题任务 bootstrap 与显式共同随机数审计
|
||||||
|
|
||||||
|
> 协议：`llm-atlas-deepseek-chat-task-bootstrap-crn-v1`
|
||||||
|
>
|
||||||
|
> 模型：`deepseek-ai/DeepSeek-V2-Lite-Chat`
|
||||||
|
>
|
||||||
|
> revision：`85864749cd611b4353ce1decdb286193298f64c7`
|
||||||
|
>
|
||||||
|
> 预注册协议：
|
||||||
|
> `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md`
|
||||||
|
>
|
||||||
|
> 冻结清单：
|
||||||
|
> `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json`
|
||||||
|
>
|
||||||
|
> 执行日期：2026-07-30
|
||||||
|
|
||||||
|
## 0. 先说结论
|
||||||
|
|
||||||
|
Round 08 把 Round 07 的“每域 4 道题 × 4 seeds”扩成：
|
||||||
|
|
||||||
|
```text
|
||||||
|
主分析：
|
||||||
|
HumanEval 32 题 × T0 × 4 conditions = 128 outputs
|
||||||
|
GSM8K 32 题 × T0 × 4 conditions = 128 outputs
|
||||||
|
|
||||||
|
随机带敏感性：
|
||||||
|
每域 4 道固定题 × T0–T3 × 4 conditions
|
||||||
|
其中 T1–T3 新增 96 outputs
|
||||||
|
|
||||||
|
正式总计：
|
||||||
|
64 tasks × 4 conditions on T0
|
||||||
|
+ 8 diagnostic tasks × 3 extra tapes × 4 conditions
|
||||||
|
= 352 outputs
|
||||||
|
```
|
||||||
|
|
||||||
|
四个条件不再只是“同 seed 对齐”，而是在同一道题、同一随机带、同一个生成步显式读取
|
||||||
|
完全相同的均匀数 `u_t`。`.3/.95` 的温度和 nucleus 分布来自固定 checkpoint 与
|
||||||
|
Transformers warper，离散取样则由本协议定义的 token-ID 顺序逆 CDF 完成。
|
||||||
|
|
||||||
|
最稳的观察不是 correctness，而是**输出长度具有明显的 domain × prompt 条件交互**：
|
||||||
|
|
||||||
|
- Code：在 period 边界下，打开 system 后平均缩短 `130.875` tokens；当前 32 题、T0
|
||||||
|
selected-task resampling band 为 `[-178.688, -83.218]`；
|
||||||
|
- Math：同一对比反而平均延长 `25.125` tokens；对应 band 为
|
||||||
|
`[7.874, 44.531]`；
|
||||||
|
- EOS 边界下的 system 长度差在两个 domain 都没有得到同样清楚的非零带；
|
||||||
|
- 所有四个预注册 correctness contrast 的 selected-task band 在 Code 和 Math 中都跨
|
||||||
|
0，不能写成稳定能力提升或下降；
|
||||||
|
- 64 条独立跨进程 replay 在 12 个冻结字段上全部 `64 / 64 exact`。
|
||||||
|
|
||||||
|
因此，本轮最合理的结论是：
|
||||||
|
|
||||||
|
> 同一个 system 文本的生成长度效应并不是孤立、固定方向的“system 开关效应”；它依赖
|
||||||
|
> 历史边界 token，并且在 Code 与 Math 两类任务上方向相反。correctness 的方向则仍有
|
||||||
|
> 明显任务与随机带不确定性。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 为什么 Round 07 还不够
|
||||||
|
|
||||||
|
Round 07 已经避免了最明显的伪样本量错误：它没有把同一道题的四个 seed 当成四道独立
|
||||||
|
题。但每域只有 4 道题：
|
||||||
|
|
||||||
|
```text
|
||||||
|
4 tasks × 4 seeds × 4 conditions
|
||||||
|
```
|
||||||
|
|
||||||
|
这仍有三个缺口：
|
||||||
|
|
||||||
|
1. 4 道题上的方向很容易被单题主导；
|
||||||
|
2. 同一个 batch seed 不等于四行使用同一个随机数；
|
||||||
|
3. 任务差异与 sampling 随机带差异没有被正交地拆开。
|
||||||
|
|
||||||
|
Round 08 因此把预算优先给更多任务，并额外留下一个小型交叉设计：
|
||||||
|
|
||||||
|
```text
|
||||||
|
32 tasks × T0 → 主 selected-task 分析
|
||||||
|
4 tasks × T0–T3 → tape sensitivity
|
||||||
|
```
|
||||||
|
|
||||||
|
`4 tasks × 4 tapes` 是 crossed repeated measures，不是 16 道独立题。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 预注册与预运行修订
|
||||||
|
|
||||||
|
### 2.1 先冻结、后生成
|
||||||
|
|
||||||
|
首版协议与 64 题选择清单在任何本轮模型输出产生前提交：
|
||||||
|
|
||||||
|
```text
|
||||||
|
6181af9 research: preregister task bootstrap CRN protocol
|
||||||
|
```
|
||||||
|
|
||||||
|
实现与 256 个 Chat prompt hash 在 smoke 前提交：
|
||||||
|
|
||||||
|
```text
|
||||||
|
a09e6b1 research: freeze Chat prompts and CRN runner
|
||||||
|
```
|
||||||
|
|
||||||
|
评估、重放与 bootstrap 的指标口径已经写入首版协议；对应计算脚本在 formal 运行期间实现
|
||||||
|
并提交：
|
||||||
|
|
||||||
|
```text
|
||||||
|
65a91ae research: add task bootstrap evaluation pipeline
|
||||||
|
```
|
||||||
|
|
||||||
|
脚本实现没有根据 correctness 结果替换题、随机带、contrast、bootstrap seed 或主指标。
|
||||||
|
|
||||||
|
### 2.2 修订 1：固定四行 batch
|
||||||
|
|
||||||
|
最初草案写成四条 trajectory 逐条 forward。实现审查发现这会把同一模型 forward 重复四
|
||||||
|
次，预计增加约四倍 GPU/CPU-offload 时间。任何 smoke 输出之前，合同修订为：
|
||||||
|
|
||||||
|
- 固定四行 batch；
|
||||||
|
- 行序 `s0_eos, s1_eos, s0_period, s1_period`；
|
||||||
|
- 四行第 `t` 步读取同一个 `u_t`；
|
||||||
|
- 某行 EOS 后，后续追加 PAD=EOS、attention mask 为 0；
|
||||||
|
- batch 内不存在跨行 attention；
|
||||||
|
- sampling 不调用 PyTorch RNG。
|
||||||
|
|
||||||
|
这项修订只改变执行效率与完成行的 padding 合同，没有看结果。
|
||||||
|
|
||||||
|
### 2.3 修订 2：纠正 Chat prompt hash
|
||||||
|
|
||||||
|
tokenizer-only 闸门在模型加载前发现，首版 manifest 把路由探针的短输入 hash 错标成了
|
||||||
|
Chat 生成 prompt hash。例如 `HumanEval/31`：
|
||||||
|
|
||||||
|
```text
|
||||||
|
路由探针 s0_eos：48 tokens
|
||||||
|
Chat 生成 s0_eos：145 tokens
|
||||||
|
```
|
||||||
|
|
||||||
|
当时：
|
||||||
|
|
||||||
|
- 本轮模型输出为 0；
|
||||||
|
- checkpoint 尚未加载；
|
||||||
|
- correctness 未计算。
|
||||||
|
|
||||||
|
清单随后：
|
||||||
|
|
||||||
|
1. 保留原字段，并改名为 `routing_probe_prompt_token_ids_sha256`；
|
||||||
|
2. 用正式 Chat renderer 重新产生 64 × 4 = 256 个 hash；
|
||||||
|
3. 写入 `chat_generation_prompt_token_ids_sha256`；
|
||||||
|
4. runner 只核验后者。
|
||||||
|
|
||||||
|
最终：
|
||||||
|
|
||||||
|
| 项目 | SHA-256 |
|
||||||
|
|---|---|
|
||||||
|
| manifest 文件 | `6313e70536c464fe598a93035576752418f08016dfd60ac246437c3b43bf2ae1` |
|
||||||
|
| manifest 规范内容 | `d15303e18345f6dec2aaf891ba812c0a0232b86f416f8f685eabbe818249e838` |
|
||||||
|
| 256 Chat prompt 合同 | `7f766be54463c7f513948dd6d63a55e12b85b9220ee177361c743cd9997f6c3b` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. source 与执行网格
|
||||||
|
|
||||||
|
### 3.1 source 选择
|
||||||
|
|
||||||
|
两个 domain 都使用前序 routing control 中已冻结的全部 32 条：
|
||||||
|
|
||||||
|
```text
|
||||||
|
sample salt = llm-atlas-deepseek-routing-template-control-v1
|
||||||
|
within_domain_index = 0..31
|
||||||
|
```
|
||||||
|
|
||||||
|
没有按 Round 07 pass、EOS、长度、文本质量或 selection rank 的“好看程度”二次筛选。
|
||||||
|
完整 source ID、source hash、selection rank、四格 prompt hash 与 tape 分配见 manifest。
|
||||||
|
|
||||||
|
### 3.2 主随机带与诊断随机带
|
||||||
|
|
||||||
|
| tape | display seed |
|
||||||
|
|---|---:|
|
||||||
|
| T0 | 2,572,353,518 |
|
||||||
|
| T1 | 399,507,326 |
|
||||||
|
| T2 | 405,284,229 |
|
||||||
|
| T3 | 568,701,915 |
|
||||||
|
|
||||||
|
全部 64 题运行 T0。每域 index `0, 8, 16, 24` 再运行 T1–T3。
|
||||||
|
|
||||||
|
### 3.3 独立 replay 子集
|
||||||
|
|
||||||
|
每域固定：
|
||||||
|
|
||||||
|
```text
|
||||||
|
0, 4, 8, 12, 16, 20, 24, 28
|
||||||
|
```
|
||||||
|
|
||||||
|
只 replay T0 × 四条件，共：
|
||||||
|
|
||||||
|
```text
|
||||||
|
16 tasks × 4 conditions = 64 cells
|
||||||
|
```
|
||||||
|
|
||||||
|
这不是看完 formal 后挑的“容易复现样本”。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 显式共同随机数取样器
|
||||||
|
|
||||||
|
### 4.1 `u_t` 的定义
|
||||||
|
|
||||||
|
对同一 `source_id + tape + step`：
|
||||||
|
|
||||||
|
```text
|
||||||
|
H = SHA256(
|
||||||
|
protocol_id + "\0uniform\0"
|
||||||
|
+ tape_label + "\0"
|
||||||
|
+ source_id + "\0"
|
||||||
|
+ decimal(step)
|
||||||
|
)
|
||||||
|
|
||||||
|
z_t = H[0:8] as big-endian uint64
|
||||||
|
u_t = (z_t + 0.5) / 2^64
|
||||||
|
```
|
||||||
|
|
||||||
|
四个 condition 使用完全相同的 `z_t / u_t`。每条 output 都记录：
|
||||||
|
|
||||||
|
-实际消费步数；
|
||||||
|
-消费前缀的规范 SHA-256；
|
||||||
|
-前 8 个 uint64 hex；
|
||||||
|
- prompt、trajectory 与文本 hash。
|
||||||
|
|
||||||
|
离线审计从协议重新派生全部 352 条 output 的 uniform 前缀：
|
||||||
|
|
||||||
|
```text
|
||||||
|
352 / 352 exact
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 概率分布
|
||||||
|
|
||||||
|
每步：
|
||||||
|
|
||||||
|
1. fixed checkpoint 产生 BF16 logits；
|
||||||
|
2. `TemperatureLogitsWarper(0.3)`；
|
||||||
|
3. `TopPLogitsWarper(0.95, min_tokens_to_keep=1)`；
|
||||||
|
4. float32 softmax；
|
||||||
|
5. token ID 顺序 float32 CDF；
|
||||||
|
6. CDF 最后一个值强制为 1；
|
||||||
|
7. `u_t` 转 float32并 clamp 到相邻可表示开区间端点；
|
||||||
|
8. `torch.searchsorted(..., right=False)` 选 token。
|
||||||
|
|
||||||
|
必须区分：
|
||||||
|
|
||||||
|
```text
|
||||||
|
模型与 .3/.95 概率分布：官方固定对象
|
||||||
|
离散 sampler：本协议的显式 inverse CDF
|
||||||
|
```
|
||||||
|
|
||||||
|
本轮没有调用：
|
||||||
|
|
||||||
|
```text
|
||||||
|
transformers.generate
|
||||||
|
torch.multinomial
|
||||||
|
```
|
||||||
|
|
||||||
|
所以不能把 trajectory 写成“官方 Transformers sampler 的 exact 输出”。
|
||||||
|
|
||||||
|
### 4.3 共同随机数不等于共同 token
|
||||||
|
|
||||||
|
共享的是概率积分尺度上的 `u_t`。prompt 条件改变 logits 后，同一个 `u_t` 会落入不同
|
||||||
|
token 的 CDF 区间。
|
||||||
|
|
||||||
|
这正是想要的配对：
|
||||||
|
|
||||||
|
```text
|
||||||
|
随机冲击相同
|
||||||
|
条件分布不同
|
||||||
|
→ token 可以不同
|
||||||
|
```
|
||||||
|
|
||||||
|
八个 domain × contrast 的主分析中：
|
||||||
|
|
||||||
|
```text
|
||||||
|
32 sources × 8 domain-contrasts
|
||||||
|
= 256 / 256 shared-uniform-prefix audits exact
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. smoke 闸门
|
||||||
|
|
||||||
|
smoke：
|
||||||
|
|
||||||
|
```text
|
||||||
|
每域 index 0,8,16,24
|
||||||
|
× T0/T1
|
||||||
|
× 4 conditions
|
||||||
|
× 16-token cap
|
||||||
|
= 64 short outputs
|
||||||
|
```
|
||||||
|
|
||||||
|
结果：
|
||||||
|
|
||||||
|
| 闸门 | 结果 |
|
||||||
|
|---|---:|
|
||||||
|
| Chat prompt hash | 256 / 256 exact |
|
||||||
|
| short outputs | 64 / 64 |
|
||||||
|
| 同进程 T0 replay | 32 / 32 exact |
|
||||||
|
| T0/T1 可比格 | 32 |
|
||||||
|
| T0/T1 trajectory 分叉 | 17 / 32 |
|
||||||
|
| RNG state 前后不变 | 16 / 16 runs |
|
||||||
|
| synthetic inverse-CDF test | passed |
|
||||||
|
| OOM / NaN / exception | 0 |
|
||||||
|
|
||||||
|
smoke 文件：
|
||||||
|
|
||||||
|
```text
|
||||||
|
/tmp/deepseek-v2-lite-chat-task-bootstrap-smoke.json
|
||||||
|
SHA-256 c868e74f88e39343f95524b4b38d0c1503f727a22d4dfa3937532514205110b5
|
||||||
|
```
|
||||||
|
|
||||||
|
smoke 不进入正式统计。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 正式执行账
|
||||||
|
|
||||||
|
### 6.1 总数
|
||||||
|
|
||||||
|
| 项目 | 数值 |
|
||||||
|
|---|---:|
|
||||||
|
| tasks | 64 |
|
||||||
|
| source × tape runs | 88 |
|
||||||
|
| outputs | 352 |
|
||||||
|
| T0 主 outputs | 256 |
|
||||||
|
| T1–T3 新增诊断 outputs | 96 |
|
||||||
|
| natural EOS | 343 |
|
||||||
|
| budget truncated | 9 |
|
||||||
|
| unique trajectory hashes | 320 |
|
||||||
|
| RNG state 前后不变 | 88 / 88 runs |
|
||||||
|
|
||||||
|
### 6.2 分 domain 的全部 formal outputs
|
||||||
|
|
||||||
|
这里含诊断题 T1–T3，只用于执行账，不是主通过率。
|
||||||
|
|
||||||
|
| domain | outputs | natural EOS | truncated | mean tokens |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| Code | 176 | 168 | 8 | 277.278 |
|
||||||
|
| Math | 176 | 175 | 1 | 183.188 |
|
||||||
|
|
||||||
|
### 6.3 资源
|
||||||
|
|
||||||
|
| 项目 | 数值 |
|
||||||
|
|---|---:|
|
||||||
|
| checkpoint load | 8.344 s |
|
||||||
|
| formal generation sum | 4,336.699 s |
|
||||||
|
| peak CUDA allocated | 29,919,644,672 bytes |
|
||||||
|
| peak CUDA allocated | 27.864 GiB |
|
||||||
|
| GPU | NVIDIA GeForce RTX 5090 |
|
||||||
|
| dtype | BF16 |
|
||||||
|
| CUDA resident | embedding + layers 0–23 |
|
||||||
|
| CPU offload | layers 24–26 + norm + lm_head |
|
||||||
|
|
||||||
|
这不是 serving benchmark。CPU-offloaded eager latency 不能外推成吞吐。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 独立 evaluator
|
||||||
|
|
||||||
|
### 7.1 HumanEval sandbox
|
||||||
|
|
||||||
|
固定：
|
||||||
|
|
||||||
|
```text
|
||||||
|
python:3.11-alpine
|
||||||
|
@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4
|
||||||
|
```
|
||||||
|
|
||||||
|
约束：
|
||||||
|
|
||||||
|
- network none；
|
||||||
|
- read-only root filesystem；
|
||||||
|
- user `65534:65534`；
|
||||||
|
- drop ALL capabilities；
|
||||||
|
- no-new-privileges；
|
||||||
|
- 256 MiB memory / swap；
|
||||||
|
- 64 PIDs；
|
||||||
|
- 0.5 CPU；
|
||||||
|
- 5 秒 timeout；
|
||||||
|
- host mounts 0。
|
||||||
|
|
||||||
|
### 7.2 T0 主评估总账
|
||||||
|
|
||||||
|
| domain | outputs | fixed-budget pass | strict pass | natural EOS | truncated | mean tokens |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|
|
||||||
|
| Code | 128 | 59 | 59 | 122 | 6 | 284.227 |
|
||||||
|
| Math | 128 | 71 | 71 | 127 | 1 | 185.969 |
|
||||||
|
|
||||||
|
不要把 `59 / 128` 或 `71 / 128` 当成标准 benchmark 分数。分母是：
|
||||||
|
|
||||||
|
```text
|
||||||
|
32 frozen tasks × 4 prompt conditions
|
||||||
|
```
|
||||||
|
|
||||||
|
同一道题在四条件中重复出现。
|
||||||
|
|
||||||
|
### 7.3 失败分类
|
||||||
|
|
||||||
|
HumanEval T0：
|
||||||
|
|
||||||
|
| outcome | count |
|
||||||
|
|---|---:|
|
||||||
|
| passed | 59 |
|
||||||
|
| assertion failed | 47 |
|
||||||
|
| runtime error | 16 |
|
||||||
|
| timeout | 3 |
|
||||||
|
| extract failed | 2 |
|
||||||
|
| syntax error | 1 |
|
||||||
|
|
||||||
|
GSM8K T0：
|
||||||
|
|
||||||
|
| outcome | count |
|
||||||
|
|---|---:|
|
||||||
|
| numeric exact | 71 |
|
||||||
|
| wrong numeric answer | 57 |
|
||||||
|
| no numeric answer | 0 |
|
||||||
|
|
||||||
|
通过 HumanEval tests 只是功能证据，不是安全证据。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. T0 四条件：先看原始格
|
||||||
|
|
||||||
|
### 8.1 Code
|
||||||
|
|
||||||
|
| condition | pass / 32 | natural EOS / 32 | mean tokens |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| `s0_eos` | 16 | 30 | 330.094 |
|
||||||
|
| `s1_eos` | 16 | 30 | 308.938 |
|
||||||
|
| `s0_period` | 15 | 31 | 314.375 |
|
||||||
|
| `s1_period` | 12 | 31 | 183.500 |
|
||||||
|
|
||||||
|
### 8.2 Math
|
||||||
|
|
||||||
|
| condition | pass / 32 | natural EOS / 32 | mean tokens |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| `s0_eos` | 19 | 31 | 181.406 |
|
||||||
|
| `s1_eos` | 19 | 32 | 190.969 |
|
||||||
|
| `s0_period` | 17 | 32 | 173.188 |
|
||||||
|
| `s1_period` | 16 | 32 | 198.313 |
|
||||||
|
|
||||||
|
同样的 `s1_period`，在 Code 中最短，在 Math 中却最长。这比一个全域平均值更重要。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. selected-task bootstrap
|
||||||
|
|
||||||
|
### 9.1 它是什么
|
||||||
|
|
||||||
|
每个 domain：
|
||||||
|
|
||||||
|
-固定 T0；
|
||||||
|
- 32 题有放回重采样 32 题；
|
||||||
|
-四条件保持题级配对；
|
||||||
|
- 10,000 次；
|
||||||
|
- `numpy.random.default_rng`；
|
||||||
|
- seed `1,364,512,825`；
|
||||||
|
-报告 mean contrast 的 percentile 2.5 / 50 / 97.5%。
|
||||||
|
|
||||||
|
准确标签：
|
||||||
|
|
||||||
|
> 当前冻结 32 题、固定 T0 的 selected-task resampling band。
|
||||||
|
|
||||||
|
它不是：
|
||||||
|
|
||||||
|
-完整 benchmark population CI；
|
||||||
|
-模型能力 CI；
|
||||||
|
- generation-seed uncertainty CI；
|
||||||
|
-因果效应 CI。
|
||||||
|
|
||||||
|
### 9.2 correctness：所有带都跨 0
|
||||||
|
|
||||||
|
右减左，单位为每题成功率：
|
||||||
|
|
||||||
|
| domain | contrast | point | selected-task band |
|
||||||
|
|---|---|---:|---:|
|
||||||
|
| Code | period at s0 | -0.0313 | [-0.1563, 0.0938] |
|
||||||
|
| Code | period at s1 | -0.1250 | [-0.3438, 0.0938] |
|
||||||
|
| Code | system at EOS | 0.0000 | [-0.1250, 0.1250] |
|
||||||
|
| Code | system at period | -0.0938 | [-0.2500, 0.0625] |
|
||||||
|
| Math | period at s0 | -0.0625 | [-0.2188, 0.0938] |
|
||||||
|
| Math | period at s1 | -0.0938 | [-0.2500, 0.0625] |
|
||||||
|
| Math | system at EOS | 0.0000 | [-0.1563, 0.1563] |
|
||||||
|
| Math | system at period | -0.0313 | [-0.1563, 0.0938] |
|
||||||
|
|
||||||
|
点估计里 period 条件经常更低，但任务重采样带都覆盖 0。不能把点估计写成稳定能力损失。
|
||||||
|
|
||||||
|
### 9.3 correctness 转移表
|
||||||
|
|
||||||
|
`right` 相对 `left`：
|
||||||
|
|
||||||
|
| domain | contrast | fail→pass | pass→fail | both pass | both fail |
|
||||||
|
|---|---|---:|---:|---:|---:|
|
||||||
|
| Code | period at s0 | 2 | 3 | 13 | 14 |
|
||||||
|
| Code | period at s1 | 4 | 8 | 8 | 12 |
|
||||||
|
| Code | system at EOS | 2 | 2 | 14 | 14 |
|
||||||
|
| Code | system at period | 2 | 5 | 10 | 15 |
|
||||||
|
| Math | period at s0 | 3 | 5 | 14 | 10 |
|
||||||
|
| Math | period at s1 | 2 | 5 | 14 | 11 |
|
||||||
|
| Math | system at EOS | 3 | 3 | 16 | 10 |
|
||||||
|
| Math | system at period | 2 | 3 | 14 | 13 |
|
||||||
|
|
||||||
|
即使平均差是 0，也可能同时有 fail→pass 与 pass→fail。例如 system at EOS 在两个 domain
|
||||||
|
都恰好净 0，但不是每题完全不变。
|
||||||
|
|
||||||
|
### 9.4 输出长度
|
||||||
|
|
||||||
|
右减左，单位 tokens：
|
||||||
|
|
||||||
|
| domain | contrast | point | selected-task band |
|
||||||
|
|---|---|---:|---:|
|
||||||
|
| Code | period at s0 | -15.719 | [-57.627, 24.938] |
|
||||||
|
| Code | period at s1 | -125.438 | [-165.031, -85.125] |
|
||||||
|
| Code | system at EOS | -21.156 | [-47.846, 2.438] |
|
||||||
|
| Code | system at period | -130.875 | [-178.688, -83.218] |
|
||||||
|
| Math | period at s0 | -8.219 | [-27.563, 8.688] |
|
||||||
|
| Math | period at s1 | 7.344 | [-9.407, 26.094] |
|
||||||
|
| Math | system at EOS | 9.563 | [-4.938, 25.188] |
|
||||||
|
| Math | system at period | 25.125 | [7.874, 44.531] |
|
||||||
|
|
||||||
|
两个清楚不跨 0 的 system-at-period 长度带方向相反：
|
||||||
|
|
||||||
|
```text
|
||||||
|
Code → system on 更短
|
||||||
|
Math → system on 更长
|
||||||
|
```
|
||||||
|
|
||||||
|
这是一条比“system prompt 会让输出变短/变长”更精确的结论：
|
||||||
|
|
||||||
|
> system 的长度关联依赖 boundary 与 domain。
|
||||||
|
|
||||||
|
### 9.5 EOS
|
||||||
|
|
||||||
|
EOS contrast 很小：
|
||||||
|
|
||||||
|
- Code：四个 contrast 的 point 在 `0` 或 `+1/32`；
|
||||||
|
- Math：同样只在 `0` 或 `+1/32`；
|
||||||
|
-大部分 source 的 EOS 状态不变。
|
||||||
|
|
||||||
|
因此，Code 中 `-130.875` token 的 system-at-period 差异主要不是简单的“更多题命中 EOS”
|
||||||
|
造成，而是在都自然结束的 trajectory 内结束得更早。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. trajectory 与 first divergence
|
||||||
|
|
||||||
|
### 10.1 共同随机数审计
|
||||||
|
|
||||||
|
| domain | contrast | uniform exact | exact full trajectory | mean common-prefix tokens |
|
||||||
|
|---|---|---:|---:|---:|
|
||||||
|
| Code | period at s0 | 32 / 32 | 0 | 13.25 |
|
||||||
|
| Code | period at s1 | 32 / 32 | 0 | 3.69 |
|
||||||
|
| Code | system at EOS | 32 / 32 | 0 | 23.91 |
|
||||||
|
| Code | system at period | 32 / 32 | 1 | 11.88 |
|
||||||
|
| Math | period at s0 | 32 / 32 | 6 | 52.53 |
|
||||||
|
| Math | period at s1 | 32 / 32 | 1 | 22.91 |
|
||||||
|
| Math | system at EOS | 32 / 32 | 6 | 59.44 |
|
||||||
|
| Math | system at period | 32 / 32 | 1 | 27.75 |
|
||||||
|
|
||||||
|
同一个 `u_t` 没有让四格“黏”成相同 token。特别是 Code 的 `s1` period contrast 平均只
|
||||||
|
共享约 3.7 个前缀 token 就分叉。
|
||||||
|
|
||||||
|
Math 的共同前缀更长、exact trajectory 更多，说明 prompt 条件的作用方式也有 domain
|
||||||
|
差异。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. 四随机带诊断
|
||||||
|
|
||||||
|
每域固定四题，不外推为 32 题总体。
|
||||||
|
|
||||||
|
### 11.1 correctness contrast 的 tape means
|
||||||
|
|
||||||
|
右减左；每个值是 4 道诊断题的均值：
|
||||||
|
|
||||||
|
#### Code
|
||||||
|
|
||||||
|
| contrast | T0 | T1 | T2 | T3 |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| period at s0 | 0.00 | 0.00 | 0.00 | 0.00 |
|
||||||
|
| period at s1 | -0.25 | 0.00 | 0.00 | 0.00 |
|
||||||
|
| system at EOS | 0.00 | 0.00 | 0.00 | 0.00 |
|
||||||
|
| system at period | -0.25 | 0.00 | 0.00 | 0.00 |
|
||||||
|
|
||||||
|
Code 的四题诊断子集中，T0 上的两个 `-0.25` 没有在 T1–T3 重现。它支持谨慎解释主
|
||||||
|
correctness 点估计。
|
||||||
|
|
||||||
|
#### Math
|
||||||
|
|
||||||
|
| contrast | T0 | T1 | T2 | T3 |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| period at s0 | 0.00 | 0.00 | -0.25 | +0.25 |
|
||||||
|
| period at s1 | -0.50 | -0.75 | -0.25 | -0.50 |
|
||||||
|
| system at EOS | 0.00 | +0.50 | -0.25 | +0.25 |
|
||||||
|
| system at period | -0.50 | -0.25 | -0.25 | -0.50 |
|
||||||
|
|
||||||
|
Math 四题上，方向与幅度都明显依赖 tape；四题太少，不能把这些数当成稳定概率。
|
||||||
|
|
||||||
|
### 11.2 长度 contrast 的 tape means
|
||||||
|
|
||||||
|
#### Code
|
||||||
|
|
||||||
|
| contrast | T0 | T1 | T2 | T3 |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| period at s0 | -14.50 | +44.50 | +13.50 | -16.75 |
|
||||||
|
| period at s1 | -108.50 | -155.00 | -21.00 | -163.25 |
|
||||||
|
| system at EOS | -59.00 | +6.00 | -90.00 | -39.50 |
|
||||||
|
| system at period | -153.00 | -193.50 | -124.50 | -186.00 |
|
||||||
|
|
||||||
|
Code 的 system-at-period 在四条 tape 上都明显为负；period-at-s1 也都为负，但 T2 幅度
|
||||||
|
较小。
|
||||||
|
|
||||||
|
#### Math
|
||||||
|
|
||||||
|
| contrast | T0 | T1 | T2 | T3 |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| period at s0 | -5.25 | +31.25 | -10.00 | +4.00 |
|
||||||
|
| period at s1 | +0.50 | +0.25 | +3.25 | -2.50 |
|
||||||
|
| system at EOS | -3.00 | +54.75 | -7.50 | +32.75 |
|
||||||
|
| system at period | +2.75 | +23.75 | +5.75 | +26.25 |
|
||||||
|
|
||||||
|
Math 四题上的 system-at-period 四条 tape 都为正，但幅度从 `+2.75` 到 `+26.25`。
|
||||||
|
|
||||||
|
诊断支持长度交互的方向，但仍只覆盖 4 道题 × 4 条 tape。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. 独立跨进程 replay
|
||||||
|
|
||||||
|
重放结果：
|
||||||
|
|
||||||
|
```text
|
||||||
|
16 sources × T0 × 4 conditions = 64 cells
|
||||||
|
```
|
||||||
|
|
||||||
|
| 字段 | exact |
|
||||||
|
|---|---:|
|
||||||
|
| run seed | 64 / 64 |
|
||||||
|
| prompt hash | 64 / 64 |
|
||||||
|
| full run uniform hash | 64 / 64 |
|
||||||
|
| output uniform-prefix hash | 64 / 64 |
|
||||||
|
| uniform steps | 64 / 64 |
|
||||||
|
| generated token IDs | 64 / 64 |
|
||||||
|
| decoded text | 64 / 64 |
|
||||||
|
| EOS state | 64 / 64 |
|
||||||
|
| truncation state | 64 / 64 |
|
||||||
|
| CPU RNG pre-state | 64 / 64 |
|
||||||
|
| CUDA RNG pre-state | 64 / 64 |
|
||||||
|
| RNG unchanged flag | 64 / 64 |
|
||||||
|
|
||||||
|
总闸门：
|
||||||
|
|
||||||
|
```text
|
||||||
|
64 / 64 all preregistered fields exact
|
||||||
|
```
|
||||||
|
|
||||||
|
这证明固定硬件/软件/精度/执行合同下的逐 token 可重放，不证明换 GPU、PyTorch、
|
||||||
|
Transformers、kernel 或 dtype 后仍 exact。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. 已知偏离与限制
|
||||||
|
|
||||||
|
### 13.1 gold 文件打开时机偏离
|
||||||
|
|
||||||
|
runner 复用了前序 `behavior.task_score` 路径：
|
||||||
|
|
||||||
|
1. 在模型加载前读取 HumanEval / GSM8K gold；
|
||||||
|
2. 每条文本生成结束后附加一个 narrow `task_score`；
|
||||||
|
3. authoritative evaluator 在全部 formal 输出冻结后另行运行。
|
||||||
|
|
||||||
|
这不满足协议中最严格的“gold 只在生成进程结束后打开”表述。
|
||||||
|
|
||||||
|
为什么它不改变 trajectory：
|
||||||
|
|
||||||
|
- gold 不进入 message / prompt；
|
||||||
|
- gold 不进入 tokenizer 输入；
|
||||||
|
- gold 不进入 logits、warper、CDF 或 `u_t`；
|
||||||
|
- gold 不改变 source、tape、max tokens 或 stop；
|
||||||
|
- `task_score` 在 token 序列生成结束后才调用；
|
||||||
|
-独立 evaluator 的 352 行结果不读取 runner 的 `task_score`；
|
||||||
|
- 64-cell replay exact。
|
||||||
|
|
||||||
|
因此它是**真实的流程隔离偏离**，但没有已知的生成因果路径。后续 runner 应删除
|
||||||
|
`task_score`，让生成进程完全不打开 gold。
|
||||||
|
|
||||||
|
### 13.2 不是完整 benchmark
|
||||||
|
|
||||||
|
HumanEval 只使用冻结 32 / 164 tasks；GSM8K 只使用冻结 32 / 1,319 test tasks。source
|
||||||
|
selection 是前序 SHA 排序合同，不是完整 benchmark。
|
||||||
|
|
||||||
|
### 13.3 一条主随机带
|
||||||
|
|
||||||
|
32 题主分析只用 T0。selected-task band 不包含 generation-tape uncertainty。
|
||||||
|
|
||||||
|
### 13.4 多随机带只有四题
|
||||||
|
|
||||||
|
T0–T3 诊断只覆盖每域四题。它揭示 sensitivity，但不能稳定估计 tape 方差。
|
||||||
|
|
||||||
|
### 13.5 counterfactual 不可部署
|
||||||
|
|
||||||
|
period cell 是把一个官方 EOS boundary ID 改为普通句点 ID。它不是官方有效聊天格式。
|
||||||
|
|
||||||
|
### 13.6 成功不是安全
|
||||||
|
|
||||||
|
HumanEval pass 只说明固定 tests 通过。它不说明：
|
||||||
|
|
||||||
|
-安全；
|
||||||
|
-鲁棒；
|
||||||
|
-无未测 bug；
|
||||||
|
-可部署。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. 证据文件与 SHA-256
|
||||||
|
|
||||||
|
| 文件 | SHA-256 |
|
||||||
|
|---|---|
|
||||||
|
| formal sampling | `ea0607f2b197fac3f794655c1538ce1f9a1cb072d637eb31d35573682e311809` |
|
||||||
|
| independent evaluation | `82b2fc5d1f854a7e0cd7aba7c70ff74d733d24221522a4f92b962478c76177ab` |
|
||||||
|
| replay generation | `6519947e2fa4327c1ba2cc506b0861f172a6bdbd4f2d6787447edaf8fbcac508` |
|
||||||
|
| reproduction comparison | `63ed39e5dcdbc2a30e516172f3657dfa453ff3b23d5bd5c49c734939242a70f5` |
|
||||||
|
| bootstrap analysis | `9ab17561ced930a668c082141f7c6e013cbda70e42b09de63d41f1b82c01a6ae` |
|
||||||
|
| compact website data | `082e3c56373e5f91d51d88b1430d14ad3b68b0f200e80901171c2fb62a1e57bf` |
|
||||||
|
| frozen manifest | `6313e70536c464fe598a93035576752418f08016dfd60ac246437c3b43bf2ae1` |
|
||||||
|
|
||||||
|
离线一致性检查：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node scripts/check-deepseek-chat-task-bootstrap-crn-data.mjs
|
||||||
|
```
|
||||||
|
|
||||||
|
通过：
|
||||||
|
|
||||||
|
```text
|
||||||
|
formal sources 64
|
||||||
|
formal runs 88
|
||||||
|
formal outputs 352
|
||||||
|
prompt hashes exact 256
|
||||||
|
uniform output hashes exact 352
|
||||||
|
RNG unchanged runs 88
|
||||||
|
evaluation rows 352
|
||||||
|
reproduction exact cells 64 / 64
|
||||||
|
CRN contrast checks 256
|
||||||
|
bootstrap resamples 10,000
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 15. 最终结论该怎么写
|
||||||
|
|
||||||
|
### 可以写
|
||||||
|
|
||||||
|
1. 在当前固定 32 道 HumanEval、T0 下，system-at-period 的平均输出长度比 system-off
|
||||||
|
短约 131 tokens，selected-task band 不跨 0；
|
||||||
|
2. 在当前固定 32 道 GSM8K、T0 下，同一 contrast 平均长约 25 tokens，band 方向相反；
|
||||||
|
3. 两个 domain 的 correctness contrast bands 都跨 0；
|
||||||
|
4. 每域四题的 T0–T3 诊断显示 correctness 对 tape 敏感，长度交互的方向相对更一致；
|
||||||
|
5. 显式 CRN 与 64-cell 跨进程 replay 都通过 exact audit。
|
||||||
|
|
||||||
|
### 不可以写
|
||||||
|
|
||||||
|
1. “system prompt 普遍让代码更短、数学更长”；
|
||||||
|
2. “period 会降低 DeepSeek 的总体能力”；
|
||||||
|
3. “这些是完整 HumanEval / GSM8K 分数”；
|
||||||
|
4. “selected-task band 是模型总体 95% CI”；
|
||||||
|
5. “period 是官方聊天格式”；
|
||||||
|
6. “通过 HumanEval tests 就安全”；
|
||||||
|
7. “本协议 trajectory 等于官方 `torch.multinomial` trajectory”。
|
||||||
|
|
||||||
|
本轮真正增加的不是一个更大的数字，而是一条更干净的推理链：
|
||||||
|
|
||||||
|
```text
|
||||||
|
固定任务
|
||||||
|
→ 固定概率分布
|
||||||
|
→ 显式共享随机冲击
|
||||||
|
→ 条件内逐题配对
|
||||||
|
→ 任务重采样
|
||||||
|
→ 独立随机带诊断
|
||||||
|
→ 跨进程逐 token 重放
|
||||||
|
```
|
||||||
|
|
||||||
|
它让“prompt 边界如何改变生成行为”从单题现象，前进到一个仍有限、但可审计的任务级机制
|
||||||
|
实验。
|
||||||
@@ -0,0 +1,347 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
|
||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { readFile, writeFile } from "node:fs/promises";
|
||||||
|
import process from "node:process";
|
||||||
|
|
||||||
|
const CONDITIONS = ["s0_eos", "s1_eos", "s0_period", "s1_period"];
|
||||||
|
const DOMAINS = ["code", "math"];
|
||||||
|
const CONTRASTS = [
|
||||||
|
"period_at_s0",
|
||||||
|
"period_at_s1",
|
||||||
|
"system_at_eos",
|
||||||
|
"system_at_period",
|
||||||
|
];
|
||||||
|
const PATHS = {
|
||||||
|
sampling: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json",
|
||||||
|
evaluation: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json",
|
||||||
|
reproduction: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-reproduction.json",
|
||||||
|
analysis: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-analysis.json",
|
||||||
|
output: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-compact.json",
|
||||||
|
};
|
||||||
|
|
||||||
|
function sha256(value) {
|
||||||
|
return createHash("sha256").update(value).digest("hex");
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseArgs(argv) {
|
||||||
|
const result = { ...PATHS };
|
||||||
|
for (let index = 2; index < argv.length; index += 2) {
|
||||||
|
const key = argv[index]?.replace(/^--/, "");
|
||||||
|
const value = argv[index + 1];
|
||||||
|
if (!(key in result) || value === undefined) {
|
||||||
|
throw new Error(`Unknown or incomplete argument: ${argv[index]}`);
|
||||||
|
}
|
||||||
|
result[key] = value;
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
function success(row) {
|
||||||
|
const evaluation = row.task_evaluation;
|
||||||
|
return row.domain === "code"
|
||||||
|
? evaluation.fixed_budget_tests_pass
|
||||||
|
: evaluation.fixed_budget_numeric_exact;
|
||||||
|
}
|
||||||
|
|
||||||
|
const paths = parseArgs(process.argv);
|
||||||
|
const inputBytes = {};
|
||||||
|
const input = {};
|
||||||
|
for (const name of ["sampling", "evaluation", "reproduction", "analysis"]) {
|
||||||
|
inputBytes[name] = await readFile(paths[name]);
|
||||||
|
input[name] = JSON.parse(inputBytes[name]);
|
||||||
|
}
|
||||||
|
const { sampling, evaluation, reproduction, analysis } = input;
|
||||||
|
const evaluationIndex = new Map(
|
||||||
|
evaluation.rows.map((row) => [
|
||||||
|
`${row.source_id}\0${row.tape_label}\0${row.condition}`,
|
||||||
|
row,
|
||||||
|
]),
|
||||||
|
);
|
||||||
|
const sampleIndex = new Map(
|
||||||
|
sampling.sources.flatMap((source) =>
|
||||||
|
source.runs.flatMap((run) =>
|
||||||
|
run.outputs.map((output) => [
|
||||||
|
`${source.id}\0${run.tape_label}\0${output.condition}`,
|
||||||
|
output,
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
|
||||||
|
const conditionTable = Object.fromEntries(
|
||||||
|
DOMAINS.map((domain) => [
|
||||||
|
domain,
|
||||||
|
CONDITIONS.map((condition) => {
|
||||||
|
const summary =
|
||||||
|
evaluation.summary.main_t0_by_domain_condition[domain][condition];
|
||||||
|
return {
|
||||||
|
condition,
|
||||||
|
outputs: summary.outputs,
|
||||||
|
success: summary.fixed_budget_success,
|
||||||
|
successRate: summary.fixed_budget_success / summary.outputs,
|
||||||
|
naturalEos: summary.natural_eos,
|
||||||
|
naturalEosRate: summary.natural_eos / summary.outputs,
|
||||||
|
meanTokens: summary.mean_generated_tokens,
|
||||||
|
outcomes: summary.task_outcomes,
|
||||||
|
};
|
||||||
|
}),
|
||||||
|
]),
|
||||||
|
);
|
||||||
|
|
||||||
|
const contrasts = Object.fromEntries(
|
||||||
|
DOMAINS.map((domain) => [
|
||||||
|
domain,
|
||||||
|
Object.fromEntries(
|
||||||
|
CONTRASTS.map((name) => {
|
||||||
|
const row =
|
||||||
|
analysis.main_t0_selected_task_analysis[domain].contrasts[name];
|
||||||
|
const metrics = Object.fromEntries(
|
||||||
|
Object.entries(row.metrics).map(([metric, value]) => [
|
||||||
|
metric,
|
||||||
|
{
|
||||||
|
point: value.right_minus_left_point,
|
||||||
|
band: value.selected_task_resampling_band,
|
||||||
|
directions: value.direction_counts,
|
||||||
|
transition: value.transition ?? null,
|
||||||
|
},
|
||||||
|
]),
|
||||||
|
);
|
||||||
|
return [
|
||||||
|
name,
|
||||||
|
{
|
||||||
|
left: row.left,
|
||||||
|
right: row.right,
|
||||||
|
metrics,
|
||||||
|
trajectory: {
|
||||||
|
sources: row.trajectory.sources,
|
||||||
|
sharedUniformExact:
|
||||||
|
row.trajectory.shared_uniform_prefix_exact,
|
||||||
|
exactTrajectories: row.trajectory.exact_trajectories,
|
||||||
|
commonPrefixTokens: row.trajectory.common_prefix_tokens,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
];
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
]),
|
||||||
|
);
|
||||||
|
|
||||||
|
const diagnostic = Object.fromEntries(
|
||||||
|
DOMAINS.map((domain) => [
|
||||||
|
domain,
|
||||||
|
{
|
||||||
|
sourceIds: analysis.multi_tape_diagnostic[domain].source_ids,
|
||||||
|
tapes: analysis.multi_tape_diagnostic[domain].tapes,
|
||||||
|
contrasts: Object.fromEntries(
|
||||||
|
CONTRASTS.map((name) => {
|
||||||
|
const row =
|
||||||
|
analysis.multi_tape_diagnostic[domain].contrasts[name];
|
||||||
|
return [
|
||||||
|
name,
|
||||||
|
{
|
||||||
|
left: row.left,
|
||||||
|
right: row.right,
|
||||||
|
success: {
|
||||||
|
matrix:
|
||||||
|
row.metrics.fixed_budget_success.matrix_task_by_tape,
|
||||||
|
tapeMeans:
|
||||||
|
row.metrics.fixed_budget_success.tape_means,
|
||||||
|
taskRangeWithinTape:
|
||||||
|
row.metrics.fixed_budget_success.task_range_within_tape,
|
||||||
|
tapeRangeWithinTask:
|
||||||
|
row.metrics.fixed_budget_success.tape_range_within_task,
|
||||||
|
},
|
||||||
|
tokens: {
|
||||||
|
matrix: row.metrics.generated_tokens.matrix_task_by_tape,
|
||||||
|
tapeMeans: row.metrics.generated_tokens.tape_means,
|
||||||
|
taskRangeWithinTape:
|
||||||
|
row.metrics.generated_tokens.task_range_within_tape,
|
||||||
|
tapeRangeWithinTask:
|
||||||
|
row.metrics.generated_tokens.tape_range_within_task,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
];
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
},
|
||||||
|
]),
|
||||||
|
);
|
||||||
|
|
||||||
|
const tasks = Object.fromEntries(
|
||||||
|
DOMAINS.map((domain) => {
|
||||||
|
const domainSources = sampling.sources
|
||||||
|
.filter((source) => source.domain === domain)
|
||||||
|
.sort((left, right) => left.within_domain_index - right.within_domain_index);
|
||||||
|
return [
|
||||||
|
domain,
|
||||||
|
domainSources.map((source) => {
|
||||||
|
const conditions = Object.fromEntries(
|
||||||
|
CONDITIONS.map((condition) => {
|
||||||
|
const evaluationRow = evaluationIndex.get(
|
||||||
|
`${source.id}\0T0\0${condition}`,
|
||||||
|
);
|
||||||
|
const samplingRow = sampleIndex.get(
|
||||||
|
`${source.id}\0T0\0${condition}`,
|
||||||
|
);
|
||||||
|
return [
|
||||||
|
condition,
|
||||||
|
{
|
||||||
|
success: success(evaluationRow),
|
||||||
|
outcome: evaluationRow.task_outcome,
|
||||||
|
tokens: evaluationRow.generated_tokens,
|
||||||
|
naturalEos: evaluationRow.hit_eos,
|
||||||
|
truncated: evaluationRow.stopped_at_max_new_tokens,
|
||||||
|
trajectoryHash:
|
||||||
|
evaluationRow.generated_token_ids_sha256.slice(0, 12),
|
||||||
|
firstTokenIds: samplingRow.generated_token_ids.slice(0, 8),
|
||||||
|
},
|
||||||
|
];
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const taskContrasts = Object.fromEntries(
|
||||||
|
CONTRASTS.map((name) => {
|
||||||
|
const sourceRow =
|
||||||
|
analysis.main_t0_selected_task_analysis[domain].contrasts[name];
|
||||||
|
const successRow =
|
||||||
|
sourceRow.metrics.fixed_budget_success.by_source.find(
|
||||||
|
(row) => row.source_id === source.id,
|
||||||
|
);
|
||||||
|
const tokenRow = sourceRow.metrics.generated_tokens.by_source.find(
|
||||||
|
(row) => row.source_id === source.id,
|
||||||
|
);
|
||||||
|
const trajectoryRow = sourceRow.trajectory.rows.find(
|
||||||
|
(row) => row.source_id === source.id,
|
||||||
|
);
|
||||||
|
return [
|
||||||
|
name,
|
||||||
|
{
|
||||||
|
successDelta: successRow.right_minus_left,
|
||||||
|
tokenDelta: tokenRow.right_minus_left,
|
||||||
|
commonPrefixTokens: trajectoryRow.common_prefix_tokens,
|
||||||
|
exactTrajectory: trajectoryRow.token_ids_exact,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
return {
|
||||||
|
id: source.id,
|
||||||
|
index: source.within_domain_index,
|
||||||
|
conditions,
|
||||||
|
contrasts: taskContrasts,
|
||||||
|
diagnosticTapes: source.runs.map((run) => run.tape_label),
|
||||||
|
};
|
||||||
|
}),
|
||||||
|
];
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
const exampleSource = sampling.sources.find(
|
||||||
|
(source) => source.id === "HumanEval/31",
|
||||||
|
);
|
||||||
|
const exampleRun = exampleSource.runs.find((run) => run.tape_label === "T0");
|
||||||
|
const compact = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
protocolId: sampling.protocol_id,
|
||||||
|
capturedAt: sampling.captured_at,
|
||||||
|
model: {
|
||||||
|
repo: sampling.model.repo,
|
||||||
|
revision: sampling.model.revision,
|
||||||
|
checkpointIdentity: sampling.model.checkpoint_identity,
|
||||||
|
dtype: sampling.model.dtype,
|
||||||
|
},
|
||||||
|
artifactHashes: Object.fromEntries(
|
||||||
|
Object.entries(inputBytes).map(([name, bytes]) => [name, sha256(bytes)]),
|
||||||
|
),
|
||||||
|
grid: {
|
||||||
|
formalSources: sampling.summary.sources,
|
||||||
|
formalRuns: sampling.summary.runs,
|
||||||
|
formalOutputs: sampling.summary.outputs,
|
||||||
|
mainT0Outputs: evaluation.summary.main_t0.outputs,
|
||||||
|
diagnosticAdditionalOutputs:
|
||||||
|
evaluation.summary.diagnostic_additional_t1_t3.outputs,
|
||||||
|
naturalEos: sampling.summary.natural_eos,
|
||||||
|
budgetTruncated: sampling.summary.budget_truncated,
|
||||||
|
uniqueTrajectories: sampling.summary.unique_generated_token_hashes,
|
||||||
|
promptHashesExact: sampling.source_contract.prompt_hash_audit.exact,
|
||||||
|
torchRngUnchangedRuns: sampling.summary.torch_rng_unchanged_runs,
|
||||||
|
},
|
||||||
|
sampler: {
|
||||||
|
name: sampling.generation_contract.decode,
|
||||||
|
temperature: sampling.generation_contract.distribution_temperature,
|
||||||
|
topP: sampling.generation_contract.distribution_top_p,
|
||||||
|
softmaxDtype: sampling.generation_contract.softmax_dtype,
|
||||||
|
cdfDtype: sampling.generation_contract.cdf_dtype,
|
||||||
|
uniformDtype: sampling.generation_contract.uniform_dtype,
|
||||||
|
transformersGenerateCalled:
|
||||||
|
sampling.generation_contract.transformers_generate_called,
|
||||||
|
torchMultinomialCalled:
|
||||||
|
sampling.generation_contract.torch_multinomial_called,
|
||||||
|
commonRandomNumbers:
|
||||||
|
sampling.seed_contract.explicit_common_random_numbers,
|
||||||
|
},
|
||||||
|
conditionTable,
|
||||||
|
contrasts,
|
||||||
|
diagnostic,
|
||||||
|
tasks,
|
||||||
|
outcomes: {
|
||||||
|
code: evaluation.summary.main_t0.by_domain.code.task_outcomes,
|
||||||
|
math: evaluation.summary.main_t0.by_domain.math.task_outcomes,
|
||||||
|
},
|
||||||
|
reproduction: reproduction.summary,
|
||||||
|
bootstrap: analysis.bootstrap_contract,
|
||||||
|
uniformExample: {
|
||||||
|
sourceId: exampleSource.id,
|
||||||
|
tape: exampleRun.tape_label,
|
||||||
|
uniformUint64FirstEightHex: exampleRun.uniform_uint64_first_eight_hex,
|
||||||
|
uniformFloat32FirstEight: exampleRun.uniform_float32_first_eight,
|
||||||
|
conditions: Object.fromEntries(
|
||||||
|
exampleRun.outputs.map((output) => [
|
||||||
|
output.condition,
|
||||||
|
{
|
||||||
|
generatedTokenIds: output.generated_token_ids.slice(0, 8),
|
||||||
|
generatedTokens: output.generated_tokens,
|
||||||
|
naturalEos: output.hit_eos,
|
||||||
|
},
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
},
|
||||||
|
deviations: [
|
||||||
|
{
|
||||||
|
id: "prompt-hash-correction",
|
||||||
|
severity: "corrected-before-output",
|
||||||
|
summary:
|
||||||
|
"The first manifest mislabeled routing-probe hashes as Chat prompt hashes; all 256 Chat hashes were corrected before model output.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "gold-loaded-in-runner",
|
||||||
|
severity: "reported-process-deviation",
|
||||||
|
summary:
|
||||||
|
"The reused runner loaded gold for a post-decode narrow task_score before generation finished. Gold never entered prompts, logits, sampling, selection, or the authoritative evaluator.",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
evidenceBoundary: [
|
||||||
|
"HumanEval and GSM8K remain separate.",
|
||||||
|
"Selected-task bands cover only the frozen 32-task frame under T0.",
|
||||||
|
"T1-T3 are sensitivity diagnostics, not extra independent tasks.",
|
||||||
|
"The explicit sampler uses the official .3/.95 distribution but is not a torch.multinomial trajectory.",
|
||||||
|
"Period prompts are counterfactual and not official-valid chats.",
|
||||||
|
],
|
||||||
|
};
|
||||||
|
|
||||||
|
await writeFile(paths.output, `${JSON.stringify(compact, null, 2)}\n`);
|
||||||
|
const outputBytes = await readFile(paths.output);
|
||||||
|
process.stdout.write(
|
||||||
|
`${JSON.stringify(
|
||||||
|
{
|
||||||
|
output: paths.output,
|
||||||
|
bytes: outputBytes.length,
|
||||||
|
sha256: sha256(outputBytes),
|
||||||
|
tasks: Object.values(tasks).reduce((sum, rows) => sum + rows.length, 0),
|
||||||
|
formalOutputs: compact.grid.formalOutputs,
|
||||||
|
reproduction: compact.reproduction,
|
||||||
|
},
|
||||||
|
null,
|
||||||
|
2,
|
||||||
|
)}\n`,
|
||||||
|
);
|
||||||
@@ -0,0 +1,217 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
|
||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { readFile } from "node:fs/promises";
|
||||||
|
import process from "node:process";
|
||||||
|
|
||||||
|
const PROTOCOL_ID = "llm-atlas-deepseek-chat-task-bootstrap-crn-v1";
|
||||||
|
const CONDITIONS = ["s0_eos", "s1_eos", "s0_period", "s1_period"];
|
||||||
|
const EXPECTED_HASHES = {
|
||||||
|
sampling: "ea0607f2b197fac3f794655c1538ce1f9a1cb072d637eb31d35573682e311809",
|
||||||
|
evaluation: "82b2fc5d1f854a7e0cd7aba7c70ff74d733d24221522a4f92b962478c76177ab",
|
||||||
|
replay: "6519947e2fa4327c1ba2cc506b0861f172a6bdbd4f2d6787447edaf8fbcac508",
|
||||||
|
reproduction: "63ed39e5dcdbc2a30e516172f3657dfa453ff3b23d5bd5c49c734939242a70f5",
|
||||||
|
analysis: "9ab17561ced930a668c082141f7c6e013cbda70e42b09de63d41f1b82c01a6ae",
|
||||||
|
manifest: "6313e70536c464fe598a93035576752418f08016dfd60ac246437c3b43bf2ae1",
|
||||||
|
};
|
||||||
|
const DEFAULT_PATHS = {
|
||||||
|
sampling: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json",
|
||||||
|
evaluation: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json",
|
||||||
|
replay: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-replay.json",
|
||||||
|
reproduction: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-reproduction.json",
|
||||||
|
analysis: "src/data/deepseek-v2-lite-chat-task-bootstrap-crn-analysis.json",
|
||||||
|
manifest: "research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json",
|
||||||
|
};
|
||||||
|
|
||||||
|
function sha256(value) {
|
||||||
|
return createHash("sha256").update(value).digest("hex");
|
||||||
|
}
|
||||||
|
|
||||||
|
function canonicalIntegerArray(values) {
|
||||||
|
return `[${values.map((value) => value.toString()).join(",")}]`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function tapeUint64(tape, sourceId, step) {
|
||||||
|
const payload = `${PROTOCOL_ID}\0uniform\0${tape}\0${sourceId}\0${step}`;
|
||||||
|
return createHash("sha256").update(payload).digest().readBigUInt64BE(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
function assert(value, message) {
|
||||||
|
if (!value) {
|
||||||
|
throw new Error(message);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const loaded = {};
|
||||||
|
for (const [name, path] of Object.entries(DEFAULT_PATHS)) {
|
||||||
|
const bytes = await readFile(path);
|
||||||
|
assert(
|
||||||
|
sha256(bytes) === EXPECTED_HASHES[name],
|
||||||
|
`${name}: file SHA-256 differs`,
|
||||||
|
);
|
||||||
|
loaded[name] = JSON.parse(bytes);
|
||||||
|
}
|
||||||
|
const { sampling, evaluation, replay, reproduction, analysis, manifest } = loaded;
|
||||||
|
for (const [name, payload] of Object.entries(loaded)) {
|
||||||
|
assert(payload.protocol_id === PROTOCOL_ID, `${name}: protocol ID differs`);
|
||||||
|
}
|
||||||
|
|
||||||
|
assert(sampling.execution_mode === "formal", "sampling: not formal mode");
|
||||||
|
assert(sampling.sources.length === 64, "sampling: source count differs");
|
||||||
|
assert(sampling.summary.runs === 88, "sampling: run count differs");
|
||||||
|
assert(sampling.summary.outputs === 352, "sampling: output count differs");
|
||||||
|
assert(sampling.summary.natural_eos === 343, "sampling: EOS count differs");
|
||||||
|
assert(sampling.summary.budget_truncated === 9, "sampling: truncation count differs");
|
||||||
|
assert(
|
||||||
|
sampling.summary.torch_rng_unchanged_runs === 88,
|
||||||
|
"sampling: RNG nonconsumption count differs",
|
||||||
|
);
|
||||||
|
assert(
|
||||||
|
sampling.source_contract.prompt_hash_audit.exact === 256,
|
||||||
|
"sampling: prompt audit differs",
|
||||||
|
);
|
||||||
|
|
||||||
|
const manifestBySource = new Map(
|
||||||
|
manifest.sources.map((source) => [source.id, source]),
|
||||||
|
);
|
||||||
|
const samplingKeys = new Set();
|
||||||
|
let uniformOutputHashesExact = 0;
|
||||||
|
let diagnosticSources = 0;
|
||||||
|
for (const source of sampling.sources) {
|
||||||
|
const frozen = manifestBySource.get(source.id);
|
||||||
|
assert(frozen, `${source.id}: absent from manifest`);
|
||||||
|
assert(source.domain === frozen.domain, `${source.id}: domain differs`);
|
||||||
|
assert(
|
||||||
|
source.within_domain_index === frozen.domain_index,
|
||||||
|
`${source.id}: domain index differs`,
|
||||||
|
);
|
||||||
|
const expectedTapes = [...frozen.main_tapes, ...frozen.diagnostic_tapes];
|
||||||
|
const observedTapes = source.runs.map((run) => run.tape_label);
|
||||||
|
assert(
|
||||||
|
JSON.stringify(observedTapes) === JSON.stringify(expectedTapes),
|
||||||
|
`${source.id}: tape assignment differs`,
|
||||||
|
);
|
||||||
|
diagnosticSources += source.runs.length === 4;
|
||||||
|
for (const run of source.runs) {
|
||||||
|
assert(run.torch_rng_unchanged, `${source.id}/${run.tape_label}: RNG changed`);
|
||||||
|
assert(
|
||||||
|
JSON.stringify(run.outputs.map((output) => output.condition))
|
||||||
|
=== JSON.stringify(CONDITIONS),
|
||||||
|
`${source.id}/${run.tape_label}: condition order differs`,
|
||||||
|
);
|
||||||
|
for (const output of run.outputs) {
|
||||||
|
const key = `${source.id}\0${run.tape_label}\0${output.condition}`;
|
||||||
|
assert(!samplingKeys.has(key), `${key}: duplicate sampling key`);
|
||||||
|
samplingKeys.add(key);
|
||||||
|
assert(
|
||||||
|
output.prompt_token_ids_sha256
|
||||||
|
=== frozen.chat_generation_prompt_token_ids_sha256[output.condition],
|
||||||
|
`${key}: prompt hash differs`,
|
||||||
|
);
|
||||||
|
const uniforms = Array.from(
|
||||||
|
{ length: output.uniform_steps_consumed },
|
||||||
|
(_, step) => tapeUint64(run.tape_label, source.id, step),
|
||||||
|
);
|
||||||
|
assert(
|
||||||
|
sha256(canonicalIntegerArray(uniforms))
|
||||||
|
=== output.uniform_uint64_prefix_sha256,
|
||||||
|
`${key}: uniform prefix hash differs`,
|
||||||
|
);
|
||||||
|
uniformOutputHashesExact += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert(diagnosticSources === 8, "sampling: diagnostic source count differs");
|
||||||
|
assert(samplingKeys.size === 352, "sampling: unique key count differs");
|
||||||
|
assert(uniformOutputHashesExact === 352, "sampling: uniform audit differs");
|
||||||
|
|
||||||
|
assert(evaluation.rows.length === 352, "evaluation: row count differs");
|
||||||
|
const evaluationKeys = new Set(
|
||||||
|
evaluation.rows.map(
|
||||||
|
(row) => `${row.source_id}\0${row.tape_label}\0${row.condition}`,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
assert(evaluationKeys.size === 352, "evaluation: duplicate keys");
|
||||||
|
assert(
|
||||||
|
[...evaluationKeys].every((key) => samplingKeys.has(key)),
|
||||||
|
"evaluation: cell key absent from sampling",
|
||||||
|
);
|
||||||
|
assert(
|
||||||
|
evaluation.summary.main_t0.outputs === 256,
|
||||||
|
"evaluation: T0 output count differs",
|
||||||
|
);
|
||||||
|
assert(
|
||||||
|
evaluation.summary.main_t0.by_domain.code.fixed_budget_success === 59,
|
||||||
|
"evaluation: code success count differs",
|
||||||
|
);
|
||||||
|
assert(
|
||||||
|
evaluation.summary.main_t0.by_domain.math.fixed_budget_success === 71,
|
||||||
|
"evaluation: math success count differs",
|
||||||
|
);
|
||||||
|
|
||||||
|
assert(replay.execution_mode === "replay", "replay: mode differs");
|
||||||
|
assert(replay.sources.length === 16, "replay: source count differs");
|
||||||
|
assert(replay.summary.outputs === 64, "replay: output count differs");
|
||||||
|
assert(
|
||||||
|
replay.summary.torch_rng_unchanged_runs === 16,
|
||||||
|
"replay: RNG nonconsumption count differs",
|
||||||
|
);
|
||||||
|
assert(
|
||||||
|
reproduction.summary.cells === 64
|
||||||
|
&& reproduction.summary.all_preregistered_fields_exact === 64,
|
||||||
|
"reproduction: exact cell count differs",
|
||||||
|
);
|
||||||
|
assert(
|
||||||
|
Object.values(reproduction.summary.by_field).every((count) => count === 64),
|
||||||
|
"reproduction: a field is not 64/64 exact",
|
||||||
|
);
|
||||||
|
|
||||||
|
assert(
|
||||||
|
analysis.bootstrap_contract.resamples === 10000
|
||||||
|
&& analysis.bootstrap_contract.seed === 1364512825,
|
||||||
|
"analysis: bootstrap contract differs",
|
||||||
|
);
|
||||||
|
let crnContrastChecks = 0;
|
||||||
|
for (const domain of ["code", "math"]) {
|
||||||
|
const domainAnalysis = analysis.main_t0_selected_task_analysis[domain];
|
||||||
|
assert(domainAnalysis.tasks === 32, `${domain}: task count differs`);
|
||||||
|
for (const contrast of Object.values(domainAnalysis.contrasts)) {
|
||||||
|
assert(
|
||||||
|
contrast.trajectory.shared_uniform_prefix_exact === 32,
|
||||||
|
`${domain}: contrast CRN audit differs`,
|
||||||
|
);
|
||||||
|
crnContrastChecks += contrast.trajectory.shared_uniform_prefix_exact;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert(crnContrastChecks === 256, "analysis: CRN contrast total differs");
|
||||||
|
|
||||||
|
process.stdout.write(
|
||||||
|
`${JSON.stringify(
|
||||||
|
{
|
||||||
|
passed: true,
|
||||||
|
protocol_id: PROTOCOL_ID,
|
||||||
|
files: EXPECTED_HASHES,
|
||||||
|
formal: {
|
||||||
|
sources: sampling.sources.length,
|
||||||
|
runs: sampling.summary.runs,
|
||||||
|
outputs: sampling.summary.outputs,
|
||||||
|
prompt_hashes_exact: sampling.source_contract.prompt_hash_audit.exact,
|
||||||
|
uniform_output_hashes_exact: uniformOutputHashesExact,
|
||||||
|
torch_rng_unchanged_runs: sampling.summary.torch_rng_unchanged_runs,
|
||||||
|
},
|
||||||
|
evaluation: {
|
||||||
|
rows: evaluation.rows.length,
|
||||||
|
main_t0_outputs: evaluation.summary.main_t0.outputs,
|
||||||
|
code_success: evaluation.summary.main_t0.by_domain.code.fixed_budget_success,
|
||||||
|
math_success: evaluation.summary.main_t0.by_domain.math.fixed_budget_success,
|
||||||
|
},
|
||||||
|
reproduction: reproduction.summary,
|
||||||
|
analysis: {
|
||||||
|
bootstrap_resamples: analysis.bootstrap_contract.resamples,
|
||||||
|
crn_contrast_checks: crnContrastChecks,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
null,
|
||||||
|
2,
|
||||||
|
)}\n`,
|
||||||
|
);
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user