feat: map DeepSeek Chat sampling robustness
This commit is contained in:
wuyang
+14
-4
@@ -1,6 +1,6 @@
|
||||
# 持续进度
|
||||
|
||||
最后更新：2026-07-29
|
||||
最后更新：2026-07-30
|
||||
|
||||
## 总体状态
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
| 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
|
||||
| Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
|
||||
| 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
|
||||
| DeepSeek 专题 | 五轮实证进行中 | 99% | 扩大 completion/task 样本与 sampling 复现，再推进 SM90 FlashMLA、FP8/pipeline、干预式 mediation 与 R1-like RL 复现 |
|
||||
| DeepSeek 专题 | 六轮实证进行中 | 99% | 扩大 sampling 的 source/task 覆盖，再推进干预式 mediation、SM90 FlashMLA、FP8/pipeline 与 R1-like RL 复现 |
|
||||
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
|
||||
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
|
||||
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
|
||||
@@ -41,7 +41,7 @@
|
||||
- [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||
- [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 十九联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十六个原创交互视图。
|
||||
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 二十联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十七个原创交互视图。
|
||||
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
||||
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
||||
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
||||
@@ -248,11 +248,18 @@
|
||||
- [x] 全深度新进程复跑的 4 / 4 source objects（去 runtime 后）、1,856 / 1,856 hidden tensor hashes、1,664 / 1,664 ordered-route hashes 与 1,664 / 1,664 route-weight hashes 全部 exact；复现核对抓到并修正 source-level content-token hash 的循环变量缺陷，正式与复跑文件随后全部重跑。
|
||||
- [x] 第十九个 DeepSeek 交互实验用四页签分开 128→512 completion、Math/Code evaluator、29-stage hidden divergence 与 26-gate route divergence；桌面、390px 移动端、键盘 tab、分域/分边/分指标交互与无横向溢出闸门通过。
|
||||
- [x] DeepSeek completion/full-depth 版本以源提交 `8bb488f`、不可变镜像 `20260729T162625Z-8bb488f` 发布；OCI index digest `sha256:297fef91…b0c02`，复用 NAS `12010→8080`、NPM host 31 / cert 41、门户 `LLM ATLAS / projects / 180`，HTTPS/2、gzip 与十六套生产 Chrome 回归全通过；保留 `20260729T144827Z-96443d4` 回滚。
|
||||
- [x] DeepSeek Round 06 在看到正式结果前预注册 official sampling 合同：4 条固定 source、8 个 SHA-256 派生 base seed、8 条固定 batch 行、`temperature=.3 / top_p=.95 / top_k=0` 与 512-token 统一上限；明确 batch-seed aligned 不是 common-random-number paired design。
|
||||
- [x] 16-token smoke 的 32 / 32 prompt hash 与 greedy baseline exact，R0 同进程重放 32 / 32 全合同 exact，R0/R1 的 32 个同格中 18 格分叉；正式 256 条 sampled outputs 全部落盘，251 条 natural EOS、5 条截断、242 个完整 token trajectory hashes。
|
||||
- [x] sampling 独立 evaluator 把 stopping、task terminal、coverage 与 correctness 分账：单条 GSM8K 的 64 个样本中 62 个 strict exact；单条 HumanEval 的 64 个样本全部 AST 可解析并进入 pinned sandbox，63 个通过 official tests。两条数学失败与一个偶数 prime-check 失败均定位到具体推理/代码错误。
|
||||
- [x] 新进程只复跑 R0/R1 的 64 格；run seed、prompt hash、完整 generated token IDs、decoded text、EOS、truncation、CPU RNG pre-state 与 CUDA RNG pre-state 八项均 64 / 64 exact，不冒充 256 / 256。
|
||||
- [x] Round 06 四份冻结 artifact SHA-256 为 formal `46c7edf…345af`、eval `078f486…24c4d8`、rerun `72d050e…91f6a0`、reproduction `4d59a77…f82d15`；compact builder 强制验证三段 hash 链与 64 / 64 复现闸门。
|
||||
- [x] 第二十个 DeepSeek 交互实验以四页签讲解八 seed 轨迹显微镜、Math/Code 失败账、aligned/nearest 样本集合比较与随机性×复现合同；完整协议、结果审计、运行脚本与 compact 数据均已记录在项目本地。
|
||||
- [x] Round 06 本地闸门通过：80 个受检文件零诊断，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；DeepSeek 历史全量与 sampling 专项真实 Chrome 回归均通过，桌面/390px 移动端无文档级溢出、无 offender、无运行时异常。
|
||||
|
||||
## 正在进行
|
||||
|
||||
- [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。
|
||||
- [ ] DeepSeek 五轮下一闸门：扩大 GSM8K/HumanEval 与语言 source 样本，加入 sampling robustness 和干预式 mediation；再推进 SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||
- [ ] DeepSeek 六轮下一闸门：扩大 GSM8K/HumanEval 与语言 source 的 sampling 覆盖，加入干预式 mediation；再推进 SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||
- [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
|
||||
- [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
|
||||
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
|
||||
@@ -426,6 +433,9 @@
|
||||
| 2026-07-29 | 开放工件证据使用 O / D / X / S / U 五种身份 | 观测、推导、本机执行、合成探针与未决矛盾不互相冒充；FlashKDA 作者 benchmark 不写成本机 benchmark |
|
||||
| 2026-07-29 | `A_log [128]` 与 expected `[96]` 保持未决 | 并列报告 checkpoint、config、remote code 与 kernel API；等待官方 loader / 修订解释，不擅自 reshape |
|
||||
| 2026-07-29 | K3 三轮开放工件里程碑用不可变镜像 `20260729T044605Z-be2b291` 发布 | OCI digest `sha256:99aa953e…d00cdf`；复用 `12010→8080`、NPM host 31 / cert 41、门户 order 180 与公开 Forgejo；十六套生产 Chrome 回归通过，保留 `20260729T040336Z-b669615` 回滚 |
|
||||
| 2026-07-30 | DeepSeek Round 06 在结果前冻结多种子 sampling 协议 | official `.3/.95` sampling、显式 top-k 0、四条 source、八个 SHA-256 seed 与固定八行 batch；batch-seed aligned 不冒充 common random numbers |
|
||||
| 2026-07-30 | sampled completion 与任务失败分账 | 256 条中 251 natural EOS、242 unique token trajectories；Math 62/64、Code 63/64，具体错误回到推理与官方 tests |
|
||||
| 2026-07-30 | sampling 的随机性与复现同时过闸 | R0/R1 31/32 同格分叉；新进程复跑 64 格的 seed、prompt、token、text、stop 与 CPU/CUDA RNG pre-state 八项全部 exact |
|
||||
|
||||
## 未决问题
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读，
|
||||
语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
|
||||
以及 86 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||
以及 87 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||
8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
|
||||
第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、
|
||||
497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图
|
||||
@@ -27,8 +27,8 @@
|
||||
`sm_120a` wheel，在 RTX 5090 上完成 6/6 官方参考 exact-match 和 K3 fixed / varlen 形状计时。详见
|
||||
[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
|
||||
[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
|
||||
DeepSeek 五轮专题以 24 张问题账、10 次技术转向、
|
||||
19 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||
DeepSeek 六轮专题以 24 张问题账、10 次技术转向、
|
||||
20 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||
并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、
|
||||
MLA/HF eager cache shapes 与 `31/31` exact 独立复跑；进一步用真实 layer-1 权重执行官方 V3
|
||||
naive/absorb 路径，实际写入 576 元素 latent cache，并以 FP32 将两种结合顺序的最大误差压到
|
||||
@@ -72,6 +72,12 @@ checkpoint 又执行 embedding、27 层、final norm 与 26 个 MoE gate 的 pro
|
||||
全深度 trace，在 1,537 个精确 interior content tokens / condition 上记录
|
||||
1,918,176 次 top-6 路由决定。新进程子集复跑达到 `1,856 / 1,856` hidden tensor hashes、
|
||||
`1,664 / 1,664` ordered route hashes 与 `1,664 / 1,664` route-weight hashes exact。
|
||||
Round 06 冻结 4 条 source、8 个 SHA-256 派生 seed 与 8 个 condition，启用 checkpoint
|
||||
随附的 `temperature=.3 / top_p=.95` 并显式 `top_k=0`，生成 256 条 sampled outputs：
|
||||
251 条 natural EOS，242 个完整 token trajectory hashes；单条 GSM8K 的 strict exact
|
||||
为 `62 / 64`，单条 HumanEval 的官方 tests pass 为 `63 / 64`，不外推为 benchmark。
|
||||
全新进程复跑 R0/R1 后，run seed、prompt hash、完整 token IDs、文本、停止状态与
|
||||
CPU/CUDA RNG pre-state 八项均为 `64 / 64` exact。
|
||||
详见
|
||||
[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md) 与
|
||||
[DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)、
|
||||
@@ -85,7 +91,9 @@ checkpoint 又执行 embedding、27 层、final norm 与 26 个 MoE gate 的 pro
|
||||
[DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md](./research/DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md) 与
|
||||
[DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md](./research/DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md)，以及
|
||||
[DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_BEHAVIOR_AUDIT.md) 与
|
||||
[DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md)。
|
||||
[DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md)、
|
||||
[DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md](./research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md) 与
|
||||
[DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md)。
|
||||
其余专题按进度账本持续扩建。
|
||||
|
||||
## 本地开发
|
||||
|
||||
@@ -672,3 +672,85 @@ See `research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_PROTOCOL.md` and
|
||||
`research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md` for the OOM
|
||||
amendment, task evaluators, completion table, depth curves, reproduction
|
||||
audit, artifact hashes, and claim boundaries.
|
||||
|
||||
## Preregistered multi-seed Chat sampling
|
||||
|
||||
`v2_lite_chat_sampling_probe.py` keeps the same four first-ranked sources,
|
||||
eight prompt conditions, 512-token cap, BF16 checkpoint, and offload contract.
|
||||
It enables the pinned official sampling values and explicitly disables top-k:
|
||||
|
||||
```text
|
||||
do_sample=true · temperature=.3 · top_p=.95 · top_k=0
|
||||
4 sources × 8 SHA-256-derived seeds × 8 conditions = 256 outputs
|
||||
```
|
||||
|
||||
The eight rows of one source remain in a fixed batch. They are batch-seed
|
||||
aligned, not common-random-number pairs: Transformers samples the whole batch
|
||||
with one `torch.multinomial` call.
|
||||
|
||||
```bash
|
||||
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps \
|
||||
python -B experiments/deepseek/v2_lite_chat_sampling_probe.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite-chat \
|
||||
--reference-routing-json \
|
||||
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||
--greedy-json src/data/deepseek-v2-lite-chat-completion-512.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--base-seeds \
|
||||
19683830 1560062173 3978401375 1280933274 \
|
||||
1467459869 1297359489 2722953988 3330978061 \
|
||||
--max-new-tokens 512 \
|
||||
--gpu-memory 28GiB \
|
||||
--cpu-memory 80GiB \
|
||||
--output src/data/deepseek-v2-lite-chat-sampling.json
|
||||
```
|
||||
|
||||
The independent evaluator preserves stopping, semantic terminal state,
|
||||
coverage, and correctness as separate ledgers. Exact duplicate code candidates
|
||||
reuse a cache keyed by candidate, task tests, and sandbox-harness hashes:
|
||||
|
||||
```bash
|
||||
python -B experiments/deepseek/v2_lite_chat_sampling_evaluator.py \
|
||||
--sampling-json src/data/deepseek-v2-lite-chat-sampling.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--sandbox-image \
|
||||
python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
|
||||
--output src/data/deepseek-v2-lite-chat-sampling-eval.json
|
||||
```
|
||||
|
||||
A fresh Python process reruns R0/R1. The reproduction comparer checks run
|
||||
seeds, prompt hashes, complete token IDs, text, stopping state, and CPU/CUDA
|
||||
RNG pre-state hashes:
|
||||
|
||||
```bash
|
||||
python -B experiments/deepseek/v2_lite_chat_sampling_reproduction.py \
|
||||
--formal-json src/data/deepseek-v2-lite-chat-sampling.json \
|
||||
--rerun-json \
|
||||
src/data/deepseek-v2-lite-chat-sampling-repro-r0r1.json \
|
||||
--output \
|
||||
src/data/deepseek-v2-lite-chat-sampling-reproduction.json
|
||||
```
|
||||
|
||||
Formal results are 251/256 natural EOS and 242/256 globally unique complete
|
||||
token trajectories. Strict GSM8K exact is 62/64; HumanEval official-test pass
|
||||
is 63/64. These repeat one task per domain and are not benchmark estimates.
|
||||
The R0/R1 fresh-process rerun reproduces all eight preregistered fields in
|
||||
64/64 cells.
|
||||
|
||||
```text
|
||||
formal 46c7edfce1409e798d6b0f06e905dd3a9d4ab6911acd8e42e2ec9d06d65345af
|
||||
eval 078f486e021ffd0e4af8ad942d001a9f2429933b2fb0fb3523b7d1da5b24c4d8
|
||||
rerun 72d050eace531bb38fb2592e39dc7f1b9c9003306c92c568dcd4acaaa691f6a0
|
||||
compare 4d59a775943459604ea2d9976bb3be01ff7b3b08d71eb0ca22c376122ef82d15
|
||||
```
|
||||
|
||||
See `research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md` and
|
||||
`research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md` for seed derivation,
|
||||
smoke gates, source-condition diversity tables, edge-set comparisons, task
|
||||
failure cases, exact reproduction scope, primary sources, and non-claims.
|
||||
|
||||
@@ -0,0 +1,560 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Evaluate sampled DeepSeek-V2-Lite-Chat completions safely.
|
||||
|
||||
Stopping, task-terminal state, evaluator coverage, and correctness remain
|
||||
separate ledgers. HumanEval candidates execute only in fresh networkless,
|
||||
read-only Docker containers. An exact candidate+task+harness cache avoids
|
||||
re-executing duplicate code while preserving one result row per sampled cell.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from statistics import mean
|
||||
from typing import Any
|
||||
|
||||
import v2_lite_chat_completion_evaluator as completion
|
||||
|
||||
|
||||
COMPARISONS = (
|
||||
("system_eos", "s0_eos", "s1_eos"),
|
||||
("system_bos", "s0_bos", "s1_bos"),
|
||||
("system_x", "s0_x", "s1_x"),
|
||||
("system_period", "s0_period", "s1_period"),
|
||||
("bos_at_s0", "s0_eos", "s0_bos"),
|
||||
("bos_at_s1", "s1_eos", "s1_bos"),
|
||||
("x_at_s0", "s0_eos", "s0_x"),
|
||||
("x_at_s1", "s1_eos", "s1_x"),
|
||||
("period_at_s0", "s0_eos", "s0_period"),
|
||||
("period_at_s1", "s1_eos", "s1_period"),
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--sampling-json", type=Path, required=True)
|
||||
parser.add_argument("--human-eval", type=Path, required=True)
|
||||
parser.add_argument("--gsm8k", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--sandbox-image",
|
||||
required=True,
|
||||
help="Pinned image reference including @sha256 digest.",
|
||||
)
|
||||
parser.add_argument("--timeout-seconds", type=float, default=5.0)
|
||||
parser.add_argument("--skip-code-execution", action="store_true")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def evaluate_code_cached(
|
||||
*,
|
||||
text: str,
|
||||
hit_eos: bool,
|
||||
task: dict[str, Any],
|
||||
image: str,
|
||||
timeout_seconds: float,
|
||||
skip_execution: bool,
|
||||
cache: dict[tuple[str, str, str], dict[str, Any]],
|
||||
) -> tuple[dict[str, Any], bool]:
|
||||
extracted = completion.extract_code(
|
||||
text,
|
||||
task["prompt"],
|
||||
task["entry_point"],
|
||||
)
|
||||
harness = completion.sandbox_harness(
|
||||
extracted["candidate"],
|
||||
task["test"],
|
||||
task["entry_point"],
|
||||
)
|
||||
cache_key = (
|
||||
extracted["candidate_sha256"],
|
||||
hashlib.sha256(task["test"].encode()).hexdigest(),
|
||||
hashlib.sha256(harness.encode()).hexdigest(),
|
||||
)
|
||||
cache_hit = cache_key in cache
|
||||
if cache_hit:
|
||||
execution = copy.deepcopy(cache[cache_key])
|
||||
else:
|
||||
execution = {
|
||||
"status": "not_run",
|
||||
"return_code": None,
|
||||
"runtime_ms": None,
|
||||
"harness_sha256": cache_key[2],
|
||||
}
|
||||
if extracted["python_ast_parse"] and not skip_execution:
|
||||
execution = completion.execute_code(
|
||||
extracted["candidate"],
|
||||
task,
|
||||
image,
|
||||
timeout_seconds,
|
||||
)
|
||||
cache[cache_key] = copy.deepcopy(execution)
|
||||
passed = execution["status"] == "passed"
|
||||
semantic_terminal = bool(
|
||||
hit_eos
|
||||
or extracted["closed_code_fence"]
|
||||
or passed
|
||||
)
|
||||
result = {
|
||||
key: value
|
||||
for key, value in extracted.items()
|
||||
if key != "candidate"
|
||||
} | {
|
||||
"semantic_terminal": semantic_terminal,
|
||||
"evaluator_covered": (
|
||||
extracted["python_ast_parse"]
|
||||
and execution["status"] != "not_run"
|
||||
),
|
||||
"execution": execution,
|
||||
"fixed_budget_tests_pass": passed,
|
||||
"strict_complete_tests_pass": (
|
||||
passed and semantic_terminal
|
||||
),
|
||||
"execution_cache_hit": cache_hit,
|
||||
}
|
||||
return result, cache_hit
|
||||
|
||||
|
||||
def output_success(row: dict[str, Any]) -> dict[str, int | None]:
|
||||
evaluation = row["task_evaluation"]
|
||||
if row["domain"] == "math":
|
||||
return {
|
||||
"fixed_budget": int(
|
||||
evaluation["fixed_budget_numeric_exact"]
|
||||
),
|
||||
"strict_complete": int(
|
||||
evaluation["strict_complete_numeric_exact"]
|
||||
),
|
||||
}
|
||||
if row["domain"] == "code":
|
||||
return {
|
||||
"fixed_budget": int(
|
||||
evaluation["fixed_budget_tests_pass"]
|
||||
),
|
||||
"strict_complete": int(
|
||||
evaluation["strict_complete_tests_pass"]
|
||||
),
|
||||
}
|
||||
return {
|
||||
"fixed_budget": None,
|
||||
"strict_complete": None,
|
||||
}
|
||||
|
||||
|
||||
def summarize_group(rows: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
math_rows = [
|
||||
row for row in rows if row["domain"] == "math"
|
||||
]
|
||||
code_rows = [
|
||||
row for row in rows if row["domain"] == "code"
|
||||
]
|
||||
answers = Counter(
|
||||
row["task_evaluation"]["predicted_final"]
|
||||
for row in math_rows
|
||||
if row["task_evaluation"]["predicted_final"] is not None
|
||||
)
|
||||
max_count = max(answers.values(), default=0)
|
||||
modes = sorted(
|
||||
answer
|
||||
for answer, count in answers.items()
|
||||
if count == max_count
|
||||
)
|
||||
gold = (
|
||||
math_rows[0]["task_evaluation"]["gold_final"]
|
||||
if math_rows
|
||||
else None
|
||||
)
|
||||
return {
|
||||
"outputs": len(rows),
|
||||
"natural_eos": sum(row["hit_eos"] for row in rows),
|
||||
"budget_truncated": sum(
|
||||
row["stopped_at_max_new_tokens"] for row in rows
|
||||
),
|
||||
"mean_generated_tokens": (
|
||||
mean(row["generated_tokens"] for row in rows)
|
||||
if rows
|
||||
else None
|
||||
),
|
||||
"completion_classes": dict(
|
||||
Counter(row["completion_class"] for row in rows)
|
||||
),
|
||||
"math": {
|
||||
"outputs": len(math_rows),
|
||||
"evaluator_covered": sum(
|
||||
row["task_evaluation"]["evaluator_covered"]
|
||||
for row in math_rows
|
||||
),
|
||||
"fixed_budget_exact": sum(
|
||||
row["task_evaluation"][
|
||||
"fixed_budget_numeric_exact"
|
||||
]
|
||||
for row in math_rows
|
||||
),
|
||||
"strict_complete_exact": sum(
|
||||
row["task_evaluation"][
|
||||
"strict_complete_numeric_exact"
|
||||
]
|
||||
for row in math_rows
|
||||
),
|
||||
"answer_frequencies": dict(answers),
|
||||
"modal_answers": modes,
|
||||
"modal_count": max_count,
|
||||
"absolute_majority_exists": (
|
||||
max_count > len(math_rows) / 2
|
||||
if math_rows
|
||||
else None
|
||||
),
|
||||
"unique_absolute_majority": (
|
||||
modes[0]
|
||||
if (
|
||||
math_rows
|
||||
and len(modes) == 1
|
||||
and max_count > len(math_rows) / 2
|
||||
)
|
||||
else None
|
||||
),
|
||||
"gold": gold,
|
||||
"unique_absolute_majority_matches_gold": (
|
||||
len(modes) == 1
|
||||
and max_count > len(math_rows) / 2
|
||||
and modes[0] == gold
|
||||
if math_rows
|
||||
else None
|
||||
),
|
||||
},
|
||||
"code": {
|
||||
"outputs": len(code_rows),
|
||||
"ast_parse": sum(
|
||||
row["task_evaluation"]["python_ast_parse"]
|
||||
for row in code_rows
|
||||
),
|
||||
"executed": sum(
|
||||
row["task_evaluation"]["execution"]["status"]
|
||||
!= "not_run"
|
||||
for row in code_rows
|
||||
),
|
||||
"execution_cache_hits": sum(
|
||||
row["task_evaluation"]["execution_cache_hit"]
|
||||
for row in code_rows
|
||||
),
|
||||
"tests_pass": sum(
|
||||
row["task_evaluation"][
|
||||
"fixed_budget_tests_pass"
|
||||
]
|
||||
for row in code_rows
|
||||
),
|
||||
"strict_complete_tests_pass": sum(
|
||||
row["task_evaluation"][
|
||||
"strict_complete_tests_pass"
|
||||
]
|
||||
for row in code_rows
|
||||
),
|
||||
"execution_statuses": dict(
|
||||
Counter(
|
||||
row["task_evaluation"]["execution"]["status"]
|
||||
for row in code_rows
|
||||
)
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def source_condition_summary(
|
||||
rows: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
result: dict[str, dict[str, Any]] = {}
|
||||
source_ids = sorted({row["source_id"] for row in rows})
|
||||
conditions = tuple(
|
||||
condition
|
||||
for _, left, right in COMPARISONS
|
||||
for condition in (left, right)
|
||||
)
|
||||
ordered_conditions = tuple(dict.fromkeys(conditions))
|
||||
for source_id in source_ids:
|
||||
result[source_id] = {}
|
||||
for condition in ordered_conditions:
|
||||
subset = [
|
||||
row
|
||||
for row in rows
|
||||
if row["source_id"] == source_id
|
||||
and row["condition"] == condition
|
||||
]
|
||||
result[source_id][condition] = summarize_group(subset)
|
||||
return result
|
||||
|
||||
|
||||
def edge_summary(rows: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
result: dict[str, dict[str, Any]] = {}
|
||||
source_ids = sorted({row["source_id"] for row in rows})
|
||||
for source_id in source_ids:
|
||||
result[source_id] = {}
|
||||
source_rows = [
|
||||
row for row in rows if row["source_id"] == source_id
|
||||
]
|
||||
for name, left, right in COMPARISONS:
|
||||
left_rows = [
|
||||
row
|
||||
for row in source_rows
|
||||
if row["condition"] == left
|
||||
]
|
||||
right_rows = [
|
||||
row
|
||||
for row in source_rows
|
||||
if row["condition"] == right
|
||||
]
|
||||
left_success = [
|
||||
output_success(row) for row in left_rows
|
||||
]
|
||||
right_success = [
|
||||
output_success(row) for row in right_rows
|
||||
]
|
||||
result[source_id][name] = {
|
||||
"left": left,
|
||||
"right": right,
|
||||
"samples_per_side": len(left_rows),
|
||||
"completion_classes": {
|
||||
"left": dict(
|
||||
Counter(
|
||||
row["completion_class"]
|
||||
for row in left_rows
|
||||
)
|
||||
),
|
||||
"right": dict(
|
||||
Counter(
|
||||
row["completion_class"]
|
||||
for row in right_rows
|
||||
)
|
||||
),
|
||||
},
|
||||
"natural_eos_difference_right_minus_left": (
|
||||
sum(row["hit_eos"] for row in right_rows)
|
||||
- sum(row["hit_eos"] for row in left_rows)
|
||||
),
|
||||
"fixed_budget_success_difference_right_minus_left": (
|
||||
sum(
|
||||
row["fixed_budget"]
|
||||
for row in right_success
|
||||
if row["fixed_budget"] is not None
|
||||
)
|
||||
- sum(
|
||||
row["fixed_budget"]
|
||||
for row in left_success
|
||||
if row["fixed_budget"] is not None
|
||||
)
|
||||
if any(
|
||||
row["fixed_budget"] is not None
|
||||
for row in [*left_success, *right_success]
|
||||
)
|
||||
else None
|
||||
),
|
||||
"strict_success_difference_right_minus_left": (
|
||||
sum(
|
||||
row["strict_complete"]
|
||||
for row in right_success
|
||||
if row["strict_complete"] is not None
|
||||
)
|
||||
- sum(
|
||||
row["strict_complete"]
|
||||
for row in left_success
|
||||
if row["strict_complete"] is not None
|
||||
)
|
||||
if any(
|
||||
row["strict_complete"] is not None
|
||||
for row in [*left_success, *right_success]
|
||||
)
|
||||
else None
|
||||
),
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
for path in (
|
||||
args.sampling_json,
|
||||
args.human_eval,
|
||||
args.gsm8k,
|
||||
):
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
if "@sha256:" not in args.sandbox_image:
|
||||
raise ValueError("--sandbox-image must include an immutable digest")
|
||||
if args.timeout_seconds <= 0:
|
||||
raise ValueError("--timeout-seconds must be positive")
|
||||
|
||||
sampling = json.loads(
|
||||
args.sampling_json.read_text(encoding="utf-8")
|
||||
)
|
||||
human_eval, gsm8k = completion.load_tasks(
|
||||
args.human_eval,
|
||||
args.gsm8k,
|
||||
)
|
||||
code_cache: dict[
|
||||
tuple[str, str, str],
|
||||
dict[str, Any],
|
||||
] = {}
|
||||
rows = []
|
||||
for source in sampling["sources"]:
|
||||
for run in source["runs"]:
|
||||
for output in run["outputs"]:
|
||||
evaluation = None
|
||||
cache_hit = False
|
||||
if source["domain"] == "math":
|
||||
evaluation = completion.evaluate_math(
|
||||
output["text"],
|
||||
output["hit_eos"],
|
||||
gsm8k[source["id"]]["answer"],
|
||||
)
|
||||
elif source["domain"] == "code":
|
||||
evaluation, cache_hit = evaluate_code_cached(
|
||||
text=output["text"],
|
||||
hit_eos=output["hit_eos"],
|
||||
task=human_eval[source["id"]],
|
||||
image=args.sandbox_image,
|
||||
timeout_seconds=args.timeout_seconds,
|
||||
skip_execution=args.skip_code_execution,
|
||||
cache=code_cache,
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"source_id": source["id"],
|
||||
"domain": source["domain"],
|
||||
"replicate_index": run["replicate_index"],
|
||||
"replicate_label": run["replicate_label"],
|
||||
"base_seed": run["base_seed"],
|
||||
"run_seed": run["run_seed"],
|
||||
"condition": output["condition"],
|
||||
"generated_tokens": output[
|
||||
"generated_tokens"
|
||||
],
|
||||
"hit_eos": output["hit_eos"],
|
||||
"stopped_at_max_new_tokens": output[
|
||||
"stopped_at_max_new_tokens"
|
||||
],
|
||||
"prompt_token_ids_sha256": output[
|
||||
"prompt_token_ids_sha256"
|
||||
],
|
||||
"generated_token_ids_sha256": output[
|
||||
"generated_token_ids_sha256"
|
||||
],
|
||||
"text_sha256": output["text_sha256"],
|
||||
"task_evaluation": evaluation,
|
||||
"completion_class": (
|
||||
completion.completion_class(
|
||||
output,
|
||||
evaluation,
|
||||
)
|
||||
),
|
||||
"code_execution_cache_hit": cache_hit,
|
||||
}
|
||||
)
|
||||
|
||||
by_condition = {
|
||||
condition: summarize_group(
|
||||
[
|
||||
row
|
||||
for row in rows
|
||||
if row["condition"] == condition
|
||||
]
|
||||
)
|
||||
for condition in sampling["seed_contract"][
|
||||
"condition_row_order"
|
||||
]
|
||||
}
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": sampling["protocol_id"],
|
||||
"input": {
|
||||
"sampling_path": str(args.sampling_json),
|
||||
"sampling_sha256": completion.sha256_file(
|
||||
args.sampling_json
|
||||
),
|
||||
"sampling_content_hash": sampling["content_hash"],
|
||||
"human_eval_sha256": completion.sha256_file(
|
||||
args.human_eval
|
||||
),
|
||||
"gsm8k_sha256": completion.sha256_file(args.gsm8k),
|
||||
"model_revision": sampling["model"]["revision"],
|
||||
"base_seeds": sampling["seed_contract"][
|
||||
"executed_base_seeds"
|
||||
],
|
||||
"max_new_tokens": sampling["generation_contract"][
|
||||
"max_new_tokens"
|
||||
],
|
||||
},
|
||||
"sandbox": {
|
||||
"image": args.sandbox_image,
|
||||
"timeout_seconds": args.timeout_seconds,
|
||||
"code_execution_skipped": args.skip_code_execution,
|
||||
"network": "none",
|
||||
"filesystem": "read-only",
|
||||
"user": "65534:65534",
|
||||
"capabilities": "ALL dropped",
|
||||
"no_new_privileges": True,
|
||||
"memory": "256m",
|
||||
"memory_swap": "256m",
|
||||
"pids_limit": 64,
|
||||
"cpus": 0.5,
|
||||
"tmpfs": "/tmp:rw,noexec,nosuid,size=16m",
|
||||
"host_mounts": 0,
|
||||
"cache_key": (
|
||||
"candidate SHA-256 + task-test SHA-256 + "
|
||||
"sandbox-harness SHA-256"
|
||||
),
|
||||
"unique_code_cache_entries": len(code_cache),
|
||||
},
|
||||
"rows": rows,
|
||||
"summary": {
|
||||
**summarize_group(rows),
|
||||
"by_condition": by_condition,
|
||||
"by_source_condition": source_condition_summary(rows),
|
||||
"by_source_edge": edge_summary(rows),
|
||||
},
|
||||
"claim_boundary": [
|
||||
"Four sources and eight seeds are not benchmark estimates.",
|
||||
"Completion-conditioned metrics are selection-biased diagnostics.",
|
||||
"A passing HumanEval test is functional evidence, not code-safety evidence.",
|
||||
"A modal sampled math answer is not standard self-consistency.",
|
||||
"Counterfactual token sequences are not official-valid chats.",
|
||||
"Repeated samples from one source are not independent tasks.",
|
||||
],
|
||||
}
|
||||
result["content_hash"] = completion.canonical_hash(rows)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
payload = args.output.read_bytes()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"sha256": completion.sha256_bytes(payload),
|
||||
"bytes": len(payload),
|
||||
"outputs": len(rows),
|
||||
"summary": {
|
||||
key: result["summary"][key]
|
||||
for key in (
|
||||
"natural_eos",
|
||||
"budget_truncated",
|
||||
"completion_classes",
|
||||
"math",
|
||||
"code",
|
||||
)
|
||||
},
|
||||
"unique_code_cache_entries": len(code_cache),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,199 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare a fresh sampling rerun with matching formal-grid cells."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--formal-json", type=Path, required=True)
|
||||
parser.add_argument("--rerun-json", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while chunk := handle.read(16 * 1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_hash(value: Any) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def run_index(
|
||||
payload: dict[str, Any],
|
||||
) -> dict[tuple[str, int, str], tuple[dict[str, Any], dict[str, Any]]]:
|
||||
return {
|
||||
(
|
||||
source["id"],
|
||||
run["base_seed"],
|
||||
output["condition"],
|
||||
): (run, output)
|
||||
for source in payload["sources"]
|
||||
for run in source["runs"]
|
||||
for output in run["outputs"]
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
for path in (args.formal_json, args.rerun_json):
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
formal = json.loads(
|
||||
args.formal_json.read_text(encoding="utf-8")
|
||||
)
|
||||
rerun = json.loads(
|
||||
args.rerun_json.read_text(encoding="utf-8")
|
||||
)
|
||||
if formal["protocol_id"] != rerun["protocol_id"]:
|
||||
raise RuntimeError("protocol IDs differ")
|
||||
if formal["model"]["revision"] != rerun["model"]["revision"]:
|
||||
raise RuntimeError("model revisions differ")
|
||||
if (
|
||||
formal["generation_contract"]
|
||||
!= rerun["generation_contract"]
|
||||
):
|
||||
raise RuntimeError("generation contracts differ")
|
||||
formal_rows = run_index(formal)
|
||||
rerun_rows = run_index(rerun)
|
||||
missing = sorted(set(rerun_rows) - set(formal_rows))
|
||||
if missing:
|
||||
raise RuntimeError(
|
||||
f"{len(missing)} rerun cells are absent from formal grid"
|
||||
)
|
||||
|
||||
rows = []
|
||||
for key in sorted(rerun_rows):
|
||||
formal_run, formal_output = formal_rows[key]
|
||||
rerun_run, rerun_output = rerun_rows[key]
|
||||
checks = {
|
||||
"run_seed_exact": (
|
||||
formal_run["run_seed"] == rerun_run["run_seed"]
|
||||
),
|
||||
"prompt_hash_exact": (
|
||||
formal_output["prompt_token_ids_sha256"]
|
||||
== rerun_output["prompt_token_ids_sha256"]
|
||||
),
|
||||
"generated_token_ids_exact": (
|
||||
formal_output["generated_token_ids"]
|
||||
== rerun_output["generated_token_ids"]
|
||||
),
|
||||
"decoded_text_exact": (
|
||||
formal_output["text"] == rerun_output["text"]
|
||||
),
|
||||
"eos_state_exact": (
|
||||
formal_output["hit_eos"]
|
||||
== rerun_output["hit_eos"]
|
||||
),
|
||||
"truncation_state_exact": (
|
||||
formal_output["stopped_at_max_new_tokens"]
|
||||
== rerun_output["stopped_at_max_new_tokens"]
|
||||
),
|
||||
"cpu_rng_pre_state_exact": (
|
||||
formal_run["rng_state_before"]["cpu_sha256"]
|
||||
== rerun_run["rng_state_before"]["cpu_sha256"]
|
||||
),
|
||||
"cuda_rng_pre_state_exact": (
|
||||
formal_run["rng_state_before"][
|
||||
"cuda_combined_sha256"
|
||||
]
|
||||
== rerun_run["rng_state_before"][
|
||||
"cuda_combined_sha256"
|
||||
]
|
||||
),
|
||||
}
|
||||
rows.append(
|
||||
{
|
||||
"source_id": key[0],
|
||||
"base_seed": key[1],
|
||||
"condition": key[2],
|
||||
**checks,
|
||||
"all_preregistered_fields_exact": all(
|
||||
checks.values()
|
||||
),
|
||||
}
|
||||
)
|
||||
check_names = [
|
||||
key
|
||||
for key in rows[0]
|
||||
if key.endswith("_exact")
|
||||
and key != "all_preregistered_fields_exact"
|
||||
]
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": formal["protocol_id"],
|
||||
"formal": {
|
||||
"path": str(args.formal_json),
|
||||
"sha256": sha256_file(args.formal_json),
|
||||
"content_hash": formal["content_hash"],
|
||||
"base_seeds": formal["seed_contract"][
|
||||
"executed_base_seeds"
|
||||
],
|
||||
},
|
||||
"rerun": {
|
||||
"path": str(args.rerun_json),
|
||||
"sha256": sha256_file(args.rerun_json),
|
||||
"content_hash": rerun["content_hash"],
|
||||
"base_seeds": rerun["seed_contract"][
|
||||
"executed_base_seeds"
|
||||
],
|
||||
},
|
||||
"rows": rows,
|
||||
"summary": {
|
||||
"cells": len(rows),
|
||||
"all_preregistered_fields_exact": sum(
|
||||
row["all_preregistered_fields_exact"]
|
||||
for row in rows
|
||||
),
|
||||
"by_field": {
|
||||
name: sum(row[name] for row in rows)
|
||||
for name in check_names
|
||||
},
|
||||
},
|
||||
"claim_boundary": [
|
||||
"Only the rerun seed subset is independently reproduced.",
|
||||
"Exact replay is scoped to the pinned software and hardware contract.",
|
||||
"Reproduction does not imply trajectories are seed-invariant.",
|
||||
],
|
||||
}
|
||||
result["content_hash"] = canonical_hash(rows)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
payload = args.output.read_bytes()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||
"bytes": len(payload),
|
||||
"summary": result["summary"],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -15,6 +15,7 @@
|
||||
"build:data:deepseek-role-block": "node scripts/build-deepseek-role-marker-block-compact.mjs",
|
||||
"build:data:deepseek-chat-behavior": "node scripts/build-deepseek-chat-behavior-compact.mjs",
|
||||
"build:data:deepseek-chat-completion-depth": "node scripts/build-deepseek-chat-completion-depth-compact.mjs",
|
||||
"build:data:deepseek-chat-sampling": "node scripts/build-deepseek-chat-sampling-compact.mjs",
|
||||
"check:site": "node scripts/check-site.mjs",
|
||||
"check:moe-browser": "node scripts/check-moe-browser.mjs",
|
||||
"check:reasoning-browser": "node scripts/check-reasoning-browser.mjs",
|
||||
@@ -31,6 +32,7 @@
|
||||
"check:evaluation-browser": "node scripts/check-evaluation-browser.mjs",
|
||||
"check:representation-browser": "node scripts/check-representation-browser.mjs",
|
||||
"check:deepseek-browser": "node scripts/check-deepseek-browser.mjs",
|
||||
"check:deepseek-sampling-browser": "node scripts/check-deepseek-sampling-browser.mjs",
|
||||
"check:k3-browser": "node scripts/check-k3-browser.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
|
||||
@@ -0,0 +1,450 @@
|
||||
# DeepSeek-V2-Lite-Chat 多种子采样稳健性审计
|
||||
|
||||
> 执行日期：2026-07-30
|
||||
>
|
||||
> 协议：`llm-atlas-deepseek-chat-sampling-v1`
|
||||
>
|
||||
> 模型：`deepseek-ai/DeepSeek-V2-Lite-Chat`
|
||||
>
|
||||
> revision：`85864749cd611b4353ce1decdb286193298f64c7`
|
||||
>
|
||||
> 预注册：`research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md`
|
||||
|
||||
## 0. 一句话结论
|
||||
|
||||
在固定官方 Chat checkpoint、固定四条 source、固定八格 prompt batch 与官方
|
||||
`temperature=0.3 / top_p=0.95` 下：
|
||||
|
||||
```text
|
||||
256 sampled outputs
|
||||
├─ 251 natural EOS
|
||||
├─ 5 budget truncated（全部来自 English / system off）
|
||||
├─ 242 个不同的完整 token trajectory hash
|
||||
├─ Math：62 / 64 strict exact
|
||||
├─ Code：63 / 64 official tests pass
|
||||
└─ 新进程 R0/R1：64 / 64 全合同字段 exact
|
||||
```
|
||||
|
||||
这说明在这份执行合同下：
|
||||
|
||||
1. sampling 确实产生了跨 seed 轨迹分叉；
|
||||
2. 同 seed 可以在新进程精确复现；
|
||||
3. 一条流畅、自然结束、可解析或可执行的回答仍然可能答错；
|
||||
4. 四条 source 与八个 seed 仍远不足以估计 benchmark 能力或完整生成分布。
|
||||
|
||||
---
|
||||
|
||||
## 1. 为什么在 greedy 之后还要做这一轮
|
||||
|
||||
Round 05 的 512-token 实验固定 `do_sample=false`。它验证的是每一步都取最高概率 token
|
||||
时的单条轨迹：
|
||||
|
||||
```text
|
||||
同一 source + 同一 checkpoint + 不同输入边界
|
||||
→ greedy token 轨迹会怎样分叉
|
||||
```
|
||||
|
||||
greedy 不能回答：
|
||||
|
||||
- 换 seed 后，同一条件是否只会复述同一条轨迹；
|
||||
- greedy 轨迹是否会出现在有限的 nucleus samples 里；
|
||||
- 两个条件的八样本集合是否有 exact 重合；
|
||||
- 完成与任务结果是否会随 seed 改变。
|
||||
|
||||
本轮只补这个证据缺口，不回头把 4 条 source 写成能力评测。
|
||||
|
||||
---
|
||||
|
||||
## 2. 冻结合同
|
||||
|
||||
### 2.1 四条 source
|
||||
|
||||
| domain | source ID | 角色 |
|
||||
|---|---|---|
|
||||
| English | `wikitext2/raw-validation/0443` | 长 continuation，最容易触及 512 上限 |
|
||||
| Chinese | `tnews/test/4855` | 中文分类式输入 |
|
||||
| Code | `HumanEval/31` | `is_prime`，可用官方 tests 执行 |
|
||||
| Math | `gsm8k/test/1069` | 可抽取数值 gold |
|
||||
|
||||
它们是前序固定 source 排序中每域第 1 条，不是看到结果后重选。
|
||||
|
||||
### 2.2 八格输入
|
||||
|
||||
```text
|
||||
system off/on × EOS / BOS / x / period
|
||||
```
|
||||
|
||||
固定行顺序：
|
||||
|
||||
```text
|
||||
s0_eos, s1_eos, s0_bos, s1_bos,
|
||||
s0_x, s1_x, s0_period, s1_period
|
||||
```
|
||||
|
||||
32 / 32 个 prompt token hash 与 Round 05 greedy 运行 exact。只有 EOS 是官方聊天序列；
|
||||
BOS、`x` 与句点都是单 ID 反事实，不能称为官方有效 chat。
|
||||
|
||||
### 2.3 sampling 参数
|
||||
|
||||
checkpoint 自带的 `generation_config.json` 固定：
|
||||
|
||||
```text
|
||||
do_sample = true
|
||||
temperature = 0.3
|
||||
top_p = 0.95
|
||||
```
|
||||
|
||||
本轮另外显式传入：
|
||||
|
||||
```text
|
||||
top_k = 0
|
||||
max_new_tokens = 512
|
||||
use_cache = true
|
||||
```
|
||||
|
||||
`top_k=0` 很重要：否则 Transformers 的通用默认 top-k 可能在不显眼处继续截断候选集合。
|
||||
|
||||
### 2.4 八个 seed
|
||||
|
||||
八个 base seed 全部由协议字符串做 SHA-256 派生：
|
||||
|
||||
| replicate | base seed |
|
||||
|---:|---:|
|
||||
| R0 | 19,683,830 |
|
||||
| R1 | 1,560,062,173 |
|
||||
| R2 | 3,978,401,375 |
|
||||
| R3 | 1,280,933,274 |
|
||||
| R4 | 1,467,459,869 |
|
||||
| R5 | 1,297,359,489 |
|
||||
| R6 | 2,722,953,988 |
|
||||
| R7 | 3,330,978,061 |
|
||||
|
||||
每个 source 再由 base seed 与 source ID 派生独立 run seed。每次 batch 前执行
|
||||
`torch.manual_seed()` 与 `torch.cuda.manual_seed_all()`，并记录 CPU/CUDA RNG 的前后
|
||||
state hash。
|
||||
|
||||
### 2.5 batch-seed aligned，不是 common random numbers
|
||||
|
||||
Transformers `4.41.2` 对 batch 调一次 `torch.multinomial`。八行共享 run seed 与调用
|
||||
时序，但不同 row 消费不同 RNG 子流。因此：
|
||||
|
||||
- 可以比较相同 replicate label 的两格；
|
||||
- 只能称为 batch-seed aligned；
|
||||
- 不能当作逐行共享同一随机数的 paired causal design；
|
||||
- 本轮不报告 paired p-value 或置信区间。
|
||||
|
||||
---
|
||||
|
||||
## 3. 执行闸门
|
||||
|
||||
### 3.1 16-token smoke
|
||||
|
||||
```text
|
||||
4 sources × R0/R1 × 8 conditions = 64 short outputs
|
||||
4 sources × R0 replay × 8 conditions = 32 replay outputs
|
||||
```
|
||||
|
||||
结果：
|
||||
|
||||
| gate | 结果 |
|
||||
|---|---:|
|
||||
| prompt hash exact | 32 / 32 |
|
||||
| R0 同进程重放全合同 exact | 32 / 32 |
|
||||
| R0 vs R1 可比格 | 32 |
|
||||
| R0 vs R1 分叉 | 18 |
|
||||
| OOM / NaN / exception | 0 |
|
||||
|
||||
16-token smoke 的 64 条输出全部触顶，符合技术检查的预期；它们没有进入正式统计。
|
||||
|
||||
### 3.2 正式网格
|
||||
|
||||
```text
|
||||
4 sources × 8 seeds × 8 conditions = 256 outputs
|
||||
```
|
||||
|
||||
结构审计：
|
||||
|
||||
| 检查 | 结果 |
|
||||
|---|---:|
|
||||
| sources | 4 |
|
||||
| runs / source | 8 |
|
||||
| outputs / source | 64 |
|
||||
| outputs / condition | 32 |
|
||||
| total outputs | 256 |
|
||||
| prompt hash exact | 32 / 32 |
|
||||
| missing / duplicate grid cells | 0 |
|
||||
|
||||
生成耗时 `2,251.225s`，CUDA peak allocated `30,868,238,336 bytes`。这是 CPU-offloaded
|
||||
eager 机制审计延迟，不是服务吞吐。
|
||||
|
||||
---
|
||||
|
||||
## 4. stopping 与轨迹多样性
|
||||
|
||||
### 4.1 总账
|
||||
|
||||
| 指标 | 结果 |
|
||||
|---|---:|
|
||||
| natural EOS | 251 / 256 |
|
||||
| budget truncated | 5 / 256 |
|
||||
| 全局 unique trajectory hashes | 242 / 256 |
|
||||
| R0/R1 同格 trajectory 不同 | 31 / 32 |
|
||||
| greedy trajectory 出现在八样本集合 | 9 / 32 source×condition sets |
|
||||
|
||||
`242/256 unique` 只说明完整 token ID 序列的 hash 不同，不说明有 242 种语义或思路。
|
||||
|
||||
### 4.2 每个 condition
|
||||
|
||||
| condition | EOS / 32 | mean tokens | 四个 source-set 的 unique 总数 / 32 | 8/8 unique sets / 4 | greedy included / 4 |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| `s0_eos` | 29 | 290.1 | 31 | 3 | 1 |
|
||||
| `s1_eos` | 32 | 241.4 | 32 | 4 | 2 |
|
||||
| `s0_bos` | 32 | 265.4 | 30 | 3 | 1 |
|
||||
| `s1_bos` | 32 | 221.2 | 32 | 4 | 1 |
|
||||
| `s0_x` | 30 | 263.6 | 32 | 4 | 0 |
|
||||
| `s1_x` | 32 | 146.8 | 30 | 3 | 2 |
|
||||
| `s0_period` | 32 | 259.5 | 32 | 4 | 0 |
|
||||
| `s1_period` | 32 | 121.7 | 26 | 3 | 2 |
|
||||
|
||||
32 个 source×condition set 中：
|
||||
|
||||
```text
|
||||
28 sets：8 unique trajectories
|
||||
2 sets：6 unique trajectories
|
||||
1 set ：7 unique trajectories
|
||||
1 set ：2 unique trajectories
|
||||
```
|
||||
|
||||
只有 `HumanEval/31 × s1_period` 收缩到 2 条完整轨迹；它不是“低创造力”的证明，只是该
|
||||
source/condition/参数下八次抽样的 exact-sequence 重复。
|
||||
|
||||
### 4.3 五个截断
|
||||
|
||||
五格全部属于 `wikitext2/raw-validation/0443`：
|
||||
|
||||
| replicate | condition | tokens |
|
||||
|---|---|---:|
|
||||
| R0 | `s0_x` | 512 |
|
||||
| R1 | `s0_eos` | 512 |
|
||||
| R4 | `s0_eos` | 512 |
|
||||
| R5 | `s0_x` | 512 |
|
||||
| R6 | `s0_eos` | 512 |
|
||||
|
||||
所以“全局 251/256 EOS”不能简化成每条 source 都同样容易完成。
|
||||
|
||||
---
|
||||
|
||||
## 5. 十条 edge 的集合比较
|
||||
|
||||
每个 source×edge 同时计算：
|
||||
|
||||
1. 8 个相同 replicate label 对的 token similarity；
|
||||
2. 左集合每条到右集合的最高 similarity；
|
||||
3. 右集合每条到左集合的最高 similarity；
|
||||
4. 两个方向 16 个最近邻值的平均；
|
||||
5. 完整 trajectory hash-set 的 intersection / union。
|
||||
|
||||
四条 source 的描述性均值：
|
||||
|
||||
| edge | aligned similarity | symmetric nearest similarity | exact intersection / union |
|
||||
|---|---:|---:|---:|
|
||||
| system · EOS | .366 | .492 | 1 / 62 |
|
||||
| system · BOS | .314 | .491 | 0 / 62 |
|
||||
| system · x | .275 | .392 | 0 / 62 |
|
||||
| system · period | .253 | .355 | 0 / 58 |
|
||||
| BOS − EOS · S0 | .444 | .589 | 0 / 61 |
|
||||
| BOS − EOS · S1 | .430 | .589 | 0 / 64 |
|
||||
| x − EOS · S0 | .332 | .470 | 0 / 63 |
|
||||
| x − EOS · S1 | .278 | .395 | 0 / 62 |
|
||||
| period − EOS · S0 | .404 | .546 | 1 / 62 |
|
||||
| period − EOS · S1 | .261 | .375 | 0 / 58 |
|
||||
|
||||
40 个 source-level edge set 中只有两个出现跨侧 exact trajectory 重合：
|
||||
|
||||
- `gsm8k/test/1069 × system_eos`；
|
||||
- `gsm8k/test/1069 × period_at_s0`。
|
||||
|
||||
nearest-neighbor 总是高于相同 label 对并不奇怪：它从 8 条右侧样本里主动选择最接近的一条。
|
||||
这不是无偏分布距离，也不能据此给 boundary effect 排名。
|
||||
|
||||
---
|
||||
|
||||
## 6. 任务账：完成、可评与正确分开
|
||||
|
||||
### 6.1 GSM8K
|
||||
|
||||
| 指标 | 结果 |
|
||||
|---|---:|
|
||||
| sampled outputs | 64 |
|
||||
| numeric evaluator covered | 64 |
|
||||
| strict-complete exact | 62 |
|
||||
| answer `300` | 62 |
|
||||
| answer `100` | 2 |
|
||||
| unique absolute majority | `300` |
|
||||
| gold | `300` |
|
||||
|
||||
两条失败分别位于：
|
||||
|
||||
- R4 / `s1_x`；
|
||||
- R7 / `s1_eos`。
|
||||
|
||||
两条都不是抽取器误判。回答明确写出 boxed `100`，错误推理是：
|
||||
|
||||
```text
|
||||
1/4 trucks failed
|
||||
→ 错写成只有 1/4 trucks delivered
|
||||
→ 5 trucks × 20 tons = 100
|
||||
```
|
||||
|
||||
正确逻辑应保留 `3/4 × 20 = 15` 辆车，得到 `300`。这说明 natural EOS、明确 final
|
||||
marker 与可抽取数值都不能保证 reasoning 正确。
|
||||
|
||||
这里的多数答案只是 1 条 GSM8K source 上的八条件×八 seed 描述，不是标准
|
||||
self-consistency benchmark。
|
||||
|
||||
### 6.2 HumanEval
|
||||
|
||||
| 指标 | 结果 |
|
||||
|---|---:|
|
||||
| sampled outputs | 64 |
|
||||
| Python AST parse | 64 |
|
||||
| sandbox evaluated | 64 |
|
||||
| official tests pass | 63 |
|
||||
| assertion failure | 1 |
|
||||
| unique candidate execution keys | 24 |
|
||||
| exact duplicate cache hits | 40 |
|
||||
|
||||
唯一失败位于 R5 / `s0_period`。候选代码：
|
||||
|
||||
- 正确排除了 `n < 2`；
|
||||
- 正确特判 `n == 2`；
|
||||
- 只检查奇数除数；
|
||||
- **没有先排除大于 2 的偶数**。
|
||||
|
||||
因此代码流畅、code fence 闭合、AST 合法、可以执行，却对偶数输入返回错误。这个案例直接
|
||||
展示了四张账：
|
||||
|
||||
```text
|
||||
NATURAL EOS
|
||||
→ TASK TERMINAL
|
||||
→ EVALUATOR COVERED
|
||||
→ ASSERTION FAILED
|
||||
```
|
||||
|
||||
每个唯一 candidate 的执行环境：
|
||||
|
||||
```text
|
||||
python:3.11-alpine
|
||||
@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4
|
||||
|
||||
network none · read-only filesystem · no host mounts
|
||||
user 65534:65534 · cap-drop ALL · no-new-privileges
|
||||
256MiB memory/swap · pids 64 · cpus .5 · timeout 5s
|
||||
```
|
||||
|
||||
cache 只复用完全相同的 candidate hash + task-test hash + harness hash；每格评测行仍保留。
|
||||
|
||||
---
|
||||
|
||||
## 7. 新进程复现
|
||||
|
||||
正式运行结束后重新启动 Python、重新加载 checkpoint，只执行 R0/R1：
|
||||
|
||||
```text
|
||||
4 sources × 2 seeds × 8 conditions = 64 outputs
|
||||
```
|
||||
|
||||
逐格结果：
|
||||
|
||||
| field | exact |
|
||||
|---|---:|
|
||||
| run seed | 64 / 64 |
|
||||
| prompt hash | 64 / 64 |
|
||||
| complete generated token IDs | 64 / 64 |
|
||||
| decoded text | 64 / 64 |
|
||||
| EOS state | 64 / 64 |
|
||||
| truncation state | 64 / 64 |
|
||||
| CPU RNG pre-state hash | 64 / 64 |
|
||||
| CUDA RNG pre-state hash | 64 / 64 |
|
||||
| all preregistered fields | **64 / 64** |
|
||||
|
||||
这只能写成“R0/R1 的 64 条轨迹在固定环境中独立复现”。R2–R7 没有新进程复跑，不能写
|
||||
256 / 256。
|
||||
|
||||
---
|
||||
|
||||
## 8. artifact hash 链
|
||||
|
||||
| artifact | bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| formal raw | 1,852,892 | `46c7edfce1409e798d6b0f06e905dd3a9d4ab6911acd8e42e2ec9d06d65345af` |
|
||||
| independent eval | 316,800 | `078f486e021ffd0e4af8ad942d001a9f2429933b2fb0fb3523b7d1da5b24c4d8` |
|
||||
| R0/R1 rerun raw | 537,669 | `72d050eace531bb38fb2592e39dc7f1b9c9003306c92c568dcd4acaaa691f6a0` |
|
||||
| reproduction comparison | 29,938 | `4d59a775943459604ea2d9976bb3be01ff7b3b08d71eb0ca22c376122ef82d15` |
|
||||
| website compact | 371,495 | `c014dc0ed6c27b19a61e7abb9b84a077b414e8472d40cc69d025693c25e1372e` |
|
||||
|
||||
compact builder 在写前端 JSON 前强制验证：
|
||||
|
||||
1. evaluator 声明的 sampling input SHA 与 formal raw 文件一致；
|
||||
2. reproduction 声明的 formal/rerun SHA 与两份 raw 一致；
|
||||
3. reproduction `all_preregistered_fields_exact == cells`。
|
||||
|
||||
任一失败都会中止构建。
|
||||
|
||||
---
|
||||
|
||||
## 9. 依赖与放置
|
||||
|
||||
| object | value |
|
||||
|---|---|
|
||||
| PyTorch | `2.11.0+cu128` |
|
||||
| Transformers | `4.41.2` |
|
||||
| dtype | BF16 |
|
||||
| local GPU | RTX 5090 · 32,607 MiB |
|
||||
| CUDA placement | embedding + layers 0–23 |
|
||||
| CPU offload | layers 24–26 + final norm + LM head |
|
||||
| max memory | GPU 28 GiB / CPU 80 GiB |
|
||||
| allocator | `expandable_segments:True` |
|
||||
| formal generation time | 2,251.225 s |
|
||||
| peak CUDA allocated | 30,868,238,336 bytes |
|
||||
|
||||
官方模型卡写出的 40GB 单 GPU BF16 边界高于本机容量，所以这不是 single-GPU BF16
|
||||
execution。
|
||||
|
||||
---
|
||||
|
||||
## 10. 一手来源
|
||||
|
||||
- [DeepSeek-V2-Lite-Chat model card](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat)
|
||||
- [Pinned official generation_config.json](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat/blob/85864749cd611b4353ce1decdb286193298f64c7/generation_config.json)
|
||||
- [Transformers 4.41.2 generation utils](https://github.com/huggingface/transformers/blob/v4.41.2/src/transformers/generation/utils.py)
|
||||
- [The Curious Case of Neural Text Degeneration / nucleus sampling](https://arxiv.org/abs/1904.09751)
|
||||
- [Self-Consistency Improves Chain of Thought Reasoning](https://arxiv.org/abs/2203.11171)
|
||||
- [HumanEval repository](https://github.com/openai/human-eval)
|
||||
- [GSM8K paper](https://arxiv.org/abs/2110.14168)
|
||||
|
||||
---
|
||||
|
||||
## 11. 可以说与不能说
|
||||
|
||||
可以说：
|
||||
|
||||
- 在固定四条 source 与官方 sampling 参数下，R0/R1 有 31/32 同格轨迹分叉；
|
||||
- 256 条样本中 251 条自然 EOS，242 条完整 token trajectory hash 不同；
|
||||
- 这 1 条 GSM8K 的 64 个样本中 62 个 strict exact；
|
||||
- 这 1 条 HumanEval 的 64 个样本中 63 个通过官方 tests；
|
||||
- 固定环境下 R0/R1 新进程复跑 64/64 全合同 exact。
|
||||
|
||||
不能说：
|
||||
|
||||
- 八个 seed 已恢复完整生成分布；
|
||||
- 242 个 hash 表示 242 种语义；
|
||||
- 4 条 source 是 benchmark；
|
||||
- 62/64 与 63/64 可以横比标准 accuracy 或 pass@k；
|
||||
- majority answer 是标准 self-consistency；
|
||||
- batch-seed aligned 是 common random numbers；
|
||||
- nearest-neighbor 是无偏分布距离；
|
||||
- 某个 condition 更“有创造力”；
|
||||
- BOS、`x`、句点是官方有效聊天格式；
|
||||
- output 差异由某层 hidden state 或 router 因果中介；
|
||||
- fixed-version exact replay 能跨 PyTorch、CUDA 或硬件保证。
|
||||
@@ -0,0 +1,352 @@
|
||||
# DeepSeek-V2-Lite-Chat 多种子采样稳健性协议
|
||||
|
||||
> 状态：已执行；正式结果、独立评测与新进程 R0/R1 复跑均通过
|
||||
>
|
||||
> 注册日期：2026-07-30
|
||||
>
|
||||
> 模型：`deepseek-ai/DeepSeek-V2-Lite-Chat`
|
||||
>
|
||||
> revision：`85864749cd611b4353ce1decdb286193298f64c7`
|
||||
>
|
||||
> 前序实验：512-token greedy completion 与全 27 层 trace
|
||||
>
|
||||
> 结果审计：`research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md`
|
||||
|
||||
## 0. 这一轮只补哪一个证据缺口
|
||||
|
||||
Round 04–05 已经证明：
|
||||
|
||||
```text
|
||||
固定 greedy 解码下：
|
||||
同一 source 的 system / 历史边界变化会让输出 token 轨迹分叉；
|
||||
512-token 预算下有 121 / 128 格自然遇到 EOS；
|
||||
子集新进程复跑可以逐 token exact。
|
||||
```
|
||||
|
||||
但 greedy 只取每一步概率最大的一个 token。它不能回答：
|
||||
|
||||
```text
|
||||
换一组随机数以后，某个条件是否仍有相似的完成率与任务结果？
|
||||
greedy 输出是采样分布中的常见轨迹，还是一条很窄的路径？
|
||||
两个条件的样本集合彼此靠近，还是只有某一对 greedy 输出不同？
|
||||
```
|
||||
|
||||
本轮因此启用 checkpoint 随附的 sampling config，固定多个 seed。它不是新的能力
|
||||
benchmark，也不尝试用 8 个样本完整估计语言分布。
|
||||
|
||||
---
|
||||
|
||||
## 1. 冻结不变的对象
|
||||
|
||||
继承 Round 05：
|
||||
|
||||
- 官方 SFT Chat checkpoint、12 个文件及 SHA-256；
|
||||
- BF16、官方 eager Python 实现、Transformers `4.41.2`；
|
||||
- 28 GiB static placement 与 `expandable_segments:True`；
|
||||
- CUDA resident：embedding + layers 0–23；
|
||||
- CPU offload residency：layers 24–26 + final norm + `lm_head`；
|
||||
- 同一 source 的八格进入同一个左填充 batch；
|
||||
- `system off/on × EOS/BOS/x/period`；
|
||||
- EOS 是官方序列；BOS、`x`、句点各是单 ID 反事实；
|
||||
- `PAD=EOS`、padding mask 为 0、`use_cache=true`；
|
||||
- 公开数据、source ID、source text hash 与 prompt token hash。
|
||||
|
||||
本轮只取前序固定次序中每域第 1 条：
|
||||
|
||||
| 域 | source |
|
||||
|---|---|
|
||||
| English | `wikitext2/raw-validation/0443` |
|
||||
| Chinese | `tnews/test/4855` |
|
||||
| Code | `HumanEval/31` |
|
||||
| Math | `gsm8k/test/1069` |
|
||||
|
||||
这是 4 条 source 的机制显微镜，不是 4 个领域的总体估计。
|
||||
|
||||
---
|
||||
|
||||
## 2. 解码配置来自哪里
|
||||
|
||||
固定 revision 的官方 `generation_config.json` 写明：
|
||||
|
||||
```json
|
||||
{
|
||||
"do_sample": true,
|
||||
"temperature": 0.3,
|
||||
"top_p": 0.95,
|
||||
"bos_token_id": 100000,
|
||||
"eos_token_id": 100001,
|
||||
"transformers_version": "4.39.3"
|
||||
}
|
||||
```
|
||||
|
||||
本轮在已固定的 Transformers `4.41.2` 执行这组参数：
|
||||
|
||||
```text
|
||||
do_sample = true
|
||||
temperature = 0.3
|
||||
top_p = 0.95
|
||||
top_k = 0（显式关闭默认 top-k）
|
||||
max_new_tokens = 512
|
||||
use_cache = true
|
||||
```
|
||||
|
||||
Holtzman et al. 2019 的 nucleus sampling 是这里 `top_p` 的方法来源；它动态保留累计概率
|
||||
达到阈值的候选集合。Wang et al. 2022 的 self-consistency 说明多条推理样本可以聚合答案，
|
||||
但本轮只在 1 条 GSM8K source 上展示多数答案，绝不把它称为标准 self-consistency
|
||||
benchmark。
|
||||
|
||||
官方文件记录的是参数默认值，不保证不同 Transformers 版本、kernel、batch 顺序与 seed
|
||||
产生相同随机轨迹；因此依赖栈和 batch 合同必须进入结果。
|
||||
|
||||
---
|
||||
|
||||
## 3. seed 如何在结果之前冻结
|
||||
|
||||
8 个 base seeds 不是手选“看起来分散”的整数。它们由以下字符串逐个做 SHA-256，并把
|
||||
前 4 bytes 按 big-endian 解释为无符号整数：
|
||||
|
||||
```text
|
||||
llm-atlas-deepseek-chat-sampling-v1/seed/{index}
|
||||
```
|
||||
|
||||
固定结果：
|
||||
|
||||
| replicate | base seed |
|
||||
|---:|---:|
|
||||
| R0 | 19,683,830 |
|
||||
| R1 | 1,560,062,173 |
|
||||
| R2 | 3,978,401,375 |
|
||||
| R3 | 1,280,933,274 |
|
||||
| R4 | 1,467,459,869 |
|
||||
| R5 | 1,297,359,489 |
|
||||
| R6 | 2,722,953,988 |
|
||||
| R7 | 3,330,978,061 |
|
||||
|
||||
为避免不同 source 重复使用同一 CUDA random stream，实际 run seed 定义为：
|
||||
|
||||
```text
|
||||
SHA256(
|
||||
"llm-atlas-deepseek-chat-sampling-v1/run\0"
|
||||
+ decimal(base_seed)
|
||||
+ "\0"
|
||||
+ source_id
|
||||
)[0:8] interpreted as unsigned big-endian
|
||||
modulo (2^63 - 1)
|
||||
```
|
||||
|
||||
每次 source×replicate batch 前：
|
||||
|
||||
```python
|
||||
torch.manual_seed(run_seed)
|
||||
torch.cuda.manual_seed_all(run_seed)
|
||||
```
|
||||
|
||||
并记录 CPU / CUDA RNG state 的执行前后 SHA-256。
|
||||
|
||||
---
|
||||
|
||||
## 4. 很重要：八格共享“批次标签”，不共享同一个随机数
|
||||
|
||||
Transformers `4.41.2` 的 `_sample()` 对整个 batch 调用：
|
||||
|
||||
```python
|
||||
torch.multinomial(probs, num_samples=1)
|
||||
```
|
||||
|
||||
它不接受逐行 `Generator`。所以同一个 source×replicate 的八格：
|
||||
|
||||
- 在同一时间步、同一 batch 中执行；
|
||||
- 共享同一个 run seed 与固定行顺序；
|
||||
- 但每一行消费不同的 RNG 子流；
|
||||
- 不是 common-random-number paired experiment。
|
||||
|
||||
因此“R3 的左格 vs R3 的右格”只能称为 **batch-seed aligned**，不能用普通 paired
|
||||
bootstrap 或 paired t-test 冒充同随机数因果对照。本轮不报告 edge 的显著性 p 值或
|
||||
置信区间。
|
||||
|
||||
固定 condition 行顺序：
|
||||
|
||||
```text
|
||||
s0_eos, s1_eos, s0_bos, s1_bos,
|
||||
s0_x, s1_x, s0_period, s1_period
|
||||
```
|
||||
|
||||
改变行顺序会改变随机轨迹，必须判为不同协议。
|
||||
|
||||
---
|
||||
|
||||
## 5. 网格与执行闸门
|
||||
|
||||
### 5.1 技术 smoke
|
||||
|
||||
```text
|
||||
4 sources × 2 base seeds (R0/R1) × 8 conditions × 16 new tokens
|
||||
= 64 short outputs
|
||||
|
||||
另将 R0 在同一进程内重放一次：
|
||||
|
||||
4 sources × 1 replay seed (R0) × 8 conditions × 16 new tokens
|
||||
= 32 replay outputs
|
||||
```
|
||||
|
||||
技术 smoke 只验证：
|
||||
|
||||
1. prompt hashes 与 Round 05 的 32 格 exact；
|
||||
2. 官方 temperature / top-p 实际进入 `generate()`；
|
||||
3. `top_k=0`，没有隐式默认 top-k；
|
||||
4. 64 个正式 smoke 输出正常，无 OOM / NaN / runtime exception；
|
||||
5. R0 在同进程按相同 seed 重跑时 32 / 32 token exact；
|
||||
6. R0 与 R1 至少有一个同 source、同 condition 的 cell 分叉；若 32 / 32 仍相同，只能继续调查，不能声称 sampling
|
||||
已生效。
|
||||
|
||||
smoke 的 16-token 输出不进入正式统计。
|
||||
|
||||
### 5.2 正式网格
|
||||
|
||||
```text
|
||||
4 sources
|
||||
× 8 base seeds
|
||||
× 8 conditions
|
||||
× 512 new-token cap
|
||||
= 256 sampled outputs
|
||||
```
|
||||
|
||||
source 顺序与 replicate 顺序固定，不根据中间结果提前停止。某一格遇到 EOS 后由
|
||||
Transformers 填 PAD；不能单独给未完成格追加预算。
|
||||
|
||||
### 5.3 独立复跑
|
||||
|
||||
正式完成后新启动 Python 进程、重新加载模型：
|
||||
|
||||
```text
|
||||
4 sources × R0/R1 × 8 conditions
|
||||
= 64 outputs
|
||||
```
|
||||
|
||||
必须逐格比较：
|
||||
|
||||
- run seed；
|
||||
- prompt token hash；
|
||||
- 完整 generated token IDs；
|
||||
- decoded text；
|
||||
- EOS / truncation state；
|
||||
- CPU / CUDA RNG pre-state hash。
|
||||
|
||||
只能写“64 / 64 sampled trajectories independently reproduced”，不能写 256 / 256。
|
||||
|
||||
---
|
||||
|
||||
## 6. completion 与任务评测仍沿用四张账
|
||||
|
||||
每条 sampled output 继续区分：
|
||||
|
||||
1. stopping：natural EOS / budget truncated / other stop；
|
||||
2. task terminal：明确 final marker 或闭合 code fence；
|
||||
3. evaluator coverage：数值可抽取或代码 AST + 沙箱可执行；
|
||||
4. correctness：GSM8K gold exact / HumanEval official tests pass。
|
||||
|
||||
HumanEval 每个 candidate 进入全新 pinned 容器，沿用 Round 05：
|
||||
|
||||
```text
|
||||
network none · read-only filesystem · user 65534:65534
|
||||
cap-drop ALL · no-new-privileges · no host mounts
|
||||
256 MiB memory/swap · pids 64 · cpus .5 · timeout 5s
|
||||
```
|
||||
|
||||
同一代码文本若在多个 seed 重复，评测器仍保存每格结果，但可以按 candidate SHA-256
|
||||
缓存一次沙箱执行；缓存必须只复用完全相同的 candidate + task + harness。
|
||||
|
||||
---
|
||||
|
||||
## 7. 预注册统计
|
||||
|
||||
### 7.1 每个 source×condition 的 8-sample 集合
|
||||
|
||||
- natural EOS / 8；
|
||||
- budget truncated / 8；
|
||||
- unique generated-token hashes / 8；
|
||||
- greedy trajectory 是否出现在 8 个 sampled hashes 中；
|
||||
- 28 对样本两两 token similarity 的 mean / min / max；
|
||||
- 生成长度 mean / min / max；
|
||||
- Math：8 个抽取答案、答案频次、majority final 与 gold；
|
||||
- Code：AST / executed / official tests pass 频次。
|
||||
|
||||
“8/8 unique”表示完整 token 序列不重复，不表示语义有八种。
|
||||
|
||||
### 7.2 每条 edge 的两个样本集合
|
||||
|
||||
十条 edge 沿用前序定义。每个 source 报告：
|
||||
|
||||
- 同 replicate label 的 8 对 token similarity；
|
||||
- 左集合每条样本到右集合的最高相似度，再与反方向平均：
|
||||
`symmetric mean nearest-neighbor similarity`；
|
||||
- 两边完整序列 hash-set intersection / union；
|
||||
- completion class 计数差；
|
||||
- Math / Code pass 频次差。
|
||||
|
||||
nearest-neighbor similarity 是 8-sample 描述量，不是分布距离的无偏估计。
|
||||
|
||||
### 7.3 汇总层
|
||||
|
||||
只做：
|
||||
|
||||
- 四条 source 合计的分子 / 分母；
|
||||
- 按域逐 source 展示；
|
||||
- 按 condition / edge 的描述性均值。
|
||||
|
||||
不做：
|
||||
|
||||
- 把 256 当作 256 个独立 benchmark tasks；
|
||||
- 忽略同一 source 的重复采样聚类；
|
||||
- 用 seed 当 source bootstrap unit；
|
||||
- 根据 1 条 Math / Code source 给置信区间；
|
||||
- best-of-8 与标准 pass@k 横比。
|
||||
|
||||
---
|
||||
|
||||
## 8. reproduction 与随机性要同时成立
|
||||
|
||||
采样实验的“可复现”不是要求所有 seed 输出相同，而是：
|
||||
|
||||
```text
|
||||
相同 checkpoint + 输入 + 软件 + 行顺序 + seed
|
||||
→ 同一条 sampled token trajectory
|
||||
|
||||
不同 run seed
|
||||
→ 允许、并预期至少一部分 trajectory 分叉
|
||||
```
|
||||
|
||||
正式结果必须同时报告：
|
||||
|
||||
- same-seed exact reproduction；
|
||||
- across-seed diversity。
|
||||
|
||||
只报告前者会把采样伪装成 greedy；只报告后者又无法判断差异来自随机数还是环境漂移。
|
||||
|
||||
---
|
||||
|
||||
## 9. 仍然禁止的结论
|
||||
|
||||
- 8 seeds 已恢复完整生成分布；
|
||||
- 某个边界“更有创造力”或“更稳定”；
|
||||
- BOS / `x` / 句点是官方有效聊天条件；
|
||||
- 1 条 GSM8K / HumanEval 代表任务能力；
|
||||
- sampled majority 等于标准 self-consistency；
|
||||
- unique sequence count 等于语义多样性；
|
||||
- batch-seed aligned 等于 common-random-number paired；
|
||||
- greedy 没出现在 8 samples 中就说明 greedy 不可信；
|
||||
- sampling 输出差异由某层 router 或 hidden state 中介；
|
||||
- CPU-offloaded eager 延迟代表服务吞吐；
|
||||
- exact seed replay 可以跨 PyTorch / Transformers / CUDA 版本保证。
|
||||
|
||||
---
|
||||
|
||||
## 10. 失败与修订规则
|
||||
|
||||
- 技术 smoke 前可以修正代码错误，但正式 seed/grid/metrics 不随输出调整；
|
||||
- OOM 时先降低 static GPU placement，保留八格 batch；任何改动都要用同 seed smoke 做
|
||||
完整 token exact 跨放置闸门；
|
||||
- 若必须拆八格 batch，正式协议作废并重新注册，因为 RNG 行消费与数值 batch content 都变了；
|
||||
- 正式运行在写完整 JSON 前失败，不保留部分 seed 做结果；
|
||||
- evaluator bug 可以修复并重跑 evaluator，但不能重选 sampled outputs；
|
||||
- same-seed 复跑不 exact 时，结果只作失败诊断，不发布 sampling robustness 结论。
|
||||
@@ -0,0 +1,334 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFileSync, statSync, writeFileSync } from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
|
||||
const root = resolve(import.meta.dirname, "..");
|
||||
const paths = {
|
||||
sampling: resolve(
|
||||
root,
|
||||
"src/data/deepseek-v2-lite-chat-sampling.json",
|
||||
),
|
||||
evaluation: resolve(
|
||||
root,
|
||||
"src/data/deepseek-v2-lite-chat-sampling-eval.json",
|
||||
),
|
||||
rerun: resolve(
|
||||
root,
|
||||
"src/data/deepseek-v2-lite-chat-sampling-repro-r0r1.json",
|
||||
),
|
||||
reproduction: resolve(
|
||||
root,
|
||||
"src/data/deepseek-v2-lite-chat-sampling-reproduction.json",
|
||||
),
|
||||
output: resolve(
|
||||
root,
|
||||
"src/data/deepseek-v2-lite-chat-sampling-compact.json",
|
||||
),
|
||||
};
|
||||
|
||||
const readJson = (path) => JSON.parse(readFileSync(path, "utf8"));
|
||||
const sha256 = (path) => createHash("sha256")
|
||||
.update(readFileSync(path))
|
||||
.digest("hex");
|
||||
const artifact = (path) => ({
|
||||
bytes: statSync(path).size,
|
||||
sha256: sha256(path),
|
||||
});
|
||||
const mean = (values) => (
|
||||
values.length
|
||||
? values.reduce((total, value) => total + value, 0) / values.length
|
||||
: null
|
||||
);
|
||||
|
||||
const sampling = readJson(paths.sampling);
|
||||
const evaluation = readJson(paths.evaluation);
|
||||
const reproduction = readJson(paths.reproduction);
|
||||
const samplingArtifact = artifact(paths.sampling);
|
||||
const evaluationArtifact = artifact(paths.evaluation);
|
||||
const rerunArtifact = artifact(paths.rerun);
|
||||
|
||||
if (evaluation.input.sampling_sha256 !== samplingArtifact.sha256) {
|
||||
throw new Error("sampling evaluator input hash contract failed");
|
||||
}
|
||||
if (reproduction.formal.sha256 !== samplingArtifact.sha256) {
|
||||
throw new Error("reproduction formal hash contract failed");
|
||||
}
|
||||
if (reproduction.rerun.sha256 !== rerunArtifact.sha256) {
|
||||
throw new Error("reproduction rerun hash contract failed");
|
||||
}
|
||||
if (
|
||||
reproduction.summary.all_preregistered_fields_exact
|
||||
!== reproduction.summary.cells
|
||||
) {
|
||||
throw new Error("sampling reproduction is not exact");
|
||||
}
|
||||
|
||||
const conditions = sampling.seed_contract.condition_row_order;
|
||||
const edges = Object.keys(
|
||||
sampling.summary.by_source_edge[sampling.sources[0].id],
|
||||
);
|
||||
const edgeLabels = {
|
||||
system_eos: "System on − off · EOS",
|
||||
system_bos: "System on − off · BOS",
|
||||
system_x: "System on − off · x",
|
||||
system_period: "System on − off · 句点",
|
||||
bos_at_s0: "BOS − EOS · system off",
|
||||
bos_at_s1: "BOS − EOS · system on",
|
||||
x_at_s0: "x − EOS · system off",
|
||||
x_at_s1: "x − EOS · system on",
|
||||
period_at_s0: "句点 − EOS · system off",
|
||||
period_at_s1: "句点 − EOS · system on",
|
||||
};
|
||||
|
||||
const evalKey = (sourceId, baseSeed, condition) => (
|
||||
`${sourceId}\0${baseSeed}\0${condition}`
|
||||
);
|
||||
const evaluationByKey = new Map(
|
||||
evaluation.rows.map((row) => [
|
||||
evalKey(row.source_id, row.base_seed, row.condition),
|
||||
row,
|
||||
]),
|
||||
);
|
||||
|
||||
const sourceRows = sampling.sources.map((source) => {
|
||||
const conditionRows = Object.fromEntries(conditions.map((condition) => {
|
||||
const rawSummary = sampling.summary.by_source_condition[
|
||||
source.id
|
||||
][condition];
|
||||
const evalSummary = evaluation.summary.by_source_condition[
|
||||
source.id
|
||||
][condition];
|
||||
const samples = source.runs.map((run) => {
|
||||
const output = run.outputs.find(
|
||||
(candidate) => candidate.condition === condition,
|
||||
);
|
||||
const assessed = evaluationByKey.get(
|
||||
evalKey(source.id, run.base_seed, condition),
|
||||
);
|
||||
if (!output || !assessed) {
|
||||
throw new Error(
|
||||
`sample/evaluation row missing: ${source.id}/${run.base_seed}/${condition}`,
|
||||
);
|
||||
}
|
||||
return {
|
||||
replicate: run.replicate_label,
|
||||
baseSeed: run.base_seed,
|
||||
runSeed: run.run_seed,
|
||||
generatedTokens: output.generated_tokens,
|
||||
hitEos: output.hit_eos,
|
||||
truncated: output.stopped_at_max_new_tokens,
|
||||
trajectoryHash: output.generated_token_ids_sha256,
|
||||
textHash: output.text_sha256,
|
||||
preview: output.text.replace(/\s+/g, " ").trim().slice(0, 220),
|
||||
completionClass: assessed.completion_class,
|
||||
math: assessed.task_evaluation && source.domain === "math"
|
||||
? {
|
||||
predicted: assessed.task_evaluation.predicted_final,
|
||||
gold: assessed.task_evaluation.gold_final,
|
||||
exact: assessed.task_evaluation.fixed_budget_numeric_exact,
|
||||
method: assessed.task_evaluation.extraction_method,
|
||||
}
|
||||
: null,
|
||||
code: assessed.task_evaluation && source.domain === "code"
|
||||
? {
|
||||
ast: assessed.task_evaluation.python_ast_parse,
|
||||
status: assessed.task_evaluation.execution.status,
|
||||
passed: assessed.task_evaluation.fixed_budget_tests_pass,
|
||||
cacheHit: assessed.task_evaluation.execution_cache_hit,
|
||||
}
|
||||
: null,
|
||||
};
|
||||
});
|
||||
return [condition, {
|
||||
samples: rawSummary.samples,
|
||||
naturalEos: rawSummary.natural_eos,
|
||||
truncated: rawSummary.budget_truncated,
|
||||
uniqueTrajectories: rawSummary.unique_generated_token_hashes,
|
||||
greedyInSamples: rawSummary.greedy_full_trajectory_in_sample_set,
|
||||
pairwiseSimilarity: rawSummary.pairwise_token_similarity,
|
||||
generatedTokens: rawSummary.generated_tokens,
|
||||
evaluation: evalSummary,
|
||||
trajectories: samples,
|
||||
}];
|
||||
}));
|
||||
return {
|
||||
id: source.id,
|
||||
domain: source.domain,
|
||||
label: source.label,
|
||||
sourceCharacters: source.source_characters,
|
||||
sourceTokens: source.source_tokens,
|
||||
conditions: conditionRows,
|
||||
edges: Object.fromEntries(edges.map((edge) => [
|
||||
edge,
|
||||
{
|
||||
...sampling.summary.by_source_edge[source.id][edge],
|
||||
evaluation: evaluation.summary.by_source_edge[source.id][edge],
|
||||
},
|
||||
])),
|
||||
};
|
||||
});
|
||||
|
||||
const conditionSummary = Object.fromEntries(conditions.map((condition) => {
|
||||
const sourceConditions = sourceRows.map(
|
||||
(source) => source.conditions[condition],
|
||||
);
|
||||
const assessed = evaluation.summary.by_condition[condition];
|
||||
return [condition, {
|
||||
outputs: assessed.outputs,
|
||||
naturalEos: assessed.natural_eos,
|
||||
truncated: assessed.budget_truncated,
|
||||
meanGeneratedTokens: assessed.mean_generated_tokens,
|
||||
uniqueTrajectoriesAcrossSourceSets: sourceConditions.reduce(
|
||||
(total, row) => total + row.uniqueTrajectories,
|
||||
0,
|
||||
),
|
||||
fullEightWayDiversitySets: sourceConditions.filter(
|
||||
(row) => row.uniqueTrajectories === 8,
|
||||
).length,
|
||||
greedyIncludedSourceSets: sourceConditions.filter(
|
||||
(row) => row.greedyInSamples,
|
||||
).length,
|
||||
math: assessed.math,
|
||||
code: assessed.code,
|
||||
}];
|
||||
}));
|
||||
|
||||
const edgeSummary = Object.fromEntries(edges.map((edge) => {
|
||||
const rows = sourceRows.map((source) => source.edges[edge]);
|
||||
return [edge, {
|
||||
label: edgeLabels[edge],
|
||||
sources: rows.length,
|
||||
meanAlignedSimilarity: mean(rows.map(
|
||||
(row) => row.batch_seed_aligned_similarity.mean,
|
||||
)),
|
||||
meanSymmetricNearestSimilarity: mean(rows.map(
|
||||
(row) => row.symmetric_mean_nearest_neighbor_similarity,
|
||||
)),
|
||||
exactHashIntersections: rows.reduce(
|
||||
(total, row) => total + row.generated_hash_set_intersection,
|
||||
0,
|
||||
),
|
||||
exactHashUnion: rows.reduce(
|
||||
(total, row) => total + row.generated_hash_set_union,
|
||||
0,
|
||||
),
|
||||
naturalEosDeltaRightMinusLeft: rows.reduce(
|
||||
(total, row) => (
|
||||
total + row.natural_eos_count_difference_right_minus_left
|
||||
),
|
||||
0,
|
||||
),
|
||||
}];
|
||||
}));
|
||||
|
||||
const result = {
|
||||
schemaVersion: 1,
|
||||
generatedAt: new Date().toISOString(),
|
||||
contract: {
|
||||
protocolId: sampling.protocol_id,
|
||||
model: sampling.model.repo,
|
||||
revision: sampling.model.revision,
|
||||
checkpointIdentity: sampling.model.checkpoint_identity,
|
||||
conditions,
|
||||
conditionFactors: sampling.generation_contract.conditions,
|
||||
edges,
|
||||
edgeLabels,
|
||||
sources: sampling.sources.length,
|
||||
replicates: sampling.seed_contract.executed_base_seeds.length,
|
||||
baseSeeds: sampling.seed_contract.executed_base_seeds,
|
||||
decode: {
|
||||
doSample: sampling.generation_contract.do_sample,
|
||||
temperature: sampling.generation_contract.temperature,
|
||||
topP: sampling.generation_contract.top_p,
|
||||
topK: sampling.generation_contract.top_k,
|
||||
maxNewTokens: sampling.generation_contract.max_new_tokens,
|
||||
},
|
||||
batchSeedAlignedNotCommonRandomNumbers: (
|
||||
sampling.seed_contract.batch_seed_aligned_not_common_random_numbers
|
||||
),
|
||||
counterfactualBoundary: (
|
||||
sampling.generation_contract.counterfactual_boundary
|
||||
),
|
||||
},
|
||||
headline: {
|
||||
outputs: sampling.summary.outputs,
|
||||
naturalEos: sampling.summary.natural_eos,
|
||||
truncated: sampling.summary.budget_truncated,
|
||||
uniqueTrajectoryHashes: (
|
||||
sampling.summary.unique_generated_token_hashes
|
||||
),
|
||||
firstTwoSeedComparableCells: (
|
||||
sampling.summary.first_two_seeds.comparable_cells
|
||||
),
|
||||
firstTwoSeedDifferentTrajectories: (
|
||||
sampling.summary.first_two_seeds.different_trajectories
|
||||
),
|
||||
promptHashExact: (
|
||||
sampling.source_contract.prompt_hash_audit.exact
|
||||
),
|
||||
promptHashCells: (
|
||||
sampling.source_contract.prompt_hash_audit.cells
|
||||
),
|
||||
greedyIncludedSourceConditionSets: sourceRows.reduce(
|
||||
(total, source) => total + conditions.filter(
|
||||
(condition) => source.conditions[condition].greedyInSamples,
|
||||
).length,
|
||||
0,
|
||||
),
|
||||
sourceConditionSets: sourceRows.length * conditions.length,
|
||||
mathExact: evaluation.summary.math.fixed_budget_exact,
|
||||
mathOutputs: evaluation.summary.math.outputs,
|
||||
mathMajority: evaluation.summary.math.unique_absolute_majority,
|
||||
mathGold: evaluation.summary.math.gold,
|
||||
codePassed: evaluation.summary.code.tests_pass,
|
||||
codeOutputs: evaluation.summary.code.outputs,
|
||||
codeUniqueExecutionKeys: (
|
||||
evaluation.sandbox.unique_code_cache_entries
|
||||
),
|
||||
reproducedCells: (
|
||||
reproduction.summary.all_preregistered_fields_exact
|
||||
),
|
||||
reproductionCells: reproduction.summary.cells,
|
||||
},
|
||||
conditions: conditionSummary,
|
||||
edges: edgeSummary,
|
||||
sources: sourceRows,
|
||||
reproduction: reproduction.summary,
|
||||
artifacts: {
|
||||
sampling: samplingArtifact,
|
||||
evaluation: evaluationArtifact,
|
||||
rerun: rerunArtifact,
|
||||
reproduction: artifact(paths.reproduction),
|
||||
},
|
||||
execution: {
|
||||
generationSeconds: sampling.sources.reduce(
|
||||
(total, source) => total + source.runs.reduce(
|
||||
(sourceTotal, run) => sourceTotal + run.generation_seconds,
|
||||
0,
|
||||
),
|
||||
0,
|
||||
),
|
||||
peakCudaMemoryAllocatedBytes: (
|
||||
sampling.execution.peak_cuda_memory_allocated_bytes
|
||||
),
|
||||
deviceMap: sampling.execution.device_map,
|
||||
transformers: sampling.execution.transformers,
|
||||
torch: sampling.execution.torch,
|
||||
sandbox: evaluation.sandbox,
|
||||
},
|
||||
claimBoundary: [
|
||||
...sampling.claim_boundary,
|
||||
...evaluation.claim_boundary,
|
||||
...reproduction.claim_boundary,
|
||||
],
|
||||
};
|
||||
|
||||
writeFileSync(
|
||||
paths.output,
|
||||
`${JSON.stringify(result, null, 2)}\n`,
|
||||
);
|
||||
console.log(JSON.stringify({
|
||||
output: paths.output,
|
||||
...artifact(paths.output),
|
||||
headline: result.headline,
|
||||
}, null, 2));
|
||||
@@ -1210,14 +1210,14 @@ console.log(JSON.stringify(report, null, 2));
|
||||
const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
|
||||
const failures = [];
|
||||
if (!overview.title.includes("为什么转向")) failures.push("专题标题异常");
|
||||
if (overview.sections !== 28 || overview.tocLinks !== 28) failures.push("二十七个编号专题加阅读链的目录结构异常");
|
||||
if (overview.sections !== 29 || overview.tocLinks !== 29) failures.push("二十八个编号专题加阅读链的目录结构异常");
|
||||
if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常");
|
||||
if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
if (overview.artifactTabs !== 13 || overview.artifactPanels !== 13 || overview.artifactLayers !== 27) failures.push("真实权重十三联实验结构异常");
|
||||
if (overview.behaviorTabs !== 4 || overview.behaviorPanels !== 4 || overview.behaviorSources !== 16 || overview.behaviorEdges !== 10) failures.push("Chat 行为实验结构异常");
|
||||
if (overview.completionDepthTabs !== 4 || overview.completionDepthPanels !== 4 || overview.hiddenStages !== 29 || overview.routerLayers !== 26) failures.push("Chat 完成度与全深度实验结构异常");
|
||||
if (overview.heroLabs !== "19 个可操作实验") failures.push("DeepSeek 实验总数账异常");
|
||||
if (overview.heroLabs !== "20 个可操作实验") failures.push("DeepSeek 实验总数账异常");
|
||||
if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
|
||||
if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常");
|
||||
|
||||
@@ -0,0 +1,278 @@
|
||||
import { writeFileSync } from "node:fs";
|
||||
|
||||
const cdpPort = process.env.CDP_PORT ?? "9230";
|
||||
const baseUrl = process.env.SITE_URL ?? "http://127.0.0.1:4327";
|
||||
const pages = await fetch(
|
||||
`http://127.0.0.1:${cdpPort}/json/list`,
|
||||
).then((response) => response.json());
|
||||
const page = pages.find((entry) => entry.type === "page");
|
||||
if (!page) throw new Error(`CDP ${cdpPort} 没有可用页面`);
|
||||
|
||||
const socket = new WebSocket(page.webSocketDebuggerUrl);
|
||||
await new Promise((resolve, reject) => {
|
||||
socket.addEventListener("open", resolve, { once: true });
|
||||
socket.addEventListener("error", reject, { once: true });
|
||||
});
|
||||
|
||||
let nextId = 0;
|
||||
const pending = new Map();
|
||||
const exceptions = [];
|
||||
socket.addEventListener("message", (event) => {
|
||||
const message = JSON.parse(event.data);
|
||||
if (message.id && pending.has(message.id)) {
|
||||
const { resolve, reject } = pending.get(message.id);
|
||||
pending.delete(message.id);
|
||||
if (message.error) reject(new Error(message.error.message));
|
||||
else resolve(message.result);
|
||||
}
|
||||
if (message.method === "Runtime.exceptionThrown") {
|
||||
exceptions.push(
|
||||
message.params.exceptionDetails.exception?.description
|
||||
?? message.params.exceptionDetails.text,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
const command = (method, params = {}) => new Promise((resolve, reject) => {
|
||||
const id = ++nextId;
|
||||
pending.set(id, { resolve, reject });
|
||||
socket.send(JSON.stringify({ id, method, params }));
|
||||
});
|
||||
const pause = (milliseconds) => new Promise(
|
||||
(resolve) => setTimeout(resolve, milliseconds),
|
||||
);
|
||||
const evaluate = async (expression) => {
|
||||
const result = await command("Runtime.evaluate", {
|
||||
expression,
|
||||
returnByValue: true,
|
||||
awaitPromise: true,
|
||||
});
|
||||
if (result.exceptionDetails) {
|
||||
throw new Error(
|
||||
result.exceptionDetails.exception?.description
|
||||
?? result.exceptionDetails.text,
|
||||
);
|
||||
}
|
||||
return result.result.value;
|
||||
};
|
||||
const navigate = async (path) => {
|
||||
await command("Page.navigate", { url: `${baseUrl}${path}` });
|
||||
for (let attempt = 0; attempt < 80; attempt += 1) {
|
||||
await pause(100);
|
||||
if (await evaluate("document.readyState === 'complete'")) return;
|
||||
}
|
||||
throw new Error(`${path} 加载超时`);
|
||||
};
|
||||
const screenshot = async (path) => {
|
||||
const result = await command("Page.captureScreenshot", {
|
||||
format: "png",
|
||||
captureBeyondViewport: false,
|
||||
});
|
||||
writeFileSync(path, Buffer.from(result.data, "base64"));
|
||||
};
|
||||
const assert = (condition, message) => {
|
||||
if (!condition) throw new Error(message);
|
||||
};
|
||||
|
||||
await command("Page.enable");
|
||||
await command("Runtime.enable");
|
||||
await command("Emulation.setDeviceMetricsOverride", {
|
||||
width: 1440,
|
||||
height: 1100,
|
||||
deviceScaleFactor: 1,
|
||||
mobile: false,
|
||||
});
|
||||
await navigate("/deepseek/");
|
||||
|
||||
const overview = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-sampling-lab]");
|
||||
document.documentElement.style.scrollBehavior = "auto";
|
||||
window.scrollTo(0, root.getBoundingClientRect().top + window.scrollY);
|
||||
return {
|
||||
tabs: root.querySelectorAll("[data-sp-tab]").length,
|
||||
panels: root.querySelectorAll("[data-sp-panel]").length,
|
||||
seedBars: root.querySelectorAll("[data-sp-seeds] button").length,
|
||||
conditions: root.querySelectorAll(".condition-overview > div").length,
|
||||
edges: root.querySelectorAll(".edge-overview > div").length,
|
||||
heroLabs: [...document.querySelectorAll(".page-facts > div")]
|
||||
.find((node) => node.querySelector("dt")?.textContent.trim() === "LABS")
|
||||
?.querySelector("dd")?.textContent.trim(),
|
||||
activePanel: root.querySelector("[data-sp-panel]:not([hidden])")
|
||||
?.dataset.spPanel,
|
||||
unique: root.querySelector("[data-sp-unique]").textContent.trim(),
|
||||
eos: root.querySelector("[data-sp-eos]").textContent.trim(),
|
||||
greedy: root.querySelector("[data-sp-greedy]").textContent.trim(),
|
||||
documentOverflow: (
|
||||
document.documentElement.scrollWidth
|
||||
- document.documentElement.clientWidth
|
||||
),
|
||||
};
|
||||
})()`);
|
||||
await pause(300);
|
||||
await screenshot("/tmp/llm-atlas-sampling-desktop.png");
|
||||
|
||||
const trajectorySwitch = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-sampling-lab]");
|
||||
const source = root.querySelector("[data-sp-source]");
|
||||
const condition = root.querySelector("[data-sp-condition]");
|
||||
source.value = "HumanEval/31";
|
||||
source.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
condition.value = "s1_period";
|
||||
condition.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
const buttons = [...root.querySelectorAll("[data-sp-seeds] button")];
|
||||
buttons.at(-1).click();
|
||||
return {
|
||||
unique: root.querySelector("[data-sp-unique]").textContent.trim(),
|
||||
eos: root.querySelector("[data-sp-eos]").textContent.trim(),
|
||||
greedy: root.querySelector("[data-sp-greedy]").textContent.trim(),
|
||||
seedBars: buttons.length,
|
||||
repeats: buttons.filter((button) => button.classList.contains("repeat")).length,
|
||||
selected: buttons.filter((button) => button.classList.contains("selected")).length,
|
||||
focus: root.querySelector("[data-sp-focus-seed]").textContent.trim(),
|
||||
hash: root.querySelector("[data-sp-focus-hash]").textContent.trim(),
|
||||
};
|
||||
})()`);
|
||||
|
||||
const taskSwitch = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-sampling-lab]");
|
||||
root.querySelector('[data-sp-tab="tasks"]').click();
|
||||
const condition = root.querySelector("[data-sp-task-condition]");
|
||||
const read = () => ({
|
||||
panel: root.querySelector("[data-sp-panel]:not([hidden])").dataset.spPanel,
|
||||
math: root.querySelector("[data-sp-math-total]").textContent.trim(),
|
||||
code: root.querySelector("[data-sp-code-total]").textContent.trim(),
|
||||
mathCells: root.querySelectorAll("[data-sp-math-cells] article").length,
|
||||
codeCells: root.querySelectorAll("[data-sp-code-cells] article").length,
|
||||
});
|
||||
condition.value = "s1_x";
|
||||
condition.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
const systemX = read();
|
||||
condition.value = "s0_period";
|
||||
condition.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
const period = read();
|
||||
return { systemX, period };
|
||||
})()`);
|
||||
|
||||
const edgeSwitch = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-sampling-lab]");
|
||||
root.querySelector('[data-sp-tab="edges"]').click();
|
||||
const source = root.querySelector("[data-sp-edge-source]");
|
||||
const edge = root.querySelector("[data-sp-edge]");
|
||||
source.value = "gsm8k/test/1069";
|
||||
source.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
edge.value = "system_eos";
|
||||
edge.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
return {
|
||||
panel: root.querySelector("[data-sp-panel]:not([hidden])").dataset.spPanel,
|
||||
aligned: root.querySelector("[data-sp-edge-aligned]").textContent.trim(),
|
||||
nearest: root.querySelector("[data-sp-edge-nearest]").textContent.trim(),
|
||||
overlap: root.querySelector("[data-sp-edge-overlap]").textContent.trim(),
|
||||
left: root.querySelectorAll("[data-sp-edge-left-set] article").length,
|
||||
right: root.querySelectorAll("[data-sp-edge-right-set] article").length,
|
||||
highlighted: root.querySelectorAll(".two-sets article.overlap").length,
|
||||
};
|
||||
})()`);
|
||||
|
||||
const keyboard = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-sampling-lab]");
|
||||
const first = root.querySelector('[data-sp-tab="trajectories"]');
|
||||
first.click();
|
||||
first.focus();
|
||||
first.dispatchEvent(new KeyboardEvent("keydown", {
|
||||
key: "ArrowRight",
|
||||
bubbles: true,
|
||||
}));
|
||||
const selected = root.querySelector(
|
||||
'[data-sp-tab][aria-selected="true"]',
|
||||
).dataset.spTab;
|
||||
root.querySelector('[data-sp-tab="reproduction"]').click();
|
||||
return {
|
||||
keyboardSelected: selected,
|
||||
visible: root.querySelector("[data-sp-panel]:not([hidden])")
|
||||
.dataset.spPanel,
|
||||
reproFields: root.querySelectorAll(".repro-fields article").length,
|
||||
};
|
||||
})()`);
|
||||
|
||||
await command("Emulation.setDeviceMetricsOverride", {
|
||||
width: 390,
|
||||
height: 844,
|
||||
deviceScaleFactor: 1,
|
||||
mobile: true,
|
||||
});
|
||||
await navigate("/deepseek/#sampling");
|
||||
const mobile = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-sampling-lab]");
|
||||
document.documentElement.style.scrollBehavior = "auto";
|
||||
window.scrollTo(0, root.getBoundingClientRect().top + window.scrollY);
|
||||
return {
|
||||
tabs: root.querySelectorAll("[data-sp-tab]").length,
|
||||
seedBars: root.querySelectorAll("[data-sp-seeds] button").length,
|
||||
documentOverflow: (
|
||||
document.documentElement.scrollWidth
|
||||
- document.documentElement.clientWidth
|
||||
),
|
||||
rootOverflow: root.scrollWidth - root.clientWidth,
|
||||
};
|
||||
})()`);
|
||||
await pause(300);
|
||||
await screenshot("/tmp/llm-atlas-sampling-mobile.png");
|
||||
|
||||
assert(overview.tabs === 4, `sampling tabs=${overview.tabs}`);
|
||||
assert(overview.panels === 4, `sampling panels=${overview.panels}`);
|
||||
assert(overview.seedBars === 8, `initial seed bars=${overview.seedBars}`);
|
||||
assert(overview.conditions === 8, `condition rows=${overview.conditions}`);
|
||||
assert(overview.edges === 10, `edge rows=${overview.edges}`);
|
||||
assert(overview.heroLabs?.startsWith("20"), `hero labs=${overview.heroLabs}`);
|
||||
assert(overview.activePanel === "trajectories", `active=${overview.activePanel}`);
|
||||
assert(overview.unique === "8 / 8", `initial unique=${overview.unique}`);
|
||||
assert(overview.eos === "5 / 8", `initial eos=${overview.eos}`);
|
||||
assert(overview.greedy === "NO", `initial greedy=${overview.greedy}`);
|
||||
assert(overview.documentOverflow === 0, `desktop overflow=${overview.documentOverflow}`);
|
||||
|
||||
assert(trajectorySwitch.unique === "2 / 8", `switched unique=${trajectorySwitch.unique}`);
|
||||
assert(trajectorySwitch.eos === "8 / 8", `switched eos=${trajectorySwitch.eos}`);
|
||||
assert(trajectorySwitch.greedy === "YES", `switched greedy=${trajectorySwitch.greedy}`);
|
||||
assert(trajectorySwitch.seedBars === 8, `switched seeds=${trajectorySwitch.seedBars}`);
|
||||
assert(trajectorySwitch.repeats >= 2, `repeat markers=${trajectorySwitch.repeats}`);
|
||||
assert(trajectorySwitch.selected === 1, `selected seeds=${trajectorySwitch.selected}`);
|
||||
assert(trajectorySwitch.focus.startsWith("R7"), `focus=${trajectorySwitch.focus}`);
|
||||
assert(trajectorySwitch.hash.includes("sha256"), `focus hash=${trajectorySwitch.hash}`);
|
||||
|
||||
assert(taskSwitch.systemX.panel === "tasks", `task panel=${taskSwitch.systemX.panel}`);
|
||||
assert(taskSwitch.systemX.math === "7 / 8 PASS", `system x math=${taskSwitch.systemX.math}`);
|
||||
assert(taskSwitch.systemX.code === "8 / 8 PASS", `system x code=${taskSwitch.systemX.code}`);
|
||||
assert(taskSwitch.period.code === "7 / 8 PASS", `period code=${taskSwitch.period.code}`);
|
||||
assert(taskSwitch.period.mathCells === 8, `math cells=${taskSwitch.period.mathCells}`);
|
||||
assert(taskSwitch.period.codeCells === 8, `code cells=${taskSwitch.period.codeCells}`);
|
||||
|
||||
assert(edgeSwitch.panel === "edges", `edge panel=${edgeSwitch.panel}`);
|
||||
assert(edgeSwitch.aligned === "77.3%", `edge aligned=${edgeSwitch.aligned}`);
|
||||
assert(edgeSwitch.nearest === "89.4%", `edge nearest=${edgeSwitch.nearest}`);
|
||||
assert(edgeSwitch.overlap === "1 / 14", `edge overlap=${edgeSwitch.overlap}`);
|
||||
assert(edgeSwitch.left === 8 && edgeSwitch.right === 8, `edge sets=${edgeSwitch.left}/${edgeSwitch.right}`);
|
||||
assert(edgeSwitch.highlighted >= 2, `edge highlights=${edgeSwitch.highlighted}`);
|
||||
|
||||
assert(keyboard.keyboardSelected === "tasks", `keyboard selected=${keyboard.keyboardSelected}`);
|
||||
assert(keyboard.visible === "reproduction", `repro visible=${keyboard.visible}`);
|
||||
assert(keyboard.reproFields === 8, `repro fields=${keyboard.reproFields}`);
|
||||
assert(mobile.tabs === 4 && mobile.seedBars === 8, `mobile controls=${mobile.tabs}/${mobile.seedBars}`);
|
||||
assert(mobile.documentOverflow === 0, `mobile document overflow=${mobile.documentOverflow}`);
|
||||
assert(mobile.rootOverflow === 0, `mobile root overflow=${mobile.rootOverflow}`);
|
||||
assert(exceptions.length === 0, `runtime exceptions: ${exceptions.join(" | ")}`);
|
||||
|
||||
console.log(JSON.stringify({
|
||||
overview,
|
||||
trajectorySwitch,
|
||||
taskSwitch,
|
||||
edgeSwitch,
|
||||
keyboard,
|
||||
mobile,
|
||||
exceptions,
|
||||
screenshots: [
|
||||
"/tmp/llm-atlas-sampling-desktop.png",
|
||||
"/tmp/llm-atlas-sampling-mobile.png",
|
||||
],
|
||||
}, null, 2));
|
||||
|
||||
socket.close();
|
||||
@@ -0,0 +1,998 @@
|
||||
---
|
||||
import rawLab from "@/data/deepseek-v2-lite-chat-sampling-compact.json";
|
||||
|
||||
const lab = rawLab as any;
|
||||
const json = JSON.stringify(lab).replaceAll("<", "\\u003c");
|
||||
const conditions = lab.contract.conditions as string[];
|
||||
const edges = lab.contract.edges as string[];
|
||||
const conditionLabels: Record<string, string> = {
|
||||
s0_eos: "S0 · EOS",
|
||||
s1_eos: "S1 · EOS",
|
||||
s0_bos: "S0 · BOS",
|
||||
s1_bos: "S1 · BOS",
|
||||
s0_x: "S0 · x",
|
||||
s1_x: "S1 · x",
|
||||
s0_period: "S0 · 句点",
|
||||
s1_period: "S1 · 句点",
|
||||
};
|
||||
const sourceLabels: Record<string, string> = {
|
||||
"wikitext2/raw-validation/0443": "English · WikiText",
|
||||
"tnews/test/4855": "中文 · TNEWS",
|
||||
"HumanEval/31": "Code · HumanEval/31",
|
||||
"gsm8k/test/1069": "Math · GSM8K/1069",
|
||||
};
|
||||
---
|
||||
|
||||
<figure class="sampling-lab" data-sampling-lab>
|
||||
<header class="sp-head">
|
||||
<div>
|
||||
<p>ROUND 06 / PREREGISTERED SAMPLING</p>
|
||||
<h3>greedy 只是一条路：固定八个 seed，打开有限的生成轨迹集合</h3>
|
||||
</div>
|
||||
<p>
|
||||
同一官方 SFT Chat checkpoint、同四条 source、同八格 prompt batch。
|
||||
只把解码切到官方 <code>temperature=.3 · top_p=.95</code>，显式关闭 top-k；
|
||||
生成、任务评测与新进程复跑仍分三层保存。
|
||||
</p>
|
||||
</header>
|
||||
|
||||
<div class="sp-ledger">
|
||||
<article class="pass"><span>SAMPLED OUTPUTS</span><b>256</b><p>4 sources × 8 seeds × 8 cells</p></article>
|
||||
<article class="pass"><span>NATURAL EOS</span><b>251 / 256</b><p>5 格在 512-token 触顶</p></article>
|
||||
<article><span>UNIQUE TRAJECTORIES</span><b>242 / 256</b><p>完整 token hash；不是语义类别</p></article>
|
||||
<article><span>MATH · STRICT</span><b>62 / 64</b><p>同一题的重复采样</p></article>
|
||||
<article><span>CODE · TESTS</span><b>63 / 64</b><p>24 个唯一执行候选</p></article>
|
||||
<article class="pass"><span>FRESH PROCESS</span><b>64 / 64</b><p>八项预注册字段 exact</p></article>
|
||||
</div>
|
||||
|
||||
<div class="sp-tabs" role="tablist" aria-label="选择采样稳健性实验视图">
|
||||
<button type="button" role="tab" data-sp-tab="trajectories" aria-selected="true">
|
||||
<span>01</span><b>八个 seed 生成了什么</b><small>trajectory microscope</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-sp-tab="tasks" aria-selected="false" tabindex="-1">
|
||||
<span>02</span><b>完成为什么仍会答错</b><small>math + code ledgers</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-sp-tab="edges" aria-selected="false" tabindex="-1">
|
||||
<span>03</span><b>两个样本集合怎样比</b><small>aligned + nearest set</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-sp-tab="reproduction" aria-selected="false" tabindex="-1">
|
||||
<span>04</span><b>随机但仍可复现</b><small>seed contract + rerun</small>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<section class="sp-panel" data-sp-panel="trajectories">
|
||||
<div class="sp-panel-lead">
|
||||
<div><span>I / TRAJECTORY MICROSCOPE</span><h4>同一格不是一个答案，而是八条有限样本</h4></div>
|
||||
<p>
|
||||
点任意 seed 查看长度、停止状态、hash 与短预览。unique 只比较完整 token IDs；
|
||||
pairwise similarity 用 token 编辑距离，不把同义改写冒充 exact。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="sp-controls">
|
||||
<label>
|
||||
<span>SOURCE</span>
|
||||
<select data-sp-source aria-label="选择采样 source">
|
||||
{lab.sources.map((source: any) => (
|
||||
<option value={source.id}>{sourceLabels[source.id]}</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>CONDITION</span>
|
||||
<select data-sp-condition aria-label="选择采样 condition">
|
||||
{conditions.map((condition) => (
|
||||
<option value={condition}>{conditionLabels[condition]}</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
<div class="sampling-config">
|
||||
<span>DECODE CONTRACT</span>
|
||||
<b>T .3 · P .95 · K 0 · CAP 512</b>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="trajectory-summary">
|
||||
<article><span>UNIQUE / 8</span><b data-sp-unique>—</b><p>完整 trajectory hash</p></article>
|
||||
<article><span>NATURAL EOS / 8</span><b data-sp-eos>—</b><p>触顶与自然结束分开</p></article>
|
||||
<article><span>GREEDY IN SET</span><b data-sp-greedy>—</b><p>八样本是否抽到 mode 轨迹</p></article>
|
||||
<article><span>PAIRWISE SIM</span><b data-sp-pairwise>—</b><p>28 对 mean · min–max</p></article>
|
||||
</div>
|
||||
|
||||
<div class="seed-stage">
|
||||
<header>
|
||||
<div><span>8 PREREGISTERED SEEDS</span><b data-sp-stage-title>—</b></div>
|
||||
<p><i class="eos"></i>EOS <i class="truncated"></i>512 截断 <i class="repeat"></i>重复完整轨迹</p>
|
||||
</header>
|
||||
<div class="seed-bars" data-sp-seeds></div>
|
||||
<article class="seed-focus" data-sp-focus>
|
||||
<div>
|
||||
<span data-sp-focus-seed>选择一个 seed</span>
|
||||
<b data-sp-focus-meta>—</b>
|
||||
</div>
|
||||
<p data-sp-focus-preview>—</p>
|
||||
<code data-sp-focus-hash>—</code>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<div class="condition-overview" aria-label="八个 condition 的采样摘要">
|
||||
<header><b>CONDITION</b><b>EOS</b><b>MEAN TOKENS</b><b>UNIQUE</b><b>GREEDY SETS</b></header>
|
||||
{conditions.map((condition) => {
|
||||
const row = lab.conditions[condition];
|
||||
return (
|
||||
<div>
|
||||
<span>{conditionLabels[condition]}</span>
|
||||
<b class={row.naturalEos === 32 ? "good" : "warn"}>{row.naturalEos} / 32</b>
|
||||
<b>{row.meanGeneratedTokens.toFixed(1)}</b>
|
||||
<b>{row.uniqueTrajectoriesAcrossSourceSets} / 32</b>
|
||||
<b>{row.greedyIncludedSourceSets} / 4</b>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
|
||||
<aside class="sp-warning">
|
||||
<b>9 / 32 个八样本集合包含 greedy 完整轨迹</b>
|
||||
<p>
|
||||
这不等于 greedy “不可信”。greedy 每步选 mode；nucleus sampling 从截断后的分布抽样。
|
||||
只抽八次没有遇到 mode 路径很正常。
|
||||
</p>
|
||||
</aside>
|
||||
</section>
|
||||
|
||||
<section class="sp-panel" data-sp-panel="tasks" hidden>
|
||||
<div class="sp-panel-lead">
|
||||
<div><span>II / TASK LEDGERS</span><h4>自然结束、能解析、能运行，仍然不保证正确</h4></div>
|
||||
<p>
|
||||
这里各只有一条 task source。64 是 8 conditions × 8 seeds，不是 64 道题；
|
||||
gold 与 tests 在生成冻结后才进入独立 evaluator。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="task-condition-control">
|
||||
<label><span>查看一个 condition 的 8 个 seed</span>
|
||||
<select data-sp-task-condition aria-label="选择任务 condition">
|
||||
{conditions.map((condition) => (
|
||||
<option value={condition}>{conditionLabels[condition]}</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
<div><span>全局任务账</span><b>Math 62 / 64 · Code 63 / 64</b></div>
|
||||
</div>
|
||||
|
||||
<div class="sample-task-grid">
|
||||
<article>
|
||||
<header><div><span>GSM8K / 1069</span><b data-sp-math-total>—</b></div><em>gold = 300</em></header>
|
||||
<div class="seed-task-cells" data-sp-math-cells></div>
|
||||
</article>
|
||||
<article>
|
||||
<header><div><span>HUMANEVAL / 31</span><b data-sp-code-total>—</b></div><em>official tests</em></header>
|
||||
<div class="seed-task-cells" data-sp-code-cells></div>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<div class="failure-cases">
|
||||
<article>
|
||||
<span>MATH FAILURE · 2 / 64</span>
|
||||
<h5>“坏掉 1/4”被误写成“只剩 1/4”</h5>
|
||||
<div class="reasoning-bug">
|
||||
<b>20 trucks × 1/4 = 5</b><i>错误语义跳跃</i><b>5 × 20 = 100</b>
|
||||
</div>
|
||||
<p>
|
||||
两条都自然 EOS，并明确输出 <code>{"\\boxed{100}"}</code>；抽取器没有错，
|
||||
是推理把仍可工作的 <code>3/4</code> 丢掉了。
|
||||
</p>
|
||||
</article>
|
||||
<article>
|
||||
<span>CODE FAILURE · 1 / 64</span>
|
||||
<h5>AST 合法、可以执行，却漏掉所有大于 2 的偶数</h5>
|
||||
<pre><code>if n < 2: return False
|
||||
if n == 2: return True
|
||||
for i in range(3, sqrt(n), 2): ...
|
||||
# 缺少 if n % 2 == 0: return False</code></pre>
|
||||
<p>official test 触发 assertion failure；流畅说明与闭合 code fence 都没有替它兜底。</p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<div class="four-ledgers">
|
||||
<article class="done"><span>01</span><b>STOPPING</b><p>EOS / budget cap</p></article>
|
||||
<i>→</i>
|
||||
<article class="done"><span>02</span><b>TASK TERMINAL</b><p>final marker / fence</p></article>
|
||||
<i>→</i>
|
||||
<article class="done"><span>03</span><b>COVERAGE</b><p>number / AST + sandbox</p></article>
|
||||
<i>→</i>
|
||||
<article><span>04</span><b>CORRECTNESS</b><p>gold / official tests</p></article>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="sp-panel" data-sp-panel="edges" hidden>
|
||||
<div class="sp-panel-lead">
|
||||
<div><span>III / SET-TO-SET COMPARISON</span><h4>不要拿一对随机回答，冒充两个条件的分布</h4></div>
|
||||
<p>
|
||||
aligned 看同 replicate label 的八对；nearest 让每条样本去另一侧寻找最接近轨迹，
|
||||
再双向平均。两者都只是八样本描述量。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="sp-controls edge-controls">
|
||||
<label><span>SOURCE</span>
|
||||
<select data-sp-edge-source aria-label="选择 edge source">
|
||||
{lab.sources.map((source: any) => (
|
||||
<option value={source.id}>{sourceLabels[source.id]}</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
<label><span>EDGE</span>
|
||||
<select data-sp-edge aria-label="选择采样 edge">
|
||||
{edges.map((edge) => (
|
||||
<option value={edge}>{lab.contract.edgeLabels[edge]}</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
<div class="sampling-config"><span>PAIRING STATUS</span><b>BATCH-SEED ALIGNED ≠ CRN</b></div>
|
||||
</div>
|
||||
|
||||
<div class="set-metrics">
|
||||
<article><span>ALIGNED MEAN</span><b data-sp-edge-aligned>—</b><i><u data-sp-edge-aligned-bar></u></i></article>
|
||||
<article><span>SYMMETRIC NEAREST</span><b data-sp-edge-nearest>—</b><i><u data-sp-edge-nearest-bar></u></i></article>
|
||||
<article><span>EXACT HASH ∩ / ∪</span><b data-sp-edge-overlap>—</b><p>完整 token trajectory</p></article>
|
||||
</div>
|
||||
|
||||
<div class="two-sets">
|
||||
<article>
|
||||
<header><span>LEFT SAMPLE SET</span><b data-sp-edge-left-label>—</b></header>
|
||||
<div data-sp-edge-left-set></div>
|
||||
</article>
|
||||
<div class="set-bridge"><b>8 ↔ 8</b><span>双向找最近邻</span><i></i></div>
|
||||
<article>
|
||||
<header><span>RIGHT SAMPLE SET</span><b data-sp-edge-right-label>—</b></header>
|
||||
<div data-sp-edge-right-set></div>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<div class="edge-overview">
|
||||
<header><b>EDGE · FOUR-SOURCE MEAN</b><b>ALIGNED</b><b>NEAREST</b><b>EXACT ∩ / ∪</b></header>
|
||||
{edges.map((edge) => {
|
||||
const row = lab.edges[edge];
|
||||
return (
|
||||
<div>
|
||||
<span>{row.label}</span>
|
||||
<b>{(row.meanAlignedSimilarity * 100).toFixed(1)}%</b>
|
||||
<b>{(row.meanSymmetricNearestSimilarity * 100).toFixed(1)}%</b>
|
||||
<b>{row.exactHashIntersections} / {row.exactHashUnion}</b>
|
||||
<i style={`--edge-nearest:${row.meanSymmetricNearestSimilarity * 100}%`}></i>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
|
||||
<aside class="sp-warning dark">
|
||||
<b>为什么 nearest 总比 aligned 高？</b>
|
||||
<p>
|
||||
nearest 主动从另一侧八条里挑最像的一条；aligned 没有这个选择自由。
|
||||
因此差值不是“条件效应”，nearest 也不是无偏分布距离。
|
||||
</p>
|
||||
</aside>
|
||||
</section>
|
||||
|
||||
<section class="sp-panel" data-sp-panel="reproduction" hidden>
|
||||
<div class="sp-panel-lead">
|
||||
<div><span>IV / RANDOMNESS × REPRODUCTION</span><h4>“会变化”和“可复现”必须同时成立</h4></div>
|
||||
<p>
|
||||
不同 seed 应该分叉；相同 checkpoint、输入、行顺序、软件与 seed 则应该回到同一条
|
||||
token trajectory。两者不是矛盾，而是采样实验的两道独立闸门。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="randomness-equation">
|
||||
<article><span>DIFFERENT RUN SEED</span><b>R0 ≠ R1</b><p>31 / 32 同格 trajectory 分叉</p></article>
|
||||
<i>+</i>
|
||||
<article><span>SAME FULL CONTRACT</span><b>R0 = R0′</b><p>新进程完整 token IDs exact</p></article>
|
||||
<i>=</i>
|
||||
<article class="result"><span>REPRODUCIBLE SAMPLING</span><b>64 / 64</b><p>随机轨迹不是环境漂移</p></article>
|
||||
</div>
|
||||
|
||||
<div class="seed-derivation">
|
||||
<header><span>SEED CONTRACT</span><b>结果之前冻结，不手选“好看 seed”</b></header>
|
||||
<div>
|
||||
<article><span>01 / BASE</span><b>SHA256(protocol / seed / index)</b><p>取前 4 bytes</p></article>
|
||||
<i>→</i>
|
||||
<article><span>02 / SOURCE</span><b>SHA256(protocol · base · source ID)</b><p>取前 8 bytes mod 2⁶³−1</p></article>
|
||||
<i>→</i>
|
||||
<article><span>03 / RNG</span><b>torch + CUDA manual seed</b><p>记录 pre/post state hash</p></article>
|
||||
<i>→</i>
|
||||
<article><span>04 / BATCH</span><b>固定八行顺序</b><p>顺序改变即新协议</p></article>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="repro-fields">
|
||||
{[
|
||||
["RUN SEED", "64 / 64"],
|
||||
["PROMPT HASH", "64 / 64"],
|
||||
["TOKEN IDS", "64 / 64"],
|
||||
["DECODED TEXT", "64 / 64"],
|
||||
["EOS STATE", "64 / 64"],
|
||||
["TRUNCATION", "64 / 64"],
|
||||
["CPU RNG PRE", "64 / 64"],
|
||||
["CUDA RNG PRE", "64 / 64"],
|
||||
].map(([label, value]) => (
|
||||
<article><span>{label}</span><b>{value}</b><i>EXACT</i></article>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="environment-lock">
|
||||
<article><span>CHECKPOINT</span><b>85864749…f64c7</b><p>12 个模型文件逐 SHA-256</p></article>
|
||||
<article><span>SOFTWARE</span><b>Torch 2.11 · TF 4.41.2</b><p>generation utils 也记录 hash</p></article>
|
||||
<article><span>PLACEMENT</span><b>CUDA L0–23 · CPU L24–26</b><p>final norm + LM head offload</p></article>
|
||||
<article><span>SCOPE</span><b>R0 / R1 only</b><p>不能扩大写成 256 / 256</p></article>
|
||||
</div>
|
||||
|
||||
<aside class="sp-warning">
|
||||
<b>exact replay 不是跨版本承诺</b>
|
||||
<p>
|
||||
更换 PyTorch、Transformers、CUDA kernel、硬件或 batch 行顺序都可能改变轨迹。
|
||||
这里证明的是固定执行合同内的可复现性。
|
||||
</p>
|
||||
</aside>
|
||||
</section>
|
||||
|
||||
<figcaption>
|
||||
<b>证据边界</b>
|
||||
<span>
|
||||
4 sources × 8 seeds 不是 benchmark，也没有恢复完整生成分布。BOS / x / 句点格不是
|
||||
官方有效聊天格式；unique hash 不是语义多样性；batch-seed aligned 不是逐行共享随机数。
|
||||
</span>
|
||||
<code>FORMAL 46c7edf…45af · EVAL 078f486…c4d8 · RERUN 72d050e…f6a0</code>
|
||||
</figcaption>
|
||||
|
||||
<script is:inline type="application/json" data-sp-data set:html={json}></script>
|
||||
</figure>
|
||||
|
||||
<script>
|
||||
document.querySelectorAll<HTMLElement>("[data-sampling-lab]").forEach((root) => {
|
||||
const payload = root.querySelector<HTMLScriptElement>("[data-sp-data]");
|
||||
if (!payload) return;
|
||||
const data = JSON.parse(payload.textContent ?? "{}");
|
||||
const one = <T extends Element>(selector: string) => (
|
||||
root.querySelector<T>(selector)
|
||||
);
|
||||
const set = (selector: string, value: string) => {
|
||||
const node = one<HTMLElement>(selector);
|
||||
if (node) node.textContent = value;
|
||||
};
|
||||
const pct = (value: number, digits = 1) => (
|
||||
`${(value * 100).toFixed(digits)}%`
|
||||
);
|
||||
const labels: Record<string, string> = {
|
||||
s0_eos: "S0 · EOS",
|
||||
s1_eos: "S1 · EOS",
|
||||
s0_bos: "S0 · BOS",
|
||||
s1_bos: "S1 · BOS",
|
||||
s0_x: "S0 · x",
|
||||
s1_x: "S1 · x",
|
||||
s0_period: "S0 · 句点",
|
||||
s1_period: "S1 · 句点",
|
||||
};
|
||||
const sourceById = new Map(
|
||||
data.sources.map((source: any) => [source.id, source]),
|
||||
);
|
||||
|
||||
const tabs = [
|
||||
...root.querySelectorAll<HTMLButtonElement>("[data-sp-tab]"),
|
||||
];
|
||||
const panels = [
|
||||
...root.querySelectorAll<HTMLElement>("[data-sp-panel]"),
|
||||
];
|
||||
tabs.forEach((button, index) => {
|
||||
button.addEventListener("click", () => {
|
||||
const target = button.dataset.spTab;
|
||||
tabs.forEach((candidate) => {
|
||||
const active = candidate === button;
|
||||
candidate.setAttribute("aria-selected", String(active));
|
||||
candidate.tabIndex = active ? 0 : -1;
|
||||
});
|
||||
panels.forEach((panel) => {
|
||||
panel.hidden = panel.dataset.spPanel !== target;
|
||||
});
|
||||
});
|
||||
button.addEventListener("keydown", (event) => {
|
||||
if (!["ArrowLeft", "ArrowRight"].includes(event.key)) return;
|
||||
event.preventDefault();
|
||||
const delta = event.key === "ArrowRight" ? 1 : -1;
|
||||
tabs[(index + delta + tabs.length) % tabs.length].click();
|
||||
tabs[(index + delta + tabs.length) % tabs.length].focus();
|
||||
});
|
||||
});
|
||||
|
||||
const sourceSelect = one<HTMLSelectElement>("[data-sp-source]");
|
||||
const conditionSelect = one<HTMLSelectElement>("[data-sp-condition]");
|
||||
const seedContainer = one<HTMLElement>("[data-sp-seeds]");
|
||||
const renderFocus = (sample: any, repeated: boolean) => {
|
||||
set(
|
||||
"[data-sp-focus-seed]",
|
||||
`${sample.replicate} · base ${sample.baseSeed.toLocaleString()}`,
|
||||
);
|
||||
set(
|
||||
"[data-sp-focus-meta]",
|
||||
`${sample.generatedTokens} TOKENS · ${sample.hitEos ? "NATURAL EOS" : "BUDGET TRUNCATED"}${repeated ? " · REPEATED HASH" : ""}`,
|
||||
);
|
||||
set(
|
||||
"[data-sp-focus-preview]",
|
||||
sample.preview || "（空文本）",
|
||||
);
|
||||
set(
|
||||
"[data-sp-focus-hash]",
|
||||
`trajectory sha256 · ${sample.trajectoryHash}`,
|
||||
);
|
||||
};
|
||||
const renderTrajectories = () => {
|
||||
const source = sourceById.get(sourceSelect?.value) as any;
|
||||
const condition = conditionSelect?.value ?? "s0_eos";
|
||||
if (!source) return;
|
||||
const row = source.conditions[condition];
|
||||
set("[data-sp-unique]", `${row.uniqueTrajectories} / 8`);
|
||||
set("[data-sp-eos]", `${row.naturalEos} / 8`);
|
||||
set("[data-sp-greedy]", row.greedyInSamples ? "YES" : "NO");
|
||||
set(
|
||||
"[data-sp-pairwise]",
|
||||
`${pct(row.pairwiseSimilarity.mean)} · ${pct(row.pairwiseSimilarity.min)}–${pct(row.pairwiseSimilarity.max)}`,
|
||||
);
|
||||
set(
|
||||
"[data-sp-stage-title]",
|
||||
`${source.id} · ${labels[condition]}`,
|
||||
);
|
||||
const counts = new Map<string, number>();
|
||||
row.trajectories.forEach((sample: any) => {
|
||||
counts.set(
|
||||
sample.trajectoryHash,
|
||||
(counts.get(sample.trajectoryHash) ?? 0) + 1,
|
||||
);
|
||||
});
|
||||
if (!seedContainer) return;
|
||||
const buttons = row.trajectories.map((sample: any, index: number) => {
|
||||
const button = document.createElement("button");
|
||||
button.type = "button";
|
||||
button.className = sample.hitEos ? "eos" : "truncated";
|
||||
const repeated = (counts.get(sample.trajectoryHash) ?? 0) > 1;
|
||||
if (repeated) button.classList.add("repeat");
|
||||
button.setAttribute(
|
||||
"aria-label",
|
||||
`${sample.replicate}, ${sample.generatedTokens} tokens`,
|
||||
);
|
||||
const label = document.createElement("span");
|
||||
label.textContent = sample.replicate;
|
||||
const chart = document.createElement("i");
|
||||
chart.style.setProperty(
|
||||
"--height",
|
||||
`${Math.max(4, sample.generatedTokens / 512 * 100)}%`,
|
||||
);
|
||||
const meta = document.createElement("b");
|
||||
meta.textContent = String(sample.generatedTokens);
|
||||
const hash = document.createElement("code");
|
||||
hash.textContent = sample.trajectoryHash.slice(0, 7);
|
||||
button.append(label, chart, meta, hash);
|
||||
button.addEventListener("click", () => {
|
||||
buttons.forEach((candidate: HTMLButtonElement) => (
|
||||
candidate.classList.toggle("selected", candidate === button)
|
||||
));
|
||||
renderFocus(sample, repeated);
|
||||
});
|
||||
if (index === 0) button.classList.add("selected");
|
||||
return button;
|
||||
});
|
||||
seedContainer.replaceChildren(...buttons);
|
||||
renderFocus(
|
||||
row.trajectories[0],
|
||||
(counts.get(row.trajectories[0].trajectoryHash) ?? 0) > 1,
|
||||
);
|
||||
};
|
||||
sourceSelect?.addEventListener("change", renderTrajectories);
|
||||
conditionSelect?.addEventListener("change", renderTrajectories);
|
||||
|
||||
const taskCondition = one<HTMLSelectElement>("[data-sp-task-condition]");
|
||||
const renderTaskCells = (
|
||||
domain: "math" | "code",
|
||||
containerSelector: string,
|
||||
totalSelector: string,
|
||||
) => {
|
||||
const source = data.sources.find(
|
||||
(candidate: any) => candidate.domain === domain,
|
||||
);
|
||||
const condition = taskCondition?.value ?? "s0_eos";
|
||||
const rows = source.conditions[condition].trajectories;
|
||||
const container = one<HTMLElement>(containerSelector);
|
||||
if (!container) return;
|
||||
let passing = 0;
|
||||
const cells = rows.map((sample: any) => {
|
||||
const article = document.createElement("article");
|
||||
const passed = domain === "math"
|
||||
? sample.math?.exact
|
||||
: sample.code?.passed;
|
||||
passing += Number(Boolean(passed));
|
||||
article.className = passed ? "pass" : "fail";
|
||||
const seed = document.createElement("span");
|
||||
seed.textContent = sample.replicate;
|
||||
const result = document.createElement("b");
|
||||
result.textContent = domain === "math"
|
||||
? `${sample.math?.predicted ?? "—"}${passed ? " ✓" : " ✕"}`
|
||||
: `${(sample.code?.status ?? "not run").replaceAll("_", " ")}${passed ? " ✓" : " ✕"}`;
|
||||
const meta = document.createElement("em");
|
||||
meta.textContent = `${sample.generatedTokens} TOKENS · ${sample.hitEos ? "EOS" : "CAP"}`;
|
||||
article.append(seed, result, meta);
|
||||
return article;
|
||||
});
|
||||
container.replaceChildren(...cells);
|
||||
set(totalSelector, `${passing} / ${rows.length} PASS`);
|
||||
};
|
||||
const renderTasks = () => {
|
||||
renderTaskCells(
|
||||
"math",
|
||||
"[data-sp-math-cells]",
|
||||
"[data-sp-math-total]",
|
||||
);
|
||||
renderTaskCells(
|
||||
"code",
|
||||
"[data-sp-code-cells]",
|
||||
"[data-sp-code-total]",
|
||||
);
|
||||
};
|
||||
taskCondition?.addEventListener("change", renderTasks);
|
||||
|
||||
const edgeSource = one<HTMLSelectElement>("[data-sp-edge-source]");
|
||||
const edgeSelect = one<HTMLSelectElement>("[data-sp-edge]");
|
||||
const renderSet = (
|
||||
samples: any[],
|
||||
otherHashes: Set<string>,
|
||||
selector: string,
|
||||
) => {
|
||||
const container = one<HTMLElement>(selector);
|
||||
if (!container) return;
|
||||
const cells = samples.map((sample) => {
|
||||
const article = document.createElement("article");
|
||||
article.className = otherHashes.has(sample.trajectoryHash)
|
||||
? "overlap"
|
||||
: "";
|
||||
const seed = document.createElement("span");
|
||||
seed.textContent = sample.replicate;
|
||||
const hash = document.createElement("b");
|
||||
hash.textContent = sample.trajectoryHash.slice(0, 8);
|
||||
const meta = document.createElement("em");
|
||||
meta.textContent = `${sample.generatedTokens}T · ${sample.hitEos ? "EOS" : "CAP"}`;
|
||||
article.append(seed, hash, meta);
|
||||
return article;
|
||||
});
|
||||
container.replaceChildren(...cells);
|
||||
};
|
||||
const renderEdge = () => {
|
||||
const source = sourceById.get(edgeSource?.value) as any;
|
||||
const edgeName = edgeSelect?.value ?? "system_eos";
|
||||
if (!source) return;
|
||||
const row = source.edges[edgeName];
|
||||
const left = source.conditions[row.left];
|
||||
const right = source.conditions[row.right];
|
||||
set(
|
||||
"[data-sp-edge-aligned]",
|
||||
pct(row.batch_seed_aligned_similarity.mean),
|
||||
);
|
||||
set(
|
||||
"[data-sp-edge-nearest]",
|
||||
pct(row.symmetric_mean_nearest_neighbor_similarity),
|
||||
);
|
||||
set(
|
||||
"[data-sp-edge-overlap]",
|
||||
`${row.generated_hash_set_intersection} / ${row.generated_hash_set_union}`,
|
||||
);
|
||||
const alignedBar = one<HTMLElement>("[data-sp-edge-aligned-bar]");
|
||||
const nearestBar = one<HTMLElement>("[data-sp-edge-nearest-bar]");
|
||||
if (alignedBar) alignedBar.style.width = pct(
|
||||
row.batch_seed_aligned_similarity.mean,
|
||||
);
|
||||
if (nearestBar) nearestBar.style.width = pct(
|
||||
row.symmetric_mean_nearest_neighbor_similarity,
|
||||
);
|
||||
set("[data-sp-edge-left-label]", labels[row.left]);
|
||||
set("[data-sp-edge-right-label]", labels[row.right]);
|
||||
renderSet(
|
||||
left.trajectories,
|
||||
new Set(right.trajectories.map(
|
||||
(sample: any) => sample.trajectoryHash,
|
||||
)),
|
||||
"[data-sp-edge-left-set]",
|
||||
);
|
||||
renderSet(
|
||||
right.trajectories,
|
||||
new Set(left.trajectories.map(
|
||||
(sample: any) => sample.trajectoryHash,
|
||||
)),
|
||||
"[data-sp-edge-right-set]",
|
||||
);
|
||||
};
|
||||
edgeSource?.addEventListener("change", renderEdge);
|
||||
edgeSelect?.addEventListener("change", renderEdge);
|
||||
|
||||
renderTrajectories();
|
||||
renderTasks();
|
||||
renderEdge();
|
||||
});
|
||||
</script>
|
||||
|
||||
<style>
|
||||
.sampling-lab {
|
||||
--ink: #182c33;
|
||||
--muted: #62706e;
|
||||
--line: rgba(25, 43, 50, .14);
|
||||
--paper: #f7f3e9;
|
||||
--teal: #2f7d70;
|
||||
--orange: #b35935;
|
||||
margin: 2.2rem 0 0;
|
||||
overflow: hidden;
|
||||
color: var(--ink);
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper);
|
||||
box-shadow: 0 28px 70px rgba(22, 35, 43, .1);
|
||||
}
|
||||
.sp-head {
|
||||
display: grid;
|
||||
grid-template-columns: 1.08fr .92fr;
|
||||
gap: 2.4rem;
|
||||
align-items: end;
|
||||
padding: 2rem;
|
||||
color: #fff;
|
||||
background:
|
||||
radial-gradient(circle at 78% 18%, rgba(212, 124, 73, .3), transparent 27%),
|
||||
radial-gradient(circle at 18% 90%, rgba(79, 165, 147, .23), transparent 30%),
|
||||
linear-gradient(137deg, #112a33, #28444a 64%, #3f4b49);
|
||||
}
|
||||
.sp-head p { margin: 0; color: rgba(255,255,255,.74); font-size: .76rem; line-height: 1.72; }
|
||||
.sp-head > div > p { color: #8ed1c5; font: 760 .6rem/1.2 var(--font-mono); letter-spacing: .11em; }
|
||||
.sp-head h3 { max-width: 680px; margin: .72rem 0 0; color: #fff; font: 760 clamp(1.55rem,3vw,2.45rem)/1.14 var(--font-display); }
|
||||
.sp-head code { padding: .08rem .25rem; color: #f3d3bd; background: rgba(0,0,0,.16); font: .64rem/1.2 var(--font-mono); }
|
||||
.sp-ledger { display: grid; grid-template-columns: repeat(6,1fr); border-bottom: 1px solid var(--line); }
|
||||
.sp-ledger article { min-width: 0; padding: 1rem; border-right: 1px solid var(--line); background: #ebe7dc; }
|
||||
.sp-ledger article:last-child { border-right: 0; }
|
||||
.sp-ledger article.pass { background: rgba(48,126,111,.12); }
|
||||
.sp-ledger span,
|
||||
.trajectory-summary span,
|
||||
.set-metrics span,
|
||||
.environment-lock span { display: block; color: #61706d; font: 730 .5rem/1.25 var(--font-mono); letter-spacing: .07em; }
|
||||
.sp-ledger b { display: block; margin-top: .45rem; font: 780 .82rem/1.2 var(--font-mono); }
|
||||
.sp-ledger p { margin: .34rem 0 0; color: #69736f; font-size: .56rem; line-height: 1.45; }
|
||||
.sp-tabs { display: grid; grid-template-columns: repeat(4,1fr); border-bottom: 1px solid var(--line); }
|
||||
.sp-tabs button { display: grid; grid-template-columns: auto 1fr; grid-template-rows: auto auto; column-gap: .72rem; min-width: 0; padding: .92rem 1rem; color: #263a40; border: 0; border-right: 1px solid var(--line); background: #fbf8f1; text-align: left; cursor: pointer; }
|
||||
.sp-tabs button:last-child { border-right: 0; }
|
||||
.sp-tabs button[aria-selected="true"] { color: #fff; background: var(--orange); }
|
||||
.sp-tabs span { grid-row: 1 / 3; opacity: .66; font: 720 .54rem/1.2 var(--font-mono); }
|
||||
.sp-tabs b { font: 730 .73rem/1.25 var(--font-display); }
|
||||
.sp-tabs small { opacity: .68; font: .51rem/1.3 var(--font-mono); }
|
||||
.sp-panel { padding: 1.55rem; }
|
||||
.sp-panel[hidden] { display: none; }
|
||||
.sp-panel-lead { display: grid; grid-template-columns: 1fr 1fr; gap: 2rem; align-items: end; margin-bottom: 1.2rem; }
|
||||
.sp-panel-lead span { color: var(--orange); font: 750 .55rem/1.2 var(--font-mono); letter-spacing: .09em; }
|
||||
.sp-panel-lead h4 { margin: .35rem 0 0; font: 750 1.24rem/1.2 var(--font-display); }
|
||||
.sp-panel-lead p { margin: 0; color: var(--muted); font-size: .68rem; line-height: 1.68; }
|
||||
.sp-controls { display: grid; grid-template-columns: 1fr 1fr .82fr; gap: 1px; border: 1px solid var(--line); background: var(--line); }
|
||||
.sp-controls > * { min-width: 0; padding: .76rem; background: #ece8de; }
|
||||
.sp-controls span,
|
||||
.task-condition-control span { display: block; margin-bottom: .42rem; color: #65716f; font: 720 .5rem/1.2 var(--font-mono); letter-spacing: .06em; }
|
||||
.sp-controls select,
|
||||
.task-condition-control select { width: 100%; min-width: 0; padding: .58rem; color: #24383e; border: 1px solid rgba(30,38,43,.2); background: #fffdf8; font: 670 .6rem/1.3 var(--font-mono); }
|
||||
.sampling-config { display: flex; flex-direction: column; justify-content: center; }
|
||||
.sampling-config b { color: #315d58; font: 720 .62rem/1.45 var(--font-mono); }
|
||||
.trajectory-summary { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
|
||||
.trajectory-summary article { padding: .8rem; background: #f0ece2; }
|
||||
.trajectory-summary b { display: block; margin-top: .38rem; color: #2b6d62; font: 780 .75rem/1.2 var(--font-mono); }
|
||||
.trajectory-summary p { margin: .35rem 0 0; color: #6c7571; font-size: .54rem; }
|
||||
.seed-stage { margin-top: 1rem; border: 1px solid var(--line); background: #eee9df; }
|
||||
.seed-stage > header { display: flex; justify-content: space-between; gap: 1rem; align-items: center; padding: .75rem .85rem; color: #e8f0ed; background: #29434a; }
|
||||
.seed-stage header span { display: block; color: #91c9c0; font: 700 .47rem/1.2 var(--font-mono); }
|
||||
.seed-stage header b { display: block; margin-top: .25rem; font: 690 .62rem/1.2 var(--font-mono); }
|
||||
.seed-stage header p { margin: 0; font: .5rem/1.4 var(--font-mono); }
|
||||
.seed-stage header p i { display: inline-block; width: .55rem; height: .55rem; margin: 0 .25rem 0 .6rem; vertical-align: -.08rem; background: #3f9b89; }
|
||||
.seed-stage header p i:first-child { margin-left: 0; }
|
||||
.seed-stage header p i.truncated { background: #b45a37; }
|
||||
.seed-stage header p i.repeat { background: #d4a44d; }
|
||||
.seed-bars { display: grid; grid-template-columns: repeat(8,1fr); gap: 1px; height: 13rem; background: var(--line); }
|
||||
.seed-bars button { position: relative; display: grid; grid-template-rows: auto 1fr auto auto; gap: .35rem; min-width: 0; padding: .62rem .48rem; color: #31464b; border: 0; background: #f7f3ea; cursor: pointer; }
|
||||
.seed-bars button.selected { background: #fffdf8; box-shadow: inset 0 0 0 2px var(--orange); }
|
||||
.seed-bars button > span { font: 760 .56rem/1 var(--font-mono); }
|
||||
.seed-bars button > i { position: relative; align-self: end; width: 100%; height: var(--height); max-height: 8.2rem; background: linear-gradient(to top, #2f7d70, #76b6aa); }
|
||||
.seed-bars button.truncated > i { background: linear-gradient(to top, #a94f31, #d68d63); }
|
||||
.seed-bars button.repeat > i::after { position: absolute; inset: .25rem; border: 1px dashed #fff6d9; content: ""; }
|
||||
.seed-bars button > b { font: 740 .6rem/1 var(--font-mono); }
|
||||
.seed-bars button > code { overflow: hidden; color: #77807c; font: .43rem/1 var(--font-mono); text-overflow: ellipsis; }
|
||||
.seed-focus { display: grid; grid-template-columns: .34fr 1fr; gap: 1rem; padding: .9rem; border-top: 1px solid var(--line); background: #fbf8f1; }
|
||||
.seed-focus span { color: var(--orange); font: 750 .53rem/1.2 var(--font-mono); }
|
||||
.seed-focus b { display: block; margin-top: .4rem; color: #345b57; font: 690 .55rem/1.45 var(--font-mono); }
|
||||
.seed-focus p { margin: 0; color: #53615f; font-size: .62rem; line-height: 1.55; }
|
||||
.seed-focus code { grid-column: 1 / -1; overflow-wrap: anywhere; color: #737d79; font: .46rem/1.4 var(--font-mono); }
|
||||
.condition-overview,
|
||||
.edge-overview { margin-top: 1rem; border: 1px solid var(--line); }
|
||||
.condition-overview > header,
|
||||
.condition-overview > div { display: grid; grid-template-columns: 1.25fr repeat(4,1fr); gap: .65rem; align-items: center; padding: .62rem .8rem; border-bottom: 1px solid rgba(30,38,43,.1); }
|
||||
.condition-overview > header { color: #e5edeb; background: #29434a; font: 680 .46rem/1.2 var(--font-mono); }
|
||||
.condition-overview > div:last-child { border-bottom: 0; }
|
||||
.condition-overview > div:nth-child(odd) { background: #ede9df; }
|
||||
.condition-overview span { font: 680 .59rem/1.2 var(--font-mono); }
|
||||
.condition-overview b { color: #5d6865; font: 670 .57rem/1.2 var(--font-mono); }
|
||||
.condition-overview b.good { color: #277563; }
|
||||
.condition-overview b.warn { color: #a95131; }
|
||||
.sp-warning { display: grid; grid-template-columns: .35fr 1fr; gap: 1rem; margin-top: 1rem; padding: .85rem; color: #5c4a3b; border: 1px solid rgba(177,91,50,.2); background: rgba(183,98,53,.09); }
|
||||
.sp-warning b { font: 730 .63rem/1.4 var(--font-display); }
|
||||
.sp-warning p { margin: 0; font-size: .61rem; line-height: 1.6; }
|
||||
.sp-warning.dark { color: #dfe9e6; border: 0; background: #29434a; }
|
||||
.task-condition-control { display: grid; grid-template-columns: 1fr 1fr; gap: 1px; border: 1px solid var(--line); background: var(--line); }
|
||||
.task-condition-control > * { min-width: 0; padding: .78rem; background: #ede9df; }
|
||||
.task-condition-control > div { display: flex; flex-direction: column; justify-content: center; }
|
||||
.task-condition-control > div b { color: #326a61; font: 740 .69rem/1.2 var(--font-mono); }
|
||||
.sample-task-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
|
||||
.sample-task-grid > article { min-width: 0; padding: .95rem; background: #efebe1; }
|
||||
.sample-task-grid > article > header { display: flex; justify-content: space-between; align-items: end; gap: .8rem; }
|
||||
.sample-task-grid header span { display: block; color: #6a7571; font: 680 .49rem/1.2 var(--font-mono); }
|
||||
.sample-task-grid header b { display: block; margin-top: .28rem; color: #326d62; font: 750 .72rem/1.2 var(--font-mono); }
|
||||
.sample-task-grid header em { color: #8c5a43; font: 680 .52rem/1.2 var(--font-mono); }
|
||||
.seed-task-cells { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: .72rem; background: var(--line); }
|
||||
.seed-task-cells article { min-width: 0; padding: .62rem; background: #fffaf2; box-shadow: inset 0 .22rem #ac5434; }
|
||||
.seed-task-cells article.pass { box-shadow: inset 0 .22rem #358274; }
|
||||
.seed-task-cells span,
|
||||
.seed-task-cells b,
|
||||
.seed-task-cells em { display: block; overflow-wrap: anywhere; }
|
||||
.seed-task-cells span { color: #727b77; font: .46rem/1.2 var(--font-mono); }
|
||||
.seed-task-cells b { margin-top: .32rem; font: 680 .55rem/1.25 var(--font-mono); }
|
||||
.seed-task-cells em { margin-top: .3rem; color: #77817d; font: .41rem/1.35 var(--font-mono); }
|
||||
.failure-cases { display: grid; grid-template-columns: 1fr 1fr; gap: 1px; margin-top: 1rem; border: 1px solid rgba(174,82,47,.19); background: rgba(174,82,47,.19); }
|
||||
.failure-cases > article { min-width: 0; padding: 1rem; background: #f4e9df; }
|
||||
.failure-cases > article > span { color: #a04d31; font: 740 .49rem/1.2 var(--font-mono); letter-spacing: .08em; }
|
||||
.failure-cases h5 { margin: .45rem 0 .75rem; font: 730 .86rem/1.3 var(--font-display); }
|
||||
.failure-cases p { margin: .7rem 0 0; color: #665f58; font-size: .6rem; line-height: 1.55; }
|
||||
.failure-cases p code { font: .52rem/1.2 var(--font-mono); }
|
||||
.reasoning-bug { display: grid; grid-template-columns: 1fr auto 1fr; gap: .5rem; align-items: center; }
|
||||
.reasoning-bug b { padding: .62rem; color: #6a3d2e; background: rgba(174,82,47,.1); font: 680 .57rem/1.35 var(--font-mono); }
|
||||
.reasoning-bug i { color: #a74f31; font: 680 .45rem/1.2 var(--font-mono); }
|
||||
.failure-cases pre { overflow: auto; margin: 0; padding: .7rem; color: #dfe9e6; background: #293f45; font: .52rem/1.55 var(--font-mono); }
|
||||
.four-ledgers { display: grid; grid-template-columns: 1fr auto 1fr auto 1fr auto 1fr; gap: .45rem; align-items: center; margin-top: 1rem; }
|
||||
.four-ledgers article { padding: .75rem; border: 1px solid var(--line); background: #eeeae0; }
|
||||
.four-ledgers article.done { background: rgba(48,126,111,.11); }
|
||||
.four-ledgers article > span { color: var(--orange); font: 730 .47rem/1 var(--font-mono); }
|
||||
.four-ledgers article > b { display: block; margin-top: .35rem; font: 720 .58rem/1.2 var(--font-mono); }
|
||||
.four-ledgers article > p { margin: .3rem 0 0; color: #6a7470; font-size: .52rem; }
|
||||
.four-ledgers > i { color: var(--orange); font: 750 .7rem/1 var(--font-mono); }
|
||||
.set-metrics { display: grid; grid-template-columns: repeat(3,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
|
||||
.set-metrics article { padding: .85rem; background: #eeeae0; }
|
||||
.set-metrics b { display: block; margin-top: .38rem; color: #2e6d62; font: 760 .76rem/1.2 var(--font-mono); }
|
||||
.set-metrics article > i { display: block; height: .38rem; margin-top: .6rem; background: rgba(47,125,112,.16); }
|
||||
.set-metrics article > i > u { display: block; width: 0; height: 100%; background: var(--teal); text-decoration: none; transition: width .25s ease; }
|
||||
.set-metrics p { margin: .45rem 0 0; color: #6c7672; font-size: .53rem; }
|
||||
.two-sets { display: grid; grid-template-columns: 1fr 5.5rem 1fr; gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
|
||||
.two-sets > article { min-width: 0; padding: .85rem; background: #efebe1; }
|
||||
.two-sets > article > header { display: flex; justify-content: space-between; gap: .7rem; }
|
||||
.two-sets header span { color: #717b77; font: .47rem/1.2 var(--font-mono); }
|
||||
.two-sets header b { color: #3a645e; font: 690 .54rem/1.2 var(--font-mono); }
|
||||
.two-sets article > div { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: .68rem; background: var(--line); }
|
||||
.two-sets article > div article { min-width: 0; padding: .55rem; background: #fffaf2; }
|
||||
.two-sets article > div article.overlap { background: #d9eee7; box-shadow: inset 0 0 0 1px var(--teal); }
|
||||
.two-sets article > div span,
|
||||
.two-sets article > div b,
|
||||
.two-sets article > div em { display: block; overflow: hidden; text-overflow: ellipsis; }
|
||||
.two-sets article > div span { color: #77817d; font: .43rem/1.2 var(--font-mono); }
|
||||
.two-sets article > div b { margin-top: .3rem; font: 680 .49rem/1.2 var(--font-mono); }
|
||||
.two-sets article > div em { margin-top: .25rem; color: #7b817e; font: .39rem/1.2 var(--font-mono); }
|
||||
.set-bridge { display: grid; place-content: center; gap: .3rem; color: #e6eeeb; background: #29434a; text-align: center; }
|
||||
.set-bridge b { font: 760 .72rem/1 var(--font-mono); }
|
||||
.set-bridge span { font: .42rem/1.3 var(--font-mono); }
|
||||
.set-bridge i { display: block; height: 1px; background: #8dc8bd; }
|
||||
.edge-overview > header,
|
||||
.edge-overview > div { position: relative; display: grid; grid-template-columns: 1.6fr repeat(3,1fr); gap: .65rem; align-items: center; padding: .62rem .8rem; overflow: hidden; border-bottom: 1px solid rgba(30,38,43,.1); }
|
||||
.edge-overview > header { color: #e5edeb; background: #29434a; font: 680 .46rem/1.2 var(--font-mono); }
|
||||
.edge-overview > div:last-child { border-bottom: 0; }
|
||||
.edge-overview > div:nth-child(odd) { background: #ede9df; }
|
||||
.edge-overview span { z-index: 1; font: 650 .56rem/1.2 var(--font-mono); }
|
||||
.edge-overview b { z-index: 1; color: #5d6965; font: 670 .55rem/1.2 var(--font-mono); }
|
||||
.edge-overview > div > i { position: absolute; right: 0; bottom: 0; width: var(--edge-nearest); height: .16rem; background: var(--teal); opacity: .7; }
|
||||
.randomness-equation { display: grid; grid-template-columns: 1fr auto 1fr auto 1fr; gap: .65rem; align-items: center; }
|
||||
.randomness-equation article { padding: 1rem; border: 1px solid var(--line); background: #eeeae0; }
|
||||
.randomness-equation article.result { color: #e9f1ef; border: 0; background: #2d776c; }
|
||||
.randomness-equation span { display: block; opacity: .72; font: .48rem/1.2 var(--font-mono); }
|
||||
.randomness-equation b { display: block; margin-top: .45rem; font: 780 .95rem/1.2 var(--font-mono); }
|
||||
.randomness-equation p { margin: .4rem 0 0; opacity: .75; font-size: .55rem; }
|
||||
.randomness-equation > i { color: var(--orange); font: 780 1rem/1 var(--font-display); }
|
||||
.seed-derivation { margin-top: 1rem; border: 1px solid var(--line); }
|
||||
.seed-derivation > header { display: flex; justify-content: space-between; gap: 1rem; padding: .72rem .85rem; color: #e6eeeb; background: #29434a; }
|
||||
.seed-derivation > header span { color: #8fc9bd; font: .47rem/1.2 var(--font-mono); }
|
||||
.seed-derivation > header b { font: 670 .56rem/1.2 var(--font-mono); }
|
||||
.seed-derivation > div { display: grid; grid-template-columns: 1fr auto 1fr auto 1fr auto 1fr; gap: .45rem; align-items: center; padding: .9rem; background: #eeeae0; }
|
||||
.seed-derivation article { min-width: 0; padding: .65rem; border: 1px solid var(--line); background: #faf7ef; }
|
||||
.seed-derivation article span { color: var(--orange); font: .44rem/1.2 var(--font-mono); }
|
||||
.seed-derivation article b { display: block; margin-top: .35rem; overflow-wrap: anywhere; font: 650 .49rem/1.35 var(--font-mono); }
|
||||
.seed-derivation article p { margin: .32rem 0 0; color: #727b77; font-size: .47rem; }
|
||||
.seed-derivation > div > i { color: var(--orange); font: 720 .65rem/1 var(--font-mono); }
|
||||
.repro-fields { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
|
||||
.repro-fields article { padding: .72rem; background: rgba(48,126,111,.1); }
|
||||
.repro-fields span { color: #5a716c; font: .46rem/1.2 var(--font-mono); }
|
||||
.repro-fields b { display: block; margin-top: .35rem; color: #296a5f; font: 740 .62rem/1.2 var(--font-mono); }
|
||||
.repro-fields i { display: block; margin-top: .28rem; color: #528078; font: .4rem/1.2 var(--font-mono); }
|
||||
.environment-lock { display: grid; grid-template-columns: repeat(4,1fr); gap: 1px; margin-top: 1rem; border: 1px solid var(--line); background: var(--line); }
|
||||
.environment-lock article { min-width: 0; padding: .78rem; background: #eeeae0; }
|
||||
.environment-lock b { display: block; margin-top: .38rem; overflow-wrap: anywhere; font: 690 .57rem/1.3 var(--font-mono); }
|
||||
.environment-lock p { margin: .35rem 0 0; color: #6c7571; font-size: .51rem; line-height: 1.45; }
|
||||
.sampling-lab > figcaption { display: grid; grid-template-columns: auto 1fr auto; gap: 1rem; align-items: center; padding: .9rem 1.1rem; color: #dfe8e6; background: #203a42; }
|
||||
.sampling-lab > figcaption b { color: #8dc8bd; font: 730 .53rem/1.2 var(--font-mono); }
|
||||
.sampling-lab > figcaption span { font-size: .54rem; line-height: 1.5; }
|
||||
.sampling-lab > figcaption code { color: #d9af95; font: .42rem/1.4 var(--font-mono); }
|
||||
@media (max-width: 900px) {
|
||||
.sp-head,
|
||||
.sp-panel-lead { grid-template-columns: 1fr; }
|
||||
.sp-ledger { grid-template-columns: repeat(3,1fr); }
|
||||
.sp-ledger article:nth-child(3) { border-right: 0; }
|
||||
.sp-tabs { grid-template-columns: 1fr 1fr; }
|
||||
.sp-controls,
|
||||
.sample-task-grid,
|
||||
.failure-cases { grid-template-columns: 1fr; }
|
||||
.two-sets { grid-template-columns: 1fr; }
|
||||
.set-bridge { min-height: 4rem; }
|
||||
.sampling-lab > figcaption { grid-template-columns: 1fr; }
|
||||
}
|
||||
@media (max-width: 640px) {
|
||||
.sp-panel { padding: 1rem; }
|
||||
.sp-head { padding: 1.35rem; }
|
||||
.sp-ledger,
|
||||
.trajectory-summary,
|
||||
.set-metrics,
|
||||
.repro-fields,
|
||||
.environment-lock { grid-template-columns: 1fr 1fr; }
|
||||
.sp-tabs { grid-template-columns: 1fr; }
|
||||
.seed-bars { grid-template-columns: repeat(4,1fr); height: 22rem; }
|
||||
.seed-focus { grid-template-columns: 1fr; }
|
||||
.condition-overview > header,
|
||||
.condition-overview > div {
|
||||
grid-template-columns: 1.2fr repeat(4,.72fr);
|
||||
gap: .25rem;
|
||||
min-width: 0;
|
||||
padding: .5rem .35rem;
|
||||
}
|
||||
.condition-overview > header b,
|
||||
.condition-overview > div b,
|
||||
.condition-overview > div span { font-size: .42rem; }
|
||||
.seed-task-cells { grid-template-columns: 1fr 1fr; }
|
||||
.four-ledgers,
|
||||
.randomness-equation,
|
||||
.seed-derivation > div { grid-template-columns: 1fr; }
|
||||
.four-ledgers > i,
|
||||
.randomness-equation > i,
|
||||
.seed-derivation > div > i { transform: rotate(90deg); text-align: center; }
|
||||
.edge-overview > header,
|
||||
.edge-overview > div {
|
||||
grid-template-columns: 1.45fr repeat(3,.65fr);
|
||||
gap: .25rem;
|
||||
min-width: 0;
|
||||
padding: .5rem .35rem;
|
||||
}
|
||||
.edge-overview > header b,
|
||||
.edge-overview > div b,
|
||||
.edge-overview > div span { font-size: .4rem; }
|
||||
}
|
||||
</style>
|
||||
|
||||
<style is:global>
|
||||
[data-sampling-lab] [data-sp-seeds] button {
|
||||
position: relative;
|
||||
display: grid;
|
||||
grid-template-rows: auto minmax(4rem, 1fr) auto auto;
|
||||
gap: .35rem;
|
||||
min-width: 0;
|
||||
padding: .62rem .48rem;
|
||||
overflow: hidden;
|
||||
color: #31464b;
|
||||
border: 0;
|
||||
background: #f7f3ea;
|
||||
cursor: pointer;
|
||||
}
|
||||
[data-sampling-lab] [data-sp-seeds] button.selected {
|
||||
background: #fffdf8;
|
||||
box-shadow: inset 0 0 0 2px #b35935;
|
||||
}
|
||||
[data-sampling-lab] [data-sp-seeds] button > span {
|
||||
font: 760 .56rem/1 var(--font-mono);
|
||||
}
|
||||
[data-sampling-lab] [data-sp-seeds] button > i {
|
||||
position: relative;
|
||||
display: block;
|
||||
align-self: end;
|
||||
width: 100%;
|
||||
height: var(--height);
|
||||
max-height: 8.2rem;
|
||||
background: linear-gradient(to top, #2f7d70, #76b6aa);
|
||||
}
|
||||
[data-sampling-lab] [data-sp-seeds] button.truncated > i {
|
||||
background: linear-gradient(to top, #a94f31, #d68d63);
|
||||
}
|
||||
[data-sampling-lab] [data-sp-seeds] button.repeat > i::after {
|
||||
position: absolute;
|
||||
inset: .25rem;
|
||||
border: 1px dashed #fff6d9;
|
||||
content: "";
|
||||
}
|
||||
[data-sampling-lab] [data-sp-seeds] button > b {
|
||||
display: block;
|
||||
font: 740 .6rem/1 var(--font-mono);
|
||||
}
|
||||
[data-sampling-lab] [data-sp-seeds] button > code {
|
||||
display: block;
|
||||
overflow: hidden;
|
||||
color: #77807c;
|
||||
font: .43rem/1 var(--font-mono);
|
||||
text-overflow: ellipsis;
|
||||
}
|
||||
[data-sampling-lab] [data-sp-math-cells] article,
|
||||
[data-sampling-lab] [data-sp-code-cells] article {
|
||||
min-width: 0;
|
||||
padding: .62rem;
|
||||
background: #fffaf2;
|
||||
box-shadow: inset 0 .22rem #ac5434;
|
||||
}
|
||||
[data-sampling-lab] [data-sp-math-cells] article.pass,
|
||||
[data-sampling-lab] [data-sp-code-cells] article.pass {
|
||||
box-shadow: inset 0 .22rem #358274;
|
||||
}
|
||||
[data-sampling-lab] [data-sp-math-cells] article > span,
|
||||
[data-sampling-lab] [data-sp-math-cells] article > b,
|
||||
[data-sampling-lab] [data-sp-math-cells] article > em,
|
||||
[data-sampling-lab] [data-sp-code-cells] article > span,
|
||||
[data-sampling-lab] [data-sp-code-cells] article > b,
|
||||
[data-sampling-lab] [data-sp-code-cells] article > em {
|
||||
display: block;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
[data-sampling-lab] [data-sp-math-cells] article > span,
|
||||
[data-sampling-lab] [data-sp-code-cells] article > span {
|
||||
color: #727b77;
|
||||
font: .46rem/1.2 var(--font-mono);
|
||||
}
|
||||
[data-sampling-lab] [data-sp-math-cells] article > b,
|
||||
[data-sampling-lab] [data-sp-code-cells] article > b {
|
||||
margin-top: .32rem;
|
||||
font: 680 .55rem/1.25 var(--font-mono);
|
||||
}
|
||||
[data-sampling-lab] [data-sp-math-cells] article > em,
|
||||
[data-sampling-lab] [data-sp-code-cells] article > em {
|
||||
margin-top: .3rem;
|
||||
color: #77817d;
|
||||
font: .41rem/1.35 var(--font-mono);
|
||||
}
|
||||
[data-sampling-lab] [data-sp-edge-left-set] article,
|
||||
[data-sampling-lab] [data-sp-edge-right-set] article {
|
||||
min-width: 0;
|
||||
padding: .55rem;
|
||||
background: #fffaf2;
|
||||
}
|
||||
[data-sampling-lab] [data-sp-edge-left-set] article.overlap,
|
||||
[data-sampling-lab] [data-sp-edge-right-set] article.overlap {
|
||||
background: #d9eee7;
|
||||
box-shadow: inset 0 0 0 1px #2f7d70;
|
||||
}
|
||||
[data-sampling-lab] [data-sp-edge-left-set] article > span,
|
||||
[data-sampling-lab] [data-sp-edge-left-set] article > b,
|
||||
[data-sampling-lab] [data-sp-edge-left-set] article > em,
|
||||
[data-sampling-lab] [data-sp-edge-right-set] article > span,
|
||||
[data-sampling-lab] [data-sp-edge-right-set] article > b,
|
||||
[data-sampling-lab] [data-sp-edge-right-set] article > em {
|
||||
display: block;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
}
|
||||
[data-sampling-lab] [data-sp-edge-left-set] article > span,
|
||||
[data-sampling-lab] [data-sp-edge-right-set] article > span {
|
||||
color: #77817d;
|
||||
font: .43rem/1.2 var(--font-mono);
|
||||
}
|
||||
[data-sampling-lab] [data-sp-edge-left-set] article > b,
|
||||
[data-sampling-lab] [data-sp-edge-right-set] article > b {
|
||||
margin-top: .3rem;
|
||||
font: 680 .49rem/1.2 var(--font-mono);
|
||||
}
|
||||
[data-sampling-lab] [data-sp-edge-left-set] article > em,
|
||||
[data-sampling-lab] [data-sp-edge-right-set] article > em {
|
||||
margin-top: .25rem;
|
||||
color: #7b817e;
|
||||
font: .39rem/1.2 var(--font-mono);
|
||||
}
|
||||
</style>
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,946 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"protocol_id": "llm-atlas-deepseek-chat-sampling-v1",
|
||||
"formal": {
|
||||
"path": "/tmp/deepseek-v2-lite-chat-sampling-formal.json",
|
||||
"sha256": "46c7edfce1409e798d6b0f06e905dd3a9d4ab6911acd8e42e2ec9d06d65345af",
|
||||
"content_hash": "7ed89b59e852136892e56da434b212e80a1f26f771e63bd6f92eeb8617a18ad5",
|
||||
"base_seeds": [
|
||||
19683830,
|
||||
1560062173,
|
||||
3978401375,
|
||||
1280933274,
|
||||
1467459869,
|
||||
1297359489,
|
||||
2722953988,
|
||||
3330978061
|
||||
]
|
||||
},
|
||||
"rerun": {
|
||||
"path": "/tmp/deepseek-v2-lite-chat-sampling-repro-r0r1.json",
|
||||
"sha256": "72d050eace531bb38fb2592e39dc7f1b9c9003306c92c568dcd4acaaa691f6a0",
|
||||
"content_hash": "f8a5fb017ab0e517b1df2899e388c47d07b4bd910912639df542703d1d9509e0",
|
||||
"base_seeds": [
|
||||
19683830,
|
||||
1560062173
|
||||
]
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "HumanEval/31",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "gsm8k/test/1069",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "tnews/test/4855",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s0_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 19683830,
|
||||
"condition": "s1_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s0_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_bos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_eos",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_period",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
},
|
||||
{
|
||||
"source_id": "wikitext2/raw-validation/0443",
|
||||
"base_seed": 1560062173,
|
||||
"condition": "s1_x",
|
||||
"run_seed_exact": true,
|
||||
"prompt_hash_exact": true,
|
||||
"generated_token_ids_exact": true,
|
||||
"decoded_text_exact": true,
|
||||
"eos_state_exact": true,
|
||||
"truncation_state_exact": true,
|
||||
"cpu_rng_pre_state_exact": true,
|
||||
"cuda_rng_pre_state_exact": true,
|
||||
"all_preregistered_fields_exact": true
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"cells": 64,
|
||||
"all_preregistered_fields_exact": 64,
|
||||
"by_field": {
|
||||
"run_seed_exact": 64,
|
||||
"prompt_hash_exact": 64,
|
||||
"generated_token_ids_exact": 64,
|
||||
"decoded_text_exact": 64,
|
||||
"eos_state_exact": 64,
|
||||
"truncation_state_exact": 64,
|
||||
"cpu_rng_pre_state_exact": 64,
|
||||
"cuda_rng_pre_state_exact": 64
|
||||
}
|
||||
},
|
||||
"claim_boundary": [
|
||||
"Only the rerun seed subset is independently reproduced.",
|
||||
"Exact replay is scoped to the pinned software and hardware contract.",
|
||||
"Reproduction does not imply trajectories are seed-invariant."
|
||||
],
|
||||
"content_hash": "d02d0a23efdea333fb0b04fc9a889c7f81eb755f8ef4a9c66f14226a230d94a3"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -5,6 +5,7 @@ import DeepSeekLab from "@/components/DeepSeekLab.astro";
|
||||
import DeepSeekArtifactLab from "@/components/DeepSeekArtifactLab.astro";
|
||||
import DeepSeekBehaviorLab from "@/components/DeepSeekBehaviorLab.astro";
|
||||
import DeepSeekCompletionDepthLab from "@/components/DeepSeekCompletionDepthLab.astro";
|
||||
import DeepSeekSamplingLab from "@/components/DeepSeekSamplingLab.astro";
|
||||
import { deepseekBranches, deepseekLedgers, deepseekPaperChain, deepseekWaves } from "@/data/deepseek";
|
||||
|
||||
const toc = [
|
||||
@@ -33,21 +34,22 @@ const toc = [
|
||||
["22", "artifact", "真实权重执行"],
|
||||
["23", "behavior", "Chat：最终生成行为"],
|
||||
["24", "completion-depth", "Chat：完成度与全深度"],
|
||||
["25", "branches", "别漏掉旁支"],
|
||||
["26", "audit", "事实、推导与教学模型"],
|
||||
["25", "sampling", "Chat：多种子采样稳健性"],
|
||||
["26", "branches", "别漏掉旁支"],
|
||||
["27", "audit", "事实、推导与教学模型"],
|
||||
["↳", "papers", "六十节点阅读链"],
|
||||
];
|
||||
---
|
||||
|
||||
<BaseLayout
|
||||
title="DeepSeek 技术谱系与真实权重深读：从 Dense、MoE、MLA 到 R1 与 V4"
|
||||
description="用二十四张问题账、十次技术转向、十九个交互实验、真实 V2-Lite Base / Chat 权重、512-token 完成度评测、29 阶段隐藏状态与 26 层 MoE 路由追踪、吸收式缓存 trace 和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
||||
description="用二十四张问题账、十次技术转向、二十个交互实验、真实 V2-Lite Base / Chat 权重、512-token 完成度评测、29 阶段隐藏状态、26 层 MoE 路由追踪与 256 条多种子采样，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
||||
section="deepseek"
|
||||
>
|
||||
<header class="page-hero deepseek-hero">
|
||||
<div class="page-hero-inner">
|
||||
<div>
|
||||
<p class="eyebrow"><span>SPOTLIGHT / DEEPSEEK · ROUND 05</span> COMPLETION × FULL DEPTH × REAL WEIGHTS</p>
|
||||
<p class="eyebrow"><span>SPOTLIGHT / DEEPSEEK · ROUND 06</span> SAMPLING × COMPLETION × FULL DEPTH</p>
|
||||
<h1>不要背模型名<br />要看懂每次为什么转向</h1>
|
||||
<p class="lead">
|
||||
这不是七篇报告的摘要，而是一套可追问、可计算、可反驳的技术谱系：
|
||||
@@ -59,9 +61,9 @@ const toc = [
|
||||
<div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
|
||||
<div><dt>LEDGERS</dt><dd>24 张问题账</dd></div>
|
||||
<div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div>
|
||||
<div><dt>LABS</dt><dd>19 个可操作实验</dd></div>
|
||||
<div><dt>LABS</dt><dd>20 个可操作实验</dd></div>
|
||||
<div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div>
|
||||
<div><dt>STATUS</dt><dd>五轮 · 全 27 层执行</dd></div>
|
||||
<div><dt>STATUS</dt><dd>六轮 · 256 条采样</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</header>
|
||||
@@ -810,8 +812,21 @@ const toc = [
|
||||
<DeepSeekCompletionDepthLab />
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="sampling">
|
||||
<p class="eyebrow"><span>25</span> GREEDY IS NOT A DISTRIBUTION</p>
|
||||
<h2>把单条最大概率路径打开：八个预注册 seed 下，轨迹集合是否仍然分叉</h2>
|
||||
<p class="lede">
|
||||
上一轮的 512-token 结果仍是 deterministic greedy：它只能看到每一步概率最大的
|
||||
一条路。这一轮冻结四条 source、八格 prompt batch 与八个 SHA-256 派生 seed，
|
||||
启用 checkpoint 随附的 <code>temperature=.3 / top_p=.95</code>，生成 256 条
|
||||
sampled outputs。停止、任务正确、集合相似度与新进程复现继续分账；
|
||||
batch-seed aligned 也明确不冒充逐行共享随机数的 paired causal design。
|
||||
</p>
|
||||
<DeepSeekSamplingLab />
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="branches">
|
||||
<p class="eyebrow"><span>25</span> THE MAIN LINE IS NOT THE WHOLE TREE</p>
|
||||
<p class="eyebrow"><span>26</span> THE MAIN LINE IS NOT THE WHOLE TREE</p>
|
||||
<h2>如果只读 V2 → V3 → R1 → V4，会漏掉五条反过来影响主线的旁支</h2>
|
||||
<div class="branch-grid">
|
||||
{deepseekBranches.map(([name, line, text, url]) => (
|
||||
@@ -831,7 +846,7 @@ const toc = [
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="audit">
|
||||
<p class="eyebrow"><span>26</span> EVIDENCE AUDIT</p>
|
||||
<p class="eyebrow"><span>27</span> EVIDENCE AUDIT</p>
|
||||
<h2>同一张页面里有三种知识，它们的语气必须不同</h2>
|
||||
<div class="audit-grid">
|
||||
<article class="reported">
|
||||
|
||||
@@ -145,18 +145,18 @@ const paths = [
|
||||
</a>
|
||||
<a class="release-card deepseek-release" href="/deepseek/">
|
||||
<div>
|
||||
<p class="eyebrow"><span>NEW / DEEPSEEK ROUND 05</span> COMPLETION · TASK TESTS · FULL DEPTH</p>
|
||||
<p class="eyebrow"><span>NEW / DEEPSEEK ROUND 06</span> MULTI-SEED SAMPLING · EXACT REPLAY</p>
|
||||
<h2>从 Dense 到百万上下文：每次创新都在偿还上一代最贵的一张账</h2>
|
||||
<p>
|
||||
用二十四张问题账和十次技术转向走完 Dense→V4，再把官方 V2-Lite-Chat 的
|
||||
128 个输出统一延长到 512-token：121 个自然 EOS、Math / Code 真实 evaluator；
|
||||
同时沿 29 个隐藏阶段与 26 个 MoE gate 追踪 1,918,176 次目标路由决定。
|
||||
在 512-token greedy 与 29-stage / 26-gate 全深度 trace 之后，再按预注册的
|
||||
8 个 seed 生成 256 条官方 nucleus samples：251 条自然 EOS、242 条不同完整
|
||||
token 轨迹；Math 62 / 64、Code 63 / 64，并在新进程复现 R0/R1 的 64 / 64 格。
|
||||
</p>
|
||||
</div>
|
||||
<dl>
|
||||
<div><dt>LINEAGE</dt><dd>1991 → 2026 · 10 次转向</dd></div>
|
||||
<div><dt>NODES</dt><dd>60 个一手 / 官方节点</dd></div>
|
||||
<div><dt>LAB</dt><dd>19 · Base / Chat / full depth</dd></div>
|
||||
<div><dt>LAB</dt><dd>20 · Base / Chat / sampling</dd></div>
|
||||
</dl>
|
||||
<span class="release-arrow" aria-hidden="true">进入 DeepSeek 完整技术谱系 →</span>
|
||||
</a>
|
||||
|
||||
@@ -15,7 +15,7 @@ const workstreams = [
|
||||
{ label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
|
||||
{ label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" },
|
||||
{ label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM，加入真实去重与 mixture traces" },
|
||||
{ label: "DeepSeek 专题", value: 99, next: "扩大 completion/task 样本、sampling robustness 与干预式 mediation，再推进 SM90 FlashMLA、FP8/pipeline 与 R1-like RL" },
|
||||
{ label: "DeepSeek 专题", value: 99, next: "扩大 sampling 的 source/task 覆盖并推进干预式 mediation、SM90 FlashMLA、FP8/pipeline 与 R1-like RL" },
|
||||
{ label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
|
||||
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
|
||||
{ label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
|
||||
@@ -50,7 +50,7 @@ const workstreams = [
|
||||
<div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
|
||||
<div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
|
||||
<div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-29 19:55 CST</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-30 01:50 CST</dd></div>
|
||||
<div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
@@ -97,12 +97,12 @@ const workstreams = [
|
||||
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
||||
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
||||
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
||||
<article><span>✓</span><h3>八十六个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验、十三联 Base 工件实验、Chat 行为与 completion/full-depth 两轮实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>八十七个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验、十三联 Base 工件实验、Chat 行为、completion/full-depth 与 multi-seed sampling 三轮实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
||||
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
||||
<article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
|
||||
<article><span>✓</span><h3>DeepSeek 五轮真实权重里程碑</h3><p>统一 512-token 预算让自然 EOS 从 31 / 128 增至 121 / 128；GSM8K strict exact 23 / 32、HumanEval 官方 tests pass 24 / 32，并保留四任务/域的样本边界。相同 Chat checkpoint 再执行 29-stage hidden 与 26-gate route trace；1,856 个 hidden hashes、1,664 个 route hashes 和 1,664 个 weight hashes 在新进程子集复跑中全部 exact。</p></article>
|
||||
<article><span>✓</span><h3>DeepSeek 六轮真实权重里程碑</h3><p>在 512-token greedy 与 29-stage / 26-gate 全深度 trace 之后，预注册 8 个 SHA-256 seed，生成 256 条 official nucleus samples：251 条 natural EOS、242 个 unique trajectory hashes；GSM8K 62 / 64 strict exact、HumanEval 63 / 64 tests pass，新进程 R0/R1 的八项合同字段 64 / 64 exact。</p></article>
|
||||
<article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
|
||||
<article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
|
||||
<article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article>
|
||||
@@ -134,7 +134,7 @@ const workstreams = [
|
||||
<div class="queue-table">
|
||||
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
|
||||
<div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div>
|
||||
<div><span>P0</span><strong>DeepSeek 五轮后续</strong><p>扩大任务与语言 source → sampling robustness → 干预式 mediation → SM90 FlashMLA / FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
||||
<div><span>P0</span><strong>DeepSeek 六轮后续</strong><p>扩大任务与语言 source 的 sampling 覆盖 → 干预式 mediation → SM90 FlashMLA / FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
||||
<div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div>
|
||||
<div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div>
|
||||
<div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div>
|
||||
|
||||
Reference in New Issue
Block a user