feat: map DeepSeek Chat sampling robustness

This commit is contained in:
wuyang
2026-07-30 02:01:50 +08:00
parent 18b16e2fdc
commit 580f69675c
21 changed files with 108346 additions and 28 deletions
+82
View File
@@ -672,3 +672,85 @@ See `research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_PROTOCOL.md` and
`research/DEEPSEEK_V2_LITE_CHAT_COMPLETION_DEPTH_AUDIT.md` for the OOM
amendment, task evaluators, completion table, depth curves, reproduction
audit, artifact hashes, and claim boundaries.
## Preregistered multi-seed Chat sampling
`v2_lite_chat_sampling_probe.py` keeps the same four first-ranked sources,
eight prompt conditions, 512-token cap, BF16 checkpoint, and offload contract.
It enables the pinned official sampling values and explicitly disables top-k:
```text
do_sample=true · temperature=.3 · top_p=.95 · top_k=0
4 sources × 8 SHA-256-derived seeds × 8 conditions = 256 outputs
```
The eight rows of one source remain in a fixed batch. They are batch-seed
aligned, not common-random-number pairs: Transformers samples the whole batch
with one `torch.multinomial` call.
```bash
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
PYTHONPATH=/path/to/transformers-4.41.2-deps \
python -B experiments/deepseek/v2_lite_chat_sampling_probe.py \
--artifact-dir /path/to/deepseek-v2-lite-chat \
--reference-routing-json \
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
--greedy-json src/data/deepseek-v2-lite-chat-completion-512.json \
--human-eval /path/to/HumanEval.jsonl.gz \
--gsm8k /path/to/gsm8k/test.jsonl \
--tnews /path/to/tnews/test.json \
--tnews-archive /path/to/tnews_public.zip \
--wikitext /path/to/wikitext-validation.parquet \
--base-seeds \
19683830 1560062173 3978401375 1280933274 \
1467459869 1297359489 2722953988 3330978061 \
--max-new-tokens 512 \
--gpu-memory 28GiB \
--cpu-memory 80GiB \
--output src/data/deepseek-v2-lite-chat-sampling.json
```
The independent evaluator preserves stopping, semantic terminal state,
coverage, and correctness as separate ledgers. Exact duplicate code candidates
reuse a cache keyed by candidate, task tests, and sandbox-harness hashes:
```bash
python -B experiments/deepseek/v2_lite_chat_sampling_evaluator.py \
--sampling-json src/data/deepseek-v2-lite-chat-sampling.json \
--human-eval /path/to/HumanEval.jsonl.gz \
--gsm8k /path/to/gsm8k/test.jsonl \
--sandbox-image \
python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
--output src/data/deepseek-v2-lite-chat-sampling-eval.json
```
A fresh Python process reruns R0/R1. The reproduction comparer checks run
seeds, prompt hashes, complete token IDs, text, stopping state, and CPU/CUDA
RNG pre-state hashes:
```bash
python -B experiments/deepseek/v2_lite_chat_sampling_reproduction.py \
--formal-json src/data/deepseek-v2-lite-chat-sampling.json \
--rerun-json \
src/data/deepseek-v2-lite-chat-sampling-repro-r0r1.json \
--output \
src/data/deepseek-v2-lite-chat-sampling-reproduction.json
```
Formal results are 251/256 natural EOS and 242/256 globally unique complete
token trajectories. Strict GSM8K exact is 62/64; HumanEval official-test pass
is 63/64. These repeat one task per domain and are not benchmark estimates.
The R0/R1 fresh-process rerun reproduces all eight preregistered fields in
64/64 cells.
```text
formal 46c7edfce1409e798d6b0f06e905dd3a9d4ab6911acd8e42e2ec9d06d65345af
eval 078f486e021ffd0e4af8ad942d001a9f2429933b2fb0fb3523b7d1da5b24c4d8
rerun 72d050eace531bb38fb2592e39dc7f1b9c9003306c92c568dcd4acaaa691f6a0
compare 4d59a775943459604ea2d9976bb3be01ff7b3b08d71eb0ca22c376122ef82d15
```
See `research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md` and
`research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md` for seed derivation,
smoke gates, source-condition diversity tables, edge-set comparisons, task
failure cases, exact reproduction scope, primary sources, and non-claims.